hubgeter opened a new pull request, #70615:
URL: https://github.com/apache/doris/pull/70615

   ### Background
   
   Iceberg tables can declare a default sort order, while each data file 
records the order that its writer actually executed. Doris builds write-sort 
expressions in FE and sends them to the BE data writer, then converts the 
reported files into Iceberg manifest entries.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #60540 (introduced the original sort support)
   PR Type: Bugfix
   
   Problem Summary:
   
   Doris skipped transform and nested sort fields when building write keys but 
still recorded the complete table sort order on the resulting data files. 
UPDATE/MERGE data writers also do not execute the transmitted sort_fields. This 
change records an order only when the actual data sink executes the complete 
supported order; other writes are recorded as unsorted without changing the 
table's default order.
   
   ### Details
   
   - **Change map:** 189 changed production lines and 556 changed test lines; 
no generated files or documentation changes. The production change adds a 
shared sort-order resolver, binds the actual file order once per transaction, 
and passes that decision through every data-file commit path. Most of the diff 
adds plan, resolver, helper, and manifest-commit regression coverage.
   - **Frequency:** Writes to tables with transform/nested sort keys, and 
UPDATE/MERGE writes to sorted tables, could create misleading file-order 
metadata.
   - **Root cause:** Planning silently dropped unsupported fields, while commit 
inferred a file's order from the table default. The row-level sink's 
sort_fields are not consumed by its data writer.
   - **Why existing tests missed it:** Plan-parity tests explicitly accepted an 
empty bucket-sort key list. Metadata tests checked table-order pinning 
separately from the actual sink plan.
   - **Fix:** Resolve the entire order before sorting; require complete 
sort_info including matching direction and NULL ordering before preserving a 
nonzero ID. Pin the effective order for shared REWRITE groups, reject 
inconsistent group decisions, and explicitly pass the pinned order to 
INSERT/OVERWRITE, REWRITE, and row-level data-file conversion. Keep the 
original table order for metadata fences.
   - **Other paths and branches:** CTAS shares the INSERT path. DELETE and 
delete-only MERGE retain their independent delete-file handling. branch-4.2 is 
affected, but its legacy FE layout requires an adapted backport rather than a 
direct cherry-pick of this connector patch.
   - **Behavior change:** Unsupported complete orders use the existing unsorted 
writer and record order 0. UPDATE/MERGE data files record order 0 until their 
data writer executes a complete sort. Supported identity orders remain 
available for INSERT/OVERWRITE/REWRITE.
   
   ### Testing
   
   - Before the planner fix, 
`IcebergWritePlanProviderTest#getWriteSortColumnsNullForBucketSortOrder` failed 
with `expected: <null> but was: <[]>` using the original planner source.
   - After the fix, the project preset `./run-fe-ut.sh --run 
'org.apache.doris.connector.iceberg.IcebergWriteSortOrderTest,org.apache.doris.connector.iceberg.IcebergWritePlanProviderTest,org.apache.doris.connector.iceberg.IcebergWriterHelperTest,org.apache.doris.connector.iceberg.IcebergConnectorTransactionTest'`
 passed all 275 selected tests locally.
   - The plan-to-transaction tests inspect the actual committed manifest 
metadata across 14 INSERT/OVERWRITE/REWRITE/UPDATE/MERGE cases, including 
identity, bucket, truncate, and nested orders.
   - Module Checkstyle passed with 0 violations. `git diff --check` passed.
   
   Validation covers plans and committed manifest metadata. Physical data-file 
row-order and cluster regression tests were not run.
   
   ### Release note
   
   New Iceberg data files are recorded as unsorted when Doris does not execute 
the complete table sort order, including unsupported transform/nested orders 
and current UPDATE/MERGE data writers. Supported complete identity sorting 
remains available. Table sort definitions and historical file metadata are 
preserved.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details).
   - Does this need documentation: No.
   - Pre-submission status: Partially verified: local FE tests and Checkstyle; 
cluster regression tests pending.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6).
   - Generated: Production changes, tests, and this description.
   - Human review: Pending maintainer review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to