hubgeter opened a new pull request, #70615: URL: https://github.com/apache/doris/pull/70615
### Background Iceberg tables can declare a default sort order, while each data file records the order that its writer actually executed. Doris builds write-sort expressions in FE and sends them to the BE data writer, then converts the reported files into Iceberg manifest entries. ### What problem does this PR solve? Issue Number: None Related PR: #60540 (introduced the original sort support) PR Type: Bugfix Problem Summary: Doris skipped transform and nested sort fields when building write keys but still recorded the complete table sort order on the resulting data files. UPDATE/MERGE data writers also do not execute the transmitted sort_fields. This change records an order only when the actual data sink executes the complete supported order; other writes are recorded as unsorted without changing the table's default order. ### Details - **Change map:** 189 changed production lines and 556 changed test lines; no generated files or documentation changes. The production change adds a shared sort-order resolver, binds the actual file order once per transaction, and passes that decision through every data-file commit path. Most of the diff adds plan, resolver, helper, and manifest-commit regression coverage. - **Frequency:** Writes to tables with transform/nested sort keys, and UPDATE/MERGE writes to sorted tables, could create misleading file-order metadata. - **Root cause:** Planning silently dropped unsupported fields, while commit inferred a file's order from the table default. The row-level sink's sort_fields are not consumed by its data writer. - **Why existing tests missed it:** Plan-parity tests explicitly accepted an empty bucket-sort key list. Metadata tests checked table-order pinning separately from the actual sink plan. - **Fix:** Resolve the entire order before sorting; require complete sort_info including matching direction and NULL ordering before preserving a nonzero ID. Pin the effective order for shared REWRITE groups, reject inconsistent group decisions, and explicitly pass the pinned order to INSERT/OVERWRITE, REWRITE, and row-level data-file conversion. Keep the original table order for metadata fences. - **Other paths and branches:** CTAS shares the INSERT path. DELETE and delete-only MERGE retain their independent delete-file handling. branch-4.2 is affected, but its legacy FE layout requires an adapted backport rather than a direct cherry-pick of this connector patch. - **Behavior change:** Unsupported complete orders use the existing unsorted writer and record order 0. UPDATE/MERGE data files record order 0 until their data writer executes a complete sort. Supported identity orders remain available for INSERT/OVERWRITE/REWRITE. ### Testing - Before the planner fix, `IcebergWritePlanProviderTest#getWriteSortColumnsNullForBucketSortOrder` failed with `expected: <null> but was: <[]>` using the original planner source. - After the fix, the project preset `./run-fe-ut.sh --run 'org.apache.doris.connector.iceberg.IcebergWriteSortOrderTest,org.apache.doris.connector.iceberg.IcebergWritePlanProviderTest,org.apache.doris.connector.iceberg.IcebergWriterHelperTest,org.apache.doris.connector.iceberg.IcebergConnectorTransactionTest'` passed all 275 selected tests locally. - The plan-to-transaction tests inspect the actual committed manifest metadata across 14 INSERT/OVERWRITE/REWRITE/UPDATE/MERGE cases, including identity, bucket, truncate, and nested orders. - Module Checkstyle passed with 0 violations. `git diff --check` passed. Validation covers plans and committed manifest metadata. Physical data-file row-order and cluster regression tests were not run. ### Release note New Iceberg data files are recorded as unsorted when Doris does not execute the complete table sort order, including unsupported transform/nested orders and current UPDATE/MERGE data writers. Supported complete identity sorting remains available. Table sort definitions and historical file metadata are preserved. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: local FE tests and Checkstyle; cluster regression tests pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Production changes, tests, and this description. - Human review: Pending maintainer review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
