Gabriel39 opened a new pull request, #66825: URL: https://github.com/apache/doris/pull/66825
### What problem does this PR solve? This is a semantic backport of the external-write and schema-evolution fixes from #66345, #66348, #66408, #66529, and #66750 to `branch-4.1`. The source PRs were implemented on the connector-plugin architecture, while `branch-4.1` still uses the legacy external-table stack. This change ports the invariants and failure handling to the branch-specific owners instead of mechanically cherry-picking plugin classes. Key behavior preserved on `branch-4.1`: - Keep Iceberg planning, serialization, and commit on one metadata generation; add exact snapshot/branch overwrite OCC checks and retry-state reset. - Handle nested schema evolution safely, including typed initial defaults, required-field history, alias collisions, nested partition sources, stable nested IDs, and collection parent null masks. - Transfer external-file ownership only through the final bounded report, retain rollback ownership until FE acknowledgement, and keep rolling compatibility for non-Iceberg report vectors. - Bound asynchronous writer and multi-partition sorter memory, including queue admission ownership and largest-sorter revocation. - Preserve writer close errors and file sizes; harden Iceberg data/delete-file cleanup and Hive deferred multipart validation/rollback. - Isolate Azure multipart writers with UUID-namespaced block IDs and commit the exact staged IDs. - Make orphan-file deletion retained-snapshot-aware, prefix-safe, GC-enabled, and bounded. - Preserve nested comments as SQL literals and isolate MTMV test job lookup. The Paimon validation-detail fix from #66408 and canonical `Paimon` capitalization from #66750 are already present in the legacy Paimon paths on `branch-4.1`; their observable contracts were verified and no plugin-only code was introduced. ### Architecture-specific notes - The Iceberg scan compatibility gate evaluates historical schema/default/equality-delete requirements only while a smooth-upgrade source BE exists. Equality deletes are still checked across every delete manifest in the selected snapshot. - Empty overwrite scans the retained target branch rather than the main branch, so divergent branch files cannot be mixed into the delete set. - The legacy Hive planner explicitly advertises deferred Azure multipart support, matching the connector-planner handshake from the source change. ### Tests - FE focused reactor suite: 174 tests, 0 failures, 0 errors, 1 skipped. - Target-branch overwrite regression verified red/green against a divergent main branch. - Regression framework `SuiteJobLookupTest`: 1 test passed. - All 35 changed C++ translation units present in the ASAN compile database built successfully. - Azure BE source passed a syntax-only compile with the repository C++ flags (it is excluded from the local build configuration). - clang-format 16 check passed for all 63 changed C/C++ source and header files. - FE Checkstyle passed with 0 violations. The full local BE unit-test link is unavailable because the configured third-party Arrow bundle lacks a header required by an unchanged source file; targeted compilation covers every changed translation unit. Pick from master: #66345 #66348 #66408 #66529 #66750 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
