suxiaogang223 opened a new pull request, #66012:
URL: https://github.com/apache/doris/pull/66012
### What problem does this PR solve?
Issue Number: None
Related PR: #62821, #65581, #66010
Problem Summary: branch-4.1 already backported the Paimon and BE partition
metadata changes through #65581, but Hive-style scans, Hudi, Iceberg, and file
load paths still had incomplete FE metadata propagation. Hive default
partitions were encoded as `\N`, which lost the distinction between NULL and a
literal `\N`. Iceberg attached partition metadata only when runtime partition
pruning was enabled, used unstable per-map ordering, and did not safely handle
files written with different partition specs.
This change:
- propagates aligned partition values and explicit NULL flags through
Hive-style scans, Hudi, and both legacy and Nereids file load paths;
- preserves a literal `\N` as data while representing Hive default
partitions as NULL;
- sends stable Iceberg identity-partition metadata for every split, keyed by
both spec ID and partition data;
- restricts Iceberg path partition keys to identity columns shared by all
partition specs, which is safe for both FileScannerV2 and the legacy scanner's
scan-level slot mapping.
### Release note
Fix partition-column materialization, NULL handling, and Iceberg partition
evolution compatibility for external table scans and file loads on branch-4.1.
### Check List (For Author)
- Test: Unit Test
- `./run-fe-ut.sh --run
org.apache.doris.common.util.BrokerUtilTest,org.apache.doris.datasource.hive.source.HiveScanNodeTest,org.apache.doris.datasource.iceberg.IcebergUtilsTest,org.apache.doris.datasource.iceberg.source.IcebergScanNodeTest,org.apache.doris.datasource.paimon.source.PaimonScanNodeTest`
- Behavior changed: Yes. External partition metadata and NULL values are
materialized consistently per split.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
