Gabriel39 opened a new pull request, #65502:
URL: https://github.com/apache/doris/pull/65502
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
FileScannerV2 had three related Iceberg delete correctness and efficiency
gaps:
- Materialized readers applied delete expressions as ordinary predicates,
keeping rows for which a delete predicate returned true.
- Equality-delete fields absent from older data files were rejected,
although Iceberg schema evolution requires those values to be read as NULL.
Name-mapped files without physical field ids also could not resolve the
equality key.
- Equality and position delete files shared by multiple data splits were
parsed again for every split, and position bounds were ignored.
This change builds an explicit keep filter by inverting delete matches,
resolves equality keys by field id or name mapping and supplies typed NULL
literals for missing fields, honors position bounds, and caches parsed delete
data in `SplitReadOptions.cache` for reuse across splits. The implementation is
limited to FileScannerV2.
### Release note
Fix Iceberg V2 row-level delete filtering across schema evolution and reuse
parsed delete files across splits.
### Check List (For Author)
- Test: Unit Test
- 26 focused `EqualityDeletePredicate`, JSON materialized-reader, and
Iceberg V2 delete tests passed
- Added V2 differential coverage to the existing Iceberg equality-delete
schema-evolution regression suite; the remote run was skipped because external
Iceberg tests are disabled in that environment
- Behavior changed: Yes, FileScannerV2 now applies Iceberg equality and
position deletes with correct semantics
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
