talatuyarer commented on PR #14264: URL: https://github.com/apache/iceberg/pull/14264#issuecomment-5649211018
@aokolnychyi I **reworked the planning** to follow the structure you describe. The core of the change is a per-snapshot "effective delete list": `CumulativeDeleteIndexes` lazily builds, for each changelog snapshot, one index of base deletes plus deletes added earlier in the range, minus delete files removed earlier in the range. Both DeletedDataFileScanTask and DeletedRowsScanTask now resolve existing deletes from that shared index, which fixes the correctness issues you and @manuzhang identified (phantom removed deletes; missing base deletes when the range has no equality deletes). Point-by-point vs your notes: - `AddedRowsScanTask`: only same-snapshot deletes are attached; existing deletes are never scanned for new files. - `DeletedDataFileScanTask`: existing deletes come from the effective delete list for the deletion snapshot, as you describe. - `REWRITE outputs`: instead of reconstructing base + deltas, DeletedRows planning scans the snapshot's live data manifest list (partition-pruned), which includes mid-range compaction outputs by construction — so new deletes applying to compacted files are covered. - Delete compactions mid-range: `REPLACE snapshots` are skipped, so their manifest churn never enters the accumulated state and the old, content-equivalent delete files are used. I also implemented your pruning suggestions as well: - The existing delete index is now scoped by the changes in the range: removed data files and added delete files define an affected partition set and an affected data file location set. Base delete manifests are pruned by the affected partitions on top of the user filter, and within the surviving manifests, file-scoped position deletes and DVs are loaded only if they reference an affected data file location (or lie in a partition where a partition-scoped delete was added), while equality and partition-scoped position deletes are loaded only for affected partitions. - `DeletedRowsScanTask` planning prunes each snapshot's data manifest scan to the partitions of that snapshot's added delete files, and when every added delete is file-scoped (DVs or single-file position deletes), entries are further pruned to the referenced file locations — your "same partition set and file name filter" point. - Net effect for DV-only workloads: the base delete scan touches only manifests and DVs for the files actually affected in the range, which is the outcome of your DV fast path. The scope is conservatively disabled when an added delete has global reach (an equality delete with an unpartitioned spec), and manifest-level partition pruning applies only while the affected partitions all belong to one spec. Let me know if you see anything in the reworked structure that still diverges from your algorithm. @manuzhang i also addressed your comments. Sorry for taking longer time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
