yadavay-amzn opened a new pull request, #17205: URL: https://github.com/apache/iceberg/pull/17205
Closes #17203. ## Problem `RemoveDanglingDeletesSparkAction` removes dangling deletion vectors (v3, PUFFIN) but not dangling v2 file-scoped position-delete files (PARQUET). A file-scoped position delete whose referenced data file has been removed, but whose sequence number is `>=` the partition's minimum data sequence number, is never cleaned up and lingers in table metadata indefinitely. - `findDanglingDvs()` restricts the referenced-data-file check to `file_format == PUFFIN`. - `findDanglingDeletes()` only removes position deletes whose `sequence_number < min_data_sequence_number`. A v2 file-scoped position delete whose referenced data file is gone, but whose sequence number stays `>=` the partition minimum (because another live data file in the partition keeps the minimum low), is caught by neither. ## Change Generalize the referenced-data-file left-join (previously DV-only) to also cover v2 file-scoped position deletes: derive the referenced data file from the delete file's `lower_bounds`/`upper_bounds` for the `file_path` field when they are equal, then left-join against the data files and flag entries with no match. A guard restricts this to single-data-file (file-scoped) deletes only: partition-scoped position deletes (which have differing lower/upper bounds) and equality deletes are excluded, so no delete that may still cover live data is ever removed. Existing DV handling is preserved (PUFFIN files continue to use `referenced_data_file` directly). ## Testing Extended `TestRemoveDanglingDeleteAction` with a case that removes a file-scoped position delete when its referenced data file is gone, and a case asserting a partition-scoped position delete is not removed. The new file-scoped case fails on the unpatched code and passes with the fix; existing DV and sequence-number cases continue to pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
