brgr-s opened a new pull request, #2936: URL: https://github.com/apache/iceberg-rust/pull/2936
## Which issue does this PR close? - Closes #2935 . ## What changes are included in this PR? The PR introduces (uncomments) `pos_deletes_by_path` in `PopulatedDeleteFileIndex` and populates the `HashMap`. To determine if a position delete file belongs in the map, it uses a helper `position_delete_target` that investigates 1. is `referenced_data_file` set? if yes, we know the delete file applies to that path 2. is `lower == upper` for the reserved field `RESERVED_FIELD_ID_DELETE_FILE_PATH`? If yes, we know that all entries point to the same path As a small ergonomic imrpovement it changes `eq_deletes_by_partition` and `pod_deletes_by_partition` to take a `(i32, Struct)` as key, where the `i32` is the `partition_spec_id` of the delete file. `get_deletes_for_data_file` can then use `pos_deletes_by_path` to lookup delete files that match that specific `data_file`. PR also adds a debug trace that could be usefull to determine if a read on a table is slow or OOMs (see linked issue) due to a large number of created delete file references. ## Are these changes tested? I have added unit tests. The change was also tested locally in conjuncture with PR #2620. The task was compacting a merge-on-read table with a huge number of delete files. The test scenario can be reproduced without PR #2620, see Issue #2935. ## AI Disclosure I used AI to investigate how Spark handles delete files, to review my changes and add more tests. I also used AI to write parts of the Issue and double-check the math. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
