ArulJerald opened a new pull request, #3989: URL: https://github.com/apache/iceberg-python/pull/3989
`_read_deletes` built its result with a dict comprehension that filtered the whole delete table once per distinct `file_path`, making the cost O(rows × distinct paths). A delete file referencing many data files therefore degraded quadratically. The ORC branch had the same shape via `unique()`. Closes #3983 # Rationale for this change Grouping by `file_path` is a single pass over the table, and it lets the Parquet and ORC branches share one code path instead of duplicating the per-path filter. Dictionary encoding of `file_path` is retained rather than dropped: positional delete files commonly repeat a small number of long paths across many rows, where the encoding is a substantial memory saving (measured ~24 MB vs ~198 MB on 2M rows referencing 2 distinct paths). Dictionaries are unified before grouping, since grouping raises `ArrowNotImplementedError: Unifying differing dictionaries` when chunks carry different ones. ## Are these changes tested? Yes. Added `test_read_deletes_many_distinct_file_paths` in `tests/io/test_pyarrow.py`, parametrized over Parquet and ORC, covering a delete file with 200 distinct `file_path` values and asserting the grouped positions are correct per path. The existing `test_read_deletes`, `test_delete`, and `test_delete_duplicates` cases continue to pass for both formats. ## Are there any user-facing changes? No. This is an internal performance change; `_read_deletes` returns the same mapping of `file_path` to positions as before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
