Hey iceberg-rust folks,
I just filed Issue 2950 (https://github.com/apache/iceberg-rust/issues/2950) regarding scaling of equality delete file application. There is already a PR aiming to fix the scaling issue by implementing a similar approach to Java, but the original authors' organization (@t3hw) has since moved away from iceberg-rust. I took in inspiration from that approach but rebuild the PR on current main and using a `RowFilter` with `ArrowPredicateFn`. I should state that this PR does not include PR #2873 (fix: avoid lost-wakeup hang in get_equality_delete_predicate_for_delete_file_path) and PR #2630 (fix(reader): avoid deadlock when loading equality deletes). Both are approved by @viirya, so hoping those PRs will land soon I'd rather rebase my changes on top of those, but I was thinking to file the PR for review anyways. The changes amount to a diff of +781/-172 (a lot of the added lines are tests) and the `CONTRIBUTING.md`says: > We generally discourage large pull requests that are over 300-500 lines of > diff. I don't think this PR could be split up into smaller PRs, so I am asking for guidance on how to proceed. Best regards Stephan
