CTTY commented on PR #2590: URL: https://github.com/apache/iceberg-rust/pull/2590#issuecomment-4654273332
> I think the key point is the design of MergingSnapshotProduder, which contains a lot of indices to speed up the confliction check. I think you meant [`DeleteFileIndex`](https://github.com/apache/iceberg/blob/1cea23eda51c9b9ddcfb88dd499b1fd14f3bf3b3/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L624-L625) in java implementation and the `conflictDetectionFilter`. In rust's [`DeleteFileIndex`](https://github.com/apache/iceberg-rust/blob/main/crates/iceberg/src/delete_file_index.rs#L56), we don't allow filter to be pushed down at this point, so I didn't include that change. In the future, we could store the filter in snapshot operations like `RowDeltaAction` and pass the filter to `SnapshotValidator::validate_no_new_deletes` easily. The current implementation won't block that change, we will only need to change the API in `SnapshotValidator` after adding conflict_detecting_filter support to `DeleteFileIndex` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
