mohitgurav20 commented on PR #25811: URL: https://github.com/apache/datafusion/pull/25811#issuecomment-6337106227
@kosiew Thanks for the clear feedback and the reproducible Criterion benchmark. I have updated the implementation in commit `d7abce7` to eliminate the repeated full-expression evaluation overhead while preserving early rejection: 1. **Direct Guarantee Evaluation**: Instead of calling `predicate.prune()` on each loaded column (which re-evaluates the entire expression tree and propagates `None` across unloaded columns), we now evaluate column-level `Guarantee::In` literals directly against the loaded `Sbbf` via `BloomFilterStatistics::is_known_not_present`. This reduces surviving-column evaluation complexity from $O(N^2)$ to $O(N)$ with no redundant expression rebuilding. 2. **I/O Filtering**: Pre-filters columns so that any predicate column lacking `Guarantee::In` guarantees does not trigger unnecessary asynchronous Bloom filter loading. 3. **Benchmarks Added**: Added the Criterion benchmark suite in `datafusion/core/benches/parquet_bloom_filter_pruning.rs` covering `reject_first`, `reject_last`, and `survive_all` across 2, 8, and 32 columns on both memory and throttled storage. All 18 combinations pass and verify expected pruning metrics. 4. **Tests & Coverage**: Added unit tests covering all branch outcomes of `is_known_not_present` and verified all 296 tests in `datafusion-datasource-parquet` pass cleanly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
