sunchao opened a new pull request, #6733: URL: https://github.com/apache/datafusion-comet/pull/6733
## Which issue does this PR close? No Comet issue is closed. Backports [apache/datafusion#25602](https://github.com/apache/datafusion/pull/25602). ## Rationale for this change Comet's runtime hash-join filters can reject decoded rows using exact hash-table membership, but Parquet statistics cannot use that membership expression to skip gaps between build keys. For build keys `50` and `350`, row groups covering `[100, 199]` and `[200, 299]` overlap the build's min/max bounds even though neither group can match. Attach DataFusion's key-range bitmap summary to the runtime filter so Parquet statistics pruning can skip those interior row groups before reading their data. The regression test prunes two of four row groups, decodes 200 rather than 400 rows, reads fewer bytes, and returns the same two joined rows in both CollectLeft and Partitioned modes. ## What changes are included in this PR? - Backport the merged bitmap implementation to DataFusion 55.1.0 by vendoring `datafusion-physical-plan` and `datafusion-pruning` and patching both together. The upstream merge uses Arrow 60; this preserves Comet's existing Arrow 59 and Iceberg dependency versions. Every existing locked package retains its version; additional packages support the vendored crates' tests. - Preserve the existing `HashTableLookupExpr::new` signature and attach the optional bitmap through a builder. Skip allocation when dynamic filtering is disabled and retain normal join behavior if optional bitmap memory cannot be reserved. - Preserve conservative handling of missing statistics and `CASE` without `ELSE`, and adapt lossless unsigned-to-signed widening casts to the older pruning API. - Add a Comet regression through `PhysicalPlanner::apply_join_dynamic_filter`, with page pruning and decode-time row filtering disabled to isolate row-group pruning. - Include upstream licenses, archive checksums, a complete patch against the published crate sources, and removal instructions in `native/vendor/README.md`. Most added source is unchanged upstream code; `native/vendor/bitmap-backport.patch` isolates the backport changes. This is a temporary compatibility backport. Runtime filtering remains opt-in; Delta scan integration is outside this PR. ## How are these changes tested? Passed locally: - `cargo test --locked -p datafusion-pruning --lib`: 99 tests. - `cargo test --locked --no-default-features -p datafusion-comet --lib dynamic_filter`: 49 tests, including the new regression in both join modes. - `cargo test --locked -p datafusion-physical-plan --lib key_range_bitmap`: 3 tests. - `cargo test --locked -p datafusion-physical-plan --lib pruning_bitmap`: 2 tests covering optional memory allocation and bitmap sharing. - Negative control: the same Comet regression fails against the original registry dependencies because zero row groups are pruned instead of two. - `cargo build --locked --workspace` with default features and JDK 17. - JVM compilation with `./mvnw compile -DskipTests -Dscalastyle.skip=true -Dspotless.skip=true` (temporary Maven proxy settings supplied separately). - `cargo fmt --all --check`, Markdown formatting checks, and `git diff --check` for non-vendored changes. The imported upstream sources retain two pre-existing trailing-whitespace lines; the provenance patch also contains blank context lines. - `cargo clippy --locked --all-targets --workspace --no-default-features -- -D warnings` on Rust 1.99, also checked on Rust 1.98.1. - Verified the source archive checksums and that applying the documented patch reconstructs both vendored crates exactly. The Spark SQL suite has not been run locally; `run-spark-4.1-tests` requests that additional CI coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
