haohuaijin opened a new pull request, #25259: URL: https://github.com/apache/datafusion/pull/25259
## Which issue does this PR close? Closes #25244. ## Rationale for this change Pushing a join dynamic filter through nested joins can silently discard matching rows when a child schema contains duplicate column names. Name-based remapping can turn a predicate on the second `id` column into a predicate on the first `id` column. Repartition can introduce the same error in parallel execution, even after the join mapping is corrected. ## What changes are included in this PR? - Add explicit output-to-input column mappings for filter remapping, while preserving the existing name-based APIs for other callers. - Make HashJoin use its column indices and output projection to map filters to each input. For semi joins, map output join keys to their paired input keys, including differently named keys. - Make Repartition preserve filter column positions. - Add focused physical optimizer tests and a SQL regression to the existing `dynamic_filter_pushdown_config.slt` file. This PR is scoped to HashJoin and Repartition; migrating other operators is left for separate changes. ## What is the testing strategy for this PR? The SQL regression verifies the same expected row with join dynamic filtering enabled and disabled, using both one and four target partitions. The focused tests also cover duplicate child column names, reordered join projections, both semi join directions, and both Repartition pushdown phases. Passed locally: - `./dev/rust_lint.sh` (including Clippy, formatting, license, spelling, and Rust documentation checks) - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test --profile ci -p datafusion --test core_integration physical_optimizer::filter_pushdown` (62 tests) - `cargo test --profile ci --test sqllogictests -- dynamic_filter_pushdown_config.slt` The `push_down_filter` benchmark was also compared against the main baseline using the unoptimized CI profile, with no regression observed. The full workspace test suite was not run to completion for this change. ## Are there any user-facing changes? Queries affected by incorrect filter remapping now retain their matching rows with join dynamic filtering enabled, including parallel execution. Existing public method signatures are preserved; the positional mapping helpers are additive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
