xudong963 opened a new pull request, #26399: URL: https://github.com/apache/datafusion/pull/26399
## Which issue does this PR close? - Closes #26396. ## Rationale for this change A nested-loop join whose lambda captures a column from the other input can fail during physical optimization with `unable to unwrap lambda`, even though the query executes correctly with physical optimization disabled. These queries should work with the default optimizer while retaining projection pushdown for expressions that can be evaluated on one join input. ## What changes are included in this PR? - Keep each lambda and its body in the enclosing high-order function's scope when extracting join-filter expressions. Ordinary sibling expressions, other function arguments, and complete high-order expressions that depend on only one input remain eligible for pushdown. - Remap captured columns and rebase lambda parameter indices when moving an expression to an input projection, extending the intermediate filter schema, and pruning unused filter columns. Nested lambdas retain their relative parameter offsets and bindings. - Add `nested_loop_join_lambda.slt` with result and physical-plan regressions. This change preserves lambda bodies within their enclosing function; it does not introduce hoisting of individual computations from inside a lambda body. ## What is the testing strategy for this PR? The new SQL test covers the original reproducer with either join-input order, mixed predicates, nested lambdas, parameter shadowing, empty/NULL arrays, and extraction of complete high-order functions onto either input. Different input and filter schema widths exercise parameter rebasing. Three `EXPLAIN` checks verify that ordinary expressions and complete high-order functions are still pushed down. Passing local checks: ```text cargo fmt --all cargo clippy --all-targets --workspace --features avro,integration-tests,extended_tests -- -D warnings cargo clippy --locked -p datafusion-physical-optimizer --all-targets --all-features -- -D warnings cargo test --locked -p datafusion-physical-optimizer projection_pushdown::test cargo test --locked -p datafusion-sqllogictest --test sqllogictests -- nested_loop_join_lambda.slt ``` The targeted tests passed: 8 existing optimizer unit tests and 9 SQL result checks plus 3 physical-plan checks. Temporarily disabling each of the three parameter-rebasing steps caused the regression file to fail. The broader `cargo clippy --locked --all-targets --all-features -- -D warnings` command reports three `clippy::mutable_key_type` errors in the unchanged `datafusion/optimizer/src/push_down_filter.rs` (lines 1178, 1182, and 1683). The same command reproduces those errors on the unmodified upstream base `cdc2fc0`; the CI feature combination listed above passes. `uv run ./dev/rust_lint.sh` passed the checks through Markdown links, including the CI clippy configuration, formatting, license headers, typos, generated documentation, and workflow configuration checks. It stopped at the security audit because the installed `cargo-audit` cannot parse the CVSS 4.0 score in advisory `RUSTSEC-2026-0073`; later script stages were not run. ## Are there any user-facing changes? Nested-loop joins with lambda predicates now plan and execute with the default physical optimizer. There are no public API or configuration changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
