xudong963 opened a new issue, #26396: URL: https://github.com/apache/datafusion/issues/26396
### Describe the bug A nested-loop join fails during physical optimization when its condition calls `array_any_match` on one input's array and the lambda captures a column from the other input. The default physical optimizer returns: ```text ProjectionPushdown caused by Error during planning: join_proj_push_down_1@2 unable to unwrap lambda from array_any_match at position 1 ``` Reproduced against Apache DataFusion `main` at [cdc2fc041b6a812a24933abc40da6475dbbf5729](https://github.com/apache/datafusion/commit/cdc2fc041b6a812a24933abc40da6475dbbf5729) (DataFusion 55.1.0 development version), using the public `SessionContext` API with `sql` and `nested_expressions` enabled. The same query executes successfully when physical optimizer rules are disabled. ### To Reproduce Use the DuckDB SQL dialect, which accepts the lambda syntax: ```sql SET datafusion.sql_parser.dialect = 'duckdb'; CREATE TABLE lists AS SELECT * FROM (VALUES ([1, 30], 'a', 7), ([40], 'b', 8)) AS v(l, k, n); SELECT s.k, s.n FROM (SELECT 25 AS cutoff) c JOIN lists s ON array_any_match(s.l, x -> x < c.cutoff); ``` The final `SELECT` fails during `ProjectionPushdown`, before execution. ### Expected behavior The query should plan and return: ```text +---+---+ | k | n | +---+---+ | a | 7 | +---+---+ ``` The first array contains `1 < 25`; the second array contains only `40`. As a control, building the session with `SessionStateBuilder::with_physical_optimizer_rules(vec![])` produces the following plan and the expected result above: ```text ProjectionExec: expr=[k@2 as k, n@3 as n] NestedLoopJoinExec: join_type=Inner, filter=array_any_match(l@1, (x) -> x@2 < cutoff@0) ProjectionExec: expr=[25 as cutoff] PlaceholderRowExec DataSourceExec: partitions=1, partition_sizes=[1] ``` ### Additional context Code inspection points to the join-filter expression extraction in [`JoinFilterRewriter`](https://github.com/apache/datafusion/blob/cdc2fc041b6a812a24933abc40da6475dbbf5729/datafusion/physical-optimizer/src/projection_pushdown.rs#L338): dependency analysis recognizes ordinary `Column` references, but the lambda parameter is a `LambdaVariable`. It sees the captured `cutoff` as a left-input dependency and attempts to extract the entire lambda into a projection, replacing it with `join_proj_push_down_1@2`. Rebuilding the higher-order function then fails in [`HigherOrderFunctionExpr::with_new_children`](https://github.com/apache/datafusion/blob/cdc2fc041b6a812a24933abc40da6475dbbf5729/datafusion/physical-expr/src/higher_order_function.rs#L425), because the second argument must remain a lambda and is now a plain column reference. This report covers the cross-input capture shown above; it does not establish that every join containing a lambda is affected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
