osipovartem opened a new pull request, #24972: URL: https://github.com/apache/datafusion/pull/24972
## Which issue does this PR close? - Closes #23022. ## Rationale for this change Predicate subqueries such as `EXISTS`, `IN`, and `NOT IN` are valid scalar expressions in a `SELECT` list, but DataFusion currently only decorrelates them when they appear in filters. Projection queries therefore reach physical planning with an unsupported logical subquery expression. This continues the work from the now-closed stale #23039 and covers multiple and correlated projection subqueries as well as SQL null semantics. ## What changes are included in this PR? - decorrelate `EXISTS`, `NOT EXISTS`, `IN`, and `NOT IN` expressions inside projections into mark joins - materialize projected `IN`/`NOT IN` values with SQL three-valued logic by tracking matches, nulls, and empty subqueries - preserve projection output names and avoid duplicate columns in correlated subquery projections ## What is the testing strategy for this PR? Optimizer plan tests cover projected `IN` rewriting. New SQLLogicTest cases cover uncorrelated and correlated `EXISTS`/`IN`, multiple subqueries in one projection, empty inputs, null inputs, and `NOT IN` three-valued logic. Locally validated with: - `cargo test -p datafusion-optimizer decorrelate_predicate_subquery` (53 passed) - `cargo test -p datafusion-sqllogictest --test sqllogictests -- subquery_projection` - `cargo clippy -p datafusion-optimizer --all-targets --all-features -- -D warnings` - `cargo fmt --all -- --check` ## Are there any user-facing changes? Yes. Predicate subqueries can now be evaluated as projected scalar expressions. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
