Awesome06 opened a new pull request, #27244:
URL: https://github.com/apache/datafusion/pull/27244
## Which issue does this PR close?
- Closes #26234.
## Rationale for this change
When a correlated subquery refers to the outer query from multiple inner
relations that share identical column names, `SubqueryAlias` assigns unique
schema identifiers (e.g. `k` and `k:1`), but `replace_qualified_name` mapped
each correlated column strictly by unqualified name into
`__correlated_sq_N.<col>`. Consequently, predicates intended for distinct
relations were collapsed onto the first matching relation while the other was
ignored, silently returning incorrect results for `EXISTS`, `NOT EXISTS`, `IN`,
and scalar subqueries.
Following maintainer guidance from @alamb on #26234 / #26236 (and
discussions in #5492 and #26672), rather than trying to patch this with
heuristic alias rewrites that risk further wrong results, I have integrated
explicit fail-fast error handling for this unsupported query pattern.
DataFusion now raises an explicit planning error when inner relation columns
have ambiguous duplicate references across relations.
## What changes are included in this PR?
- `datafusion/optimizer/src/decorrelate.rs`:
- Updated `build_join_filter` signature to accept `subquery_schema:
&DFSchema`.
- Added duplicate field check against `subquery_schema.fields()`. If any
correlated column has duplicate references across relations, it fails fast with
`Unsupported correlated subquery: inner relation column '{col}' has ambiguous
duplicate references across relations under alias '{subquery_alias}'`.
- Added unit test in `test_build_join_filter` asserting an error on
duplicate qualified field names.
- `datafusion/optimizer/src/decorrelate_predicate_subquery.rs`: Passed
`new_plan.schema()` into `build_join_filter`.
- `datafusion/optimizer/src/scalar_subquery_to_join.rs`: Passed
`decorrelated_subquery.schema()` into `build_join_filter`.
- `datafusion/sqllogictest/test_files/subquery.slt`: Added regression test
suite covering `EXISTS`, `NOT EXISTS`, `IN`, and scalar subqueries asserting
the plan error, along with a positive control query ensuring non-ambiguous
single-relation correlations continue to succeed.
## What is the testing strategy for this PR?
- **Sqllogictests:** Verified via `cargo test -p datafusion-sqllogictest
--test sqllogictests -- subquery` covering 5 regression error queries and 1
positive control query.
- **Unit & Integration Tests:** Verified via `cargo test -p
datafusion-optimizer` (975 unit tests + 29 integration tests + 5 doctests
passing).
- **Linters:** Verified via `cargo fmt --all -- --check` and `cargo clippy
--all-targets --workspace -- -D warnings`.
## Are there any user-facing changes?
Queries with correlated subqueries that previously produced silent wrong
results due to colliding inner column names will now fail fast with an explicit
error (`Unsupported correlated subquery: inner relation column ... has
ambiguous duplicate references across relations`) rather than returning
incorrect rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
