LukaZdravic opened a new pull request, #59591:
URL: https://github.com/apache/spark/pull/59591
### What changes were proposed in this pull request?
Two fixes for hidden `USING` columns (like `d.k` after `r LEFT JOIN d USING
(k)`) below an ASOF JOIN, one in each analyzer:
- Fixed-point: an ASOF JOIN with `USING` now passes up the hidden columns of
its inputs. Added one shared helper for regular `USING` and ASOF one ->
`NaturalAndUsingJoinResolution.buildUsingJoinProject`.
- Single-pass: `AsOfJoinResolver` now keeps the IDs of hidden columns when
it merges the ID mappings of the two sides. Before, when the same table was on
both sides, `d.k` in `MATCH_CONDITION` was remapped to the other copy's `k`.
### Why are the changes needed?
Each bug makes dual-run fail. The single-pass one also fails the query at
run time.
```sql
CREATE OR REPLACE TEMP VIEW using_left(k, lts) AS VALUES (1, 10), (2, 20),
(8, 30);
CREATE OR REPLACE TEMP VIEW using_right(k, rts) AS VALUES (1, 5), (2, 12),
(8, 15);
CREATE OR REPLACE TEMP VIEW using_keys(k) AS VALUES (1), (2);
SELECT l.lts, d.k
FROM using_left l ASOF JOIN (using_right r LEFT JOIN using_keys d USING (k))
MATCH_CONDITION (l.lts >= r.rts) USING (k);
```
Before: fixed-point fails with `UNRESOLVED_COLUMN` for `d.k`. After: `(10,
1), (20, 2), (30, NULL)`.
```sql
SELECT x.k
FROM (using_right r LEFT JOIN using_keys d USING (k))
ASOF JOIN using_keys x MATCH_CONDITION (d.k >= x.k);
```
Before: single-pass plans `AsOfJoin (x.k >= x.k)` and the query fails with
`INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND`. After: `(1), (2)`.
### Does this PR introduce _any_ user-facing change?
Yes. The two queries above now return rows instead of failing.
### How was this patch tested?
New queries FVT-ASOF-2-043 to 049 in `join-asof-semantics.sql`. Tests run
both analyzers and compare their plans. Without the fixed-point fix, 043 to 045
and 047 to 049 fail. Without the single-pass fix, 046 and 047 fail. The ASOF
suites and the regular join tests (`using-join.sql`, `natural-join.sql`,
`DataFrameJoinSuite`, `ResolveNaturalJoinSuite`) still pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
