LukaZdravic opened a new pull request, #59301:
URL: https://github.com/apache/spark/pull/59301
### What changes were proposed in this pull request?
Subqueries can now use outer columns inside an ASOF JOIN, like a regular
JOIN: in either input and in ON. LEFT ASOF JOIN still rejects them in the right
input, like LEFT JOIN.
The left input always carries the outer values, and ON matches them with the
right input, so each left row picks a right row of its own outer row. An outer
column in MATCH_CONDITION fails with
`ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE`.
Follow-up: the planner checks this match row by row instead of using it as a
join key, which can be slow. A separate PR can make it a key, as
`ExtractEquiJoinKeys` does for regular joins. This means that it would not
check the equality of tags for each row but checking only by groups and 100%
skipping the checks for the tags that are not the same. On a big number of
different tags this can prove quite benefitial e.g. today they are all in 1
group so 100k rows get scanned by each tag even though they do not need to,
with this fix that 100k would get broken up into lets say 1000 groups ( 1000
different tags ) of 100 different rows, so different tags get at most 100
checks.
### Why are the changes needed?
A regular JOIN supports these queries, but ASOF JOIN rejected them:
```sql
SELECT s.symbol,
(SELECT max(q.bid_price)
FROM (SELECT * FROM trades t WHERE t.symbol = s.symbol) t
ASOF JOIN quotes q MATCH_CONDITION (t.trade_time >= q.quote_time) ON
t.symbol = q.symbol)
FROM VALUES ('AAPL'), ('MSFT') AS s(symbol);
```
Before: `ACCESSING_OUTER_QUERY_COLUMN_IS_NOT_ALLOWED`. After: `AAPL 180.20`,
`MSFT 420.50`.
### Does this PR introduce _any_ user-facing change?
Yes. These queries now run. An outer column in MATCH_CONDITION now fails
during analysis instead of in the optimizer.
### How was this patch tested?
New queries in `join-asof-crossfeature.sql` (FVT-ASOF-9-011 to 022) cover
scalar, IN, and EXISTS subqueries, LEFT ASOF JOIN, a NULL outer value, and 2
rejected cases. Each result matches the subquery run once per outer value. New
tests in `DecorrelateInnerQuerySuite` and `AsOfJoinSQLSuite` check the plans
and the error. Without the fix, the new tests fail.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
