LukaZdravic opened a new pull request, #59102:
URL: https://github.com/apache/spark/pull/59102
### What changes were proposed in this pull request?
Forward ASOF joins (`t.k <= r.k` or `t.k < r.k`) in `SortMergeAsOfJoinExec`
now keep the first right row that matches. Before, they reused the Nearest
loop, which computes `right - left` for every candidate and keeps the smallest.
The right rows are sorted ascending, so the first match is always the
closest one. This mirrors Backward, which keeps the last match. The operator
now reads the direction (Backward, Forward, Nearest) from the as-of condition.
Nearest, and any condition shape it does not recognize, keep the old distance
loop.
### Why are the changes needed?
The subtraction overflows when values are far apart, even though the answer
never needs it:
```sql
SET spark.sql.join.asofJoin.enabled=true;
SELECT r.tag
FROM VALUES (-100) AS t(k)
ASOF JOIN VALUES (-50, 'near'), (2147483647, 'far') AS r(k, tag)
MATCH_CONDITION (t.k <= r.k);
```
- ANSI on: fails with `ARITHMETIC_OVERFLOW`.
- ANSI off: `2147483647 - (-100)` wraps to a negative number, so it returns
`far`.
- Expected, and the result after this PR: `near`.
The same bug hits BIGINT, DECIMAL(38, 0), and both interval types, as an
error or a wrong result.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New tests in `AsOfJoinSortMergeSQLSuite`:
- One overflow test per type (INT, BIGINT, DECIMAL(38, 0), INTERVAL YEAR TO
MONTH, INTERVAL DAY TO SECOND). Each runs `<=` and `<`, with ANSI on and off,
and expects `near`.
- A forward test where the closest row fails an extra `ON` filter, so the
join must skip it and return the next row.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
