[
https://issues.apache.org/jira/browse/SPARK-59829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić resolved SPARK-59829.
----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59102
[https://github.com/apache/spark/pull/59102]
> Fix overflow in forward ASOF join by keeping the first match
> ------------------------------------------------------------
>
> Key: SPARK-59829
> URL: https://issues.apache.org/jira/browse/SPARK-59829
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Luka Zdravic
> Assignee: Luka Zdravic
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Forward ASOF joins ({{t.k <= r.k}} or {{t.k < r.k}}) in
> {{SortMergeAsOfJoinExec}} compute {{right - left}} for every candidate row to
> find the closest one. This overflows when the values are far apart.
> {code:sql}
> SET spark.sql.join.asofJoin.enabled=true;
> SELECT r.tag
> FROM VALUES (-100) AS t(k)
> ASOF JOIN VALUES (-50, 'near'), (2147483647, 'far') AS r(k, tag)
> MATCH_CONDITION (t.k <= r.k);
> {code}
>
> * ANSI on: fails with ARITHMETIC_OVERFLOW.
> * ANSI off: the subtraction wraps to a negative number, so it returns
> {{far}}.
> * Expected: {{near}}.
>
> The same bug hits BIGINT, DECIMAL(38, 0), and both interval types, as an
> error or a wrong result.
>
> The subtraction is not needed. The right rows are sorted ascending, so the
> first matching row is always the closest. Backward joins already work this
> way: they keep the last match and never subtract. The fix makes Forward keep
> the first match. Nearest keeps the distance loop, because it needs it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
