Nikolina Vraneš created SPARK-59974:
---------------------------------------
Summary: Reduce distance evaluations in nearest ASOF joins
Key: SPARK-59974
URL: https://issues.apache.org/jira/browse/SPARK-59974
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Nikolina Vraneš
Nearest ASOF joins evaluate the distance expression for each eligible row in a
sorted right-side group. For integral and timestamp keys, only the closest
eligible rows on either side of the left key need comparison.
Compare these boundary candidates with exact arithmetic to avoid integer
overflow. Compare timestamps by elapsed UTC time, including full nanosecond
precision. Keep the distance-based scan for fractional distances, where
rounding can affect ties.
The tests cover duplicate keys, overflow with ANSI mode on and off, a daylight
saving transition, nanosecond precision, residual predicates, spills, and
floating-point ties.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
