[
https://issues.apache.org/jira/browse/SPARK-60196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60196:
-----------------------------------
Labels: pull-request-available (was: )
> Fix nearest matching and support DATE/TIMESTAMP keys in sort-merge ASOF joins
> -----------------------------------------------------------------------------
>
> Key: SPARK-60196
> URL: https://issues.apache.org/jira/browse/SPARK-60196
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Nikolina Vraneš
> Priority: Major
> Labels: pull-request-available
>
> DataFrame sort-merge ASOF joins fail during planning with DATE and TIMESTAMP
> keys. Nearest joins can also select the wrong row when distance arithmetic
> overflows or rounds, or when the scan stops at duplicate keys.
> h3. Changes
> * Support DATE, TIMESTAMP, TIMESTAMP_NTZ and DATE/TIMESTAMP key pairs in
> sort-merge planning for all three directions.
> * Compare nearest gaps using exact key values to avoid integral overflow,
> decimal precision loss and floating-point rounding. TIMESTAMP gaps measure
> elapsed time.
> * Keep the closest eligible predecessor and stop at the first eligible
> successor. Exact matches win when allowed, and equal gaps prefer the
> predecessor.
> * Read the nearest scanner's keys from the planner's sort expressions,
> including swapped and constant ASOF keys.
> * Reject mixed TIMESTAMP_LTZ/TIMESTAMP_NTZ key families during analysis when
> sort-merge ASOF is enabled, because sorting and comparison can otherwise
> disagree around daylight saving time transitions.
> h3. Testing
> Regression coverage includes numeric precision and overflow, DATE and
> timestamp combinations, nanosecond keys, tolerance, excluded exact matches,
> residual conditions, spilling, both analyzers and the nearest scan's early
> stop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
