[
https://issues.apache.org/jira/browse/SPARK-60196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolina Vraneš updated SPARK-60196:
------------------------------------
Description:
DataFrame sort-merge ASOF joins fail during planning with DATE and TIMESTAMP
keys. Nearest joins can also select the wrong row when distance arithmetic
overflows or rounds, or when the scan stops at duplicate keys.
h3. Changes
* Support DATE, TIMESTAMP, TIMESTAMP_NTZ and DATE/TIMESTAMP key pairs in
sort-merge planning for all three directions.
* Compare nearest gaps using exact key values to avoid integral overflow,
decimal precision loss and floating-point rounding. TIMESTAMP gaps measure
elapsed time.
* Keep the closest eligible predecessor and stop at the first eligible
successor. Exact matches win when allowed, and equal gaps prefer the
predecessor.
* Read the nearest scanner's keys from the planner's sort expressions,
including swapped and constant ASOF keys.
* Reject mixed TIMESTAMP_LTZ/TIMESTAMP_NTZ key families during analysis when
sort-merge ASOF is enabled, because sorting and comparison can otherwise
disagree around daylight saving time transitions.
h3. Testing
Regression coverage includes numeric precision and overflow, DATE and timestamp
combinations, nanosecond keys, tolerance, excluded exact matches, residual
conditions, spilling, both analyzers and the nearest scan's early stop.
Summary: Fix nearest matching and support DATE/TIMESTAMP keys in
sort-merge ASOF joins (was: Support DATE and TIMESTAMP keys in the sort-merge
as-of join)
> Fix nearest matching and support DATE/TIMESTAMP keys in sort-merge ASOF joins
> -----------------------------------------------------------------------------
>
> Key: SPARK-60196
> URL: https://issues.apache.org/jira/browse/SPARK-60196
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Nikolina Vraneš
> Priority: Major
>
> DataFrame sort-merge ASOF joins fail during planning with DATE and TIMESTAMP
> keys. Nearest joins can also select the wrong row when distance arithmetic
> overflows or rounds, or when the scan stops at duplicate keys.
> h3. Changes
> * Support DATE, TIMESTAMP, TIMESTAMP_NTZ and DATE/TIMESTAMP key pairs in
> sort-merge planning for all three directions.
> * Compare nearest gaps using exact key values to avoid integral overflow,
> decimal precision loss and floating-point rounding. TIMESTAMP gaps measure
> elapsed time.
> * Keep the closest eligible predecessor and stop at the first eligible
> successor. Exact matches win when allowed, and equal gaps prefer the
> predecessor.
> * Read the nearest scanner's keys from the planner's sort expressions,
> including swapped and constant ASOF keys.
> * Reject mixed TIMESTAMP_LTZ/TIMESTAMP_NTZ key families during analysis when
> sort-merge ASOF is enabled, because sorting and comparison can otherwise
> disagree around daylight saving time transitions.
> h3. Testing
> Regression coverage includes numeric precision and overflow, DATE and
> timestamp combinations, nanosecond keys, tolerance, excluded exact matches,
> residual conditions, spilling, both analyzers and the nearest scan's early
> stop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
