[
https://issues.apache.org/jira/browse/SPARK-60582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolina Vraneš updated SPARK-60582:
------------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> Use resolved keys for DataFrame ASOF sort-merge joins
> -----------------------------------------------------
>
> Key: SPARK-60582
> URL: https://issues.apache.org/jira/browse/SPARK-60582
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Nikolina Vraneš
> Priority: Major
>
> DataFrame sort-merge ASOF joins derive their sort keys from subtraction
> expressions. For STRING keys this introduces numeric casts, while the
> matching condition compares strings as text. For example, text ordering puts
> '10' before '9', but numeric ordering reverses them. Non-numeric strings
> become NULL in these numeric sort keys with ANSI off. Sorting and matching
> can therefore disagree.
> Resolve and store DataFrame ASOF keys directly during analysis, sharing SQL
> ASOF validation and comparison-order helpers. This allows backward/forward
> STRING and TIME matching without subtraction, reuses SQL timestamp ordering
> for mixed LTZ/NTZ keys (including composite keys), and preserves explicit
> casts and decimal values used for nearest matching.
> Select execution during analysis and retain that decision through
> optimization and planning. Build subtraction-based distance expressions only
> for the correlated-subquery implementation, eliminating optimizer reanalysis
> and configuration replay. Apply the same resolution and tolerance validation
> in both analyzers, including Connect keys.
> With sort-merge enabled, reject constant keys and incompatible collations.
> Reject nearest STRING keys with either flag value; callers that intend
> numeric distance must cast explicitly. The DataFrame sort-merge flag remains
> off by default.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
