Nikolina Vraneš created SPARK-60582:
---------------------------------------
Summary: Use resolved keys for DataFrame ASOF sort-merge joins
Key: SPARK-60582
URL: https://issues.apache.org/jira/browse/SPARK-60582
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Nikolina Vraneš
DataFrame sort-merge ASOF joins derive their sort keys from subtraction
expressions. For STRING keys this introduces numeric casts, while the matching
condition compares strings as text. For example, text ordering puts '10' before
'9', but numeric ordering reverses them. Non-numeric strings become NULL in
these numeric sort keys with ANSI off. Sorting and matching can therefore
disagree.
Resolve and store DataFrame ASOF keys directly during analysis, sharing SQL
ASOF validation and comparison-order helpers. This allows backward/forward
STRING and TIME matching without subtraction, reuses SQL timestamp ordering for
mixed LTZ/NTZ keys (including composite keys), and preserves explicit casts and
decimal values used for nearest matching.
Select execution during analysis and retain that decision through optimization
and planning. Build subtraction-based distance expressions only for the
correlated-subquery implementation, eliminating optimizer reanalysis and
configuration replay. Apply the same resolution and tolerance validation in
both analyzers, including Connect keys.
With sort-merge enabled, reject constant keys and incompatible collations.
Reject nearest STRING keys with either flag value; callers that intend numeric
distance must cast explicitly. The DataFrame sort-merge flag remains off by
default.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
