Luka Zdravic created SPARK-60839:
------------------------------------
Summary: Use null-safe equality (<=>) as a key in the sort-merge
ASOF join
Key: SPARK-60839
URL: https://issues.apache.org/jira/browse/SPARK-60839
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Luka Zdravic
The sort-merge ASOF join uses only the `=` parts of ON as keys. A `<=>` part
becomes a row-by-row check, because the scanner skips rows with a NULL key. So
an ASOF join whose only key is `<=>`, like `t LEFT ASOF JOIN q MATCH_CONDITION
(t.ts >= q.ts) ON t.sym <=> q.sym`, has no keys and runs both sides in one
partition.
AsOfJoinSelection.extractEquiJoinKeys can do the same as regular joins. These
joins
are split by key and still match NULL to NULL.
Tests: NULL keys on both sides match with `<=>` and do not match with `=`, for
INNER and LEFT, in every direction, and the plan is split by the new keys.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
