Luka Zdravic created SPARK-61247:
------------------------------------
Summary: ASOF join returns wrong rows when the left ON key is NOT
NULL and the right ON key is nullable
Key: SPARK-61247
URL: https://issues.apache.org/jira/browse/SPARK-61247
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.4.0
Reporter: Luka Zdravic
SortMergeAsOfJoinScanner orders the ON keys with the nullability of the left
keys only. When a left key is NOT NULL, the generated ordering skips the NULL
check, so a NULL right key compares as its empty value: 0 for a number, and a
null string for a STRING key.
So a NULL right key can match a left key of 0, a left key can miss its match
because the scan stops at the NULL row, and STRING keys fail with a
NullPointerException.
Example (spark.sql.join.asofJoin.enabled=true):
SELECT t.k, q.tag FROM VALUES (-1, 10), (0, 10) AS t(k, ts)
LEFT ASOF JOIN VALUES (CAST(NULL AS INT), 1, 'null key'), (-1, 2, 'minus
one') AS q(k, ts, tag)
MATCH_CONDITION (t.ts >= q.ts) ON t.k = q.k
Expected: (-1, 'minus one'), (0, NULL)
Actual: (-1, NULL), (0, 'null key')
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
