[
https://issues.apache.org/jira/browse/SPARK-61247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luka Zdravic updated SPARK-61247:
---------------------------------
Summary: ASOF join returns wrong rows or fails with a NullPointerException
when the left ON key is NOT NULL and the right ON key has NULLs (was: ASOF
join returns wrong rows when the left ON key is NOT NULL and the right ON key
is nullable)
> ASOF join returns wrong rows or fails with a NullPointerException when the
> left ON key is NOT NULL and the right ON key has NULLs
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61247
> URL: https://issues.apache.org/jira/browse/SPARK-61247
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Luka Zdravic
> Priority: Minor
>
> SortMergeAsOfJoinScanner orders the ON keys with the nullability of the
> left keys only. When a left key is NOT NULL, the generated ordering skips the
> NULL check, so a NULL right key compares as its empty value: 0 for a number,
> and a null string for a STRING key.
> So a NULL right key can match a left key of 0, a left key can miss its
> match because the scan stops at the NULL row, and STRING keys fail with a
> NullPointerException.
> Example (spark.sql.join.asofJoin.enabled=true):
>
> SELECT t.k, q.tag FROM VALUES (-1, 10), (0, 10) AS t(k, ts)
> LEFT ASOF JOIN VALUES (CAST(NULL AS INT), 1, 'null key'), (-1, 2, 'minus
> one') AS q(k, ts, tag)
> MATCH_CONDITION (t.ts >= q.ts) ON t.k = q.k
>
> Expected: (-1, 'minus one'), (0, NULL)
> Actual: (-1, NULL), (0, 'null key')
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
