Gustavo de Morais created FLINK-41223:
-----------------------------------------
Summary: Batch joins drop NULL-safe matches on IS NOT DISTINCT
FROM keys when column statistics report many NULLs
Key: FLINK-41223
URL: https://issues.apache.org/jira/browse/FLINK-41223
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 2.4.0
Reporter: Gustavo de Morais
A batch join on {{IS NOT DISTINCT FROM}} should match NULL with NULL. It
doesn't when the column statistics report more than 2,000,000 NULLs for the
join key.
{code:sql}
-- t1 and t2 both contain (1, 10) and (NULL, 20)
-- the column statistics of t1.a and t2.a report nullCount = 3000000
SELECT t1.a, t1.b, t2.b FROM t1 JOIN t2 ON t1.a IS NOT DISTINCT FROM t2.a;
{code}
Expected: (1, 10, 10) and (NULL, 20, 20). This is also what we get without
statistics.
Actual: only (1, 10, 10).
{{JoinDeriveNullFilterRule}} adds {{IS NOT NULL}} below the join for keys with
many NULLs. It also does that for {{IS NOT DISTINCT FROM}} keys, which removes
the NULL rows that should match. We should skip null-safe keys, for example by
using {{JoinInfo#createWithStrictEquality}} instead of
{{Join#analyzeCondition}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
