Gustavo de Morais created FLINK-41224:
-----------------------------------------
Summary: Filter out rows with a NULL join key before streaming
inner joins
Key: FLINK-41224
URL: https://issues.apache.org/jira/browse/FLINK-41224
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Reporter: Gustavo de Morais
Since FLINK-41034, streaming joins no longer store rows whose join key can
never match. These rows are still read, serialized and shuffled to the join.
They all share the NULL key, so they all go to the same subtask, which then
processes and discards them.
Batch avoids part of this with {{JoinDeriveNullFilterRule}}, but only when
column statistics report many NULLs, and streaming has no such rule. Calcite's
{{JoinDeriveIsNotNullFilterRule}} adds an {{IS NOT NULL}} filter below an inner
join for every field that the join condition rejects NULLs for. It leaves
null-safe keys ({{IS NOT DISTINCT FROM}}) alone. We should add it to the
streaming rule sets.
The filter changes the operator IDs of pipelines that are not restored from a
compiled plan, so it should be behind an option that is off by default.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
