Zoltán Borók-Nagy created IMPALA-15441:
------------------------------------------
Summary: Outer join simplification treats non-NULL-propagating
expressions (IN lists, concat_ws, is_nan, nested AND/OR) as NULL-rejecting
Key: IMPALA-15441
URL: https://issues.apache.org/jira/browse/IMPALA-15441
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Zoltán Borók-Nagy
With ENABLE_OUTER_JOIN_TO_INNER_TRANSFORMATION=true,
Analyzer.simplifyOuterJoins()
(IMPALA-5022) decides whether a conjunct rejects the NULLs of an outer-joined
tuple with
isTrueWithNullSlots(). That function replaces every SlotRef with NULL,
including the slots
of the preserved side, and evaluates the conjunct. A conjunct that is not TRUE
this way
is treated as NULL-rejecting.
For conjuncts that also reference the preserved side, this only works if the
expression
propagates NULL: a NULL from the outer-joined side must make the whole conjunct
NULL or
FALSE for any values of the preserved side. isNullableConjunct() excludes the
known
exceptions: OR, IS [NOT] NULL, IS [NOT] DISTINCT FROM, CASE, conditional
functions,
UDFs and non-deterministic functions. It misses these:
* An IN predicate with non-constant values in its list. {{t1.c IN (t2.x, 1)}} is
{{t1.c = t2.x OR t1.c = 1}}, and it is TRUE for {{t1.c = 1}} even when t2 is
null-extended. EqualityDisjunctsToInRule can also produce this form from an OR
written
by the user.
* Functions that return non-NULL for NULL input: concat_ws() skips NULL
arguments, and
is_nan()/is_inf() return FALSE for NULL.
* AND/OR nested inside another expression, e.g. {{(t1.a = 1 AND t2.b = 1) =
false}}. For
the null-extended row with {{t1.a = 0}} this is {{(FALSE AND NULL) = false}},
which is
TRUE. pushNegationToOperands() only handles NOT at the top.
The outer join is then turned into an inner join and rows of the preserved side
are lost.
This only happens when ENABLE_OUTER_JOIN_TO_INNER_TRANSFORMATION is enabled,
which is not
the default. It blocks IMPALA-15430.
h3. Repro
In every query t2 never matches ({{t1.id = t2.id + 100}}), so t2 is
null-extended for all
8 rows of functional.alltypestiny. "off" and "on" are the results with the
option set to
false and to true. The "off" results are correct.
||WHERE clause of {{select count(*) from functional.alltypestiny t1 left join
functional.alltypestiny t2 on t1.id = t2.id + 100}}||off||on||
|{{where t1.int_col in (t2.int_col, 1)}}|4|0|
|{{where concat_ws('-', t1.string_col, t2.string_col) = t1.string_col}}|8|0|
|{{where is_nan(t2.double_col) = (t1.double_col > 0)}}|4|0|
|{{where (t1.int_col = 1 and t2.int_col = 1) = false}}|4|0|
In each case the LEFT OUTER JOIN is planned as an INNER JOIN:
{noformat}
02:HASH JOIN [INNER JOIN]
| hash predicates: t1.id = t2.id + 100
| other predicates: t1.int_col IN (t2.int_col, 1)
{noformat}
h3. Proposed fix
Short term, treat these forms as possibly nullable in isNullableConjunct(): IN
predicates
with any non-constant list element, concat_ws, is_nan, is_inf, and AND/OR below
a non-compound node. More robust: replace the list of exceptions with a
whitelist of
expressions known to propagate NULL (comparisons, arithmetic, casts, and
builtins with
NULL-in/NULL-out semantics). The same helper also drives the predicate
propagation of
IMPALA-11008, so check that too. Add the queries above to
QueryTest/outer-to-inner-joins.test, which runs with the option both off and on.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
