Xianqing He created IMPALA-15604:
------------------------------------
Summary: SimplifyDistinctFromRule produces wrong results with
nondeterministic expressions
Key: IMPALA-15604
URL: https://issues.apache.org/jira/browse/IMPALA-15604
Project: IMPALA
Issue Type: Bug
Components: Frontend
Affects Versions: Impala 5.0.0
Reporter: Xianqing He
h3. Problem
{{SimplifyDistinctFromRule}} rewrites {{x IS DISTINCT FROM x}} to {{false}} and
{{x IS NOT DISTINCT FROM x}} to {{true}} whenever the two operands are
structurally equal. {{Expr.equals()}} only compares the expression tree, so
calls to nondeterministic functions such as {{rand()}}, {{random()}}, and
{{uuid()}} are considered equal even though separate invocations can return
different values. The rewrite therefore changes query results.
h3. Reproducer
{code:sql}
-- With expr rewrites disabled, the two uuid() calls return different values.
SET enable_expr_rewrites=false;
SELECT uuid() IS DISTINCT FROM uuid(); -- true
SELECT nullif(uuid(), uuid()); -- non-NULL UUID
SET enable_expr_rewrites=true;
SELECT uuid() IS DISTINCT FROM uuid(); -- false (incorrect)
SELECT nullif(uuid(), uuid()); -- NULL (incorrect)
{code}
{{nullif(x, y)}} is converted to {{IF(x IS DISTINCT FROM y, x, NULL)}}, so the
same issue also affects conditional expressions. Similar behavior can occur
with {{rand()}}/{{random()}} when the expression is evaluated more than once
(see IMPALA-14146).
h3. Proposed fix
* Skip the rewrite when either operand contains a nondeterministic expression.
The check must recurse through the whole expression tree, not just direct
children.
* Cover builtin nondeterministic functions ({{rand}}, {{random}}, {{uuid}}) and
conservatively treat UDFs as nondeterministic because Impala has no determinism
metadata.
* Prefer the existing helpers, for example
{{expr.contains(Predicates.or(Expr.IS_NONDETERMINISTIC_BUILTIN_FN_PREDICATE,
Expr.IS_UDF_PREDICATE))}}.
h3. Testing
* Add regression tests in {{ExprRewriteRulesTest}} for {{uuid() IS DISTINCT
FROM uuid()}}, {{uuid() IS NOT DISTINCT FROM uuid()}}, and {{nullif(uuid(),
uuid())}}.
* Add a nested case such as {{length(uuid()) IS DISTINCT FROM length(uuid())}}
or {{abs(rand()) <=> abs(rand())}} to verify the check is recursive.
* Verify existing deterministic simplifications such as {{bool_col IS DISTINCT
FROM bool_col}} still fire.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
