Asif created SPARK-55167:
----------------------------
Summary: Bug in Constraints code where dual use of IsNull and
EqualsNullSafe( null), causes a conflict between PushDownPredicates and
InferNewFiltersFrom Constraint, causing loss of idempotency achievement
Key: SPARK-55167
URL: https://issues.apache.org/jira/browse/SPARK-55167
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0
Reporter: Asif
A bug in Constraints code , where a conditon which says attribute X should
always be null, results in a Constraint Creation of the form EqualsNullsafe(X,
null)..
Since this logically equivalent to IsNull(X), but sematically not same, the
rule InferNewFiltersfromConstraint results in creation of a new Filter, despite
the presence of IsNull(X) as an existing Filter.
This behaviour clashes with PushDownPred rule, resulting in a Push/Pull
behaviour causing idempotency to not be achieved.
So far the bug is suppressed because rule InferFiltersFromConstraint is invoked
as ONCE only rule.
But for fixing the bug <>, the rule InferFiltersFromConstraint should be part
of the default rule set running a FIXED POINT iteration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
