[
https://issues.apache.org/jira/browse/SPARK-55167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asif resolved SPARK-55167.
--------------------------
Resolution: Not A Bug
> Bug in Constraints code where dual use of IsNull and EqualsNullSafe( X,
> null), causes a conflict between PushDownPredicates and InferNewFiltersFrom
> Constraint, causing loss of idempotency achievement
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-55167
> URL: https://issues.apache.org/jira/browse/SPARK-55167
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.1.1
> Reporter: Asif
> Priority: Major
>
> A bug in Constraints code , where a conditon which says attribute X should
> always be null, results in a Constraint Creation of the form
> EqualsNullsafe(X, null)..
> Since this logically equivalent to IsNull(X), but sematically not same, the
> rule InferNewFiltersfromConstraint results in creation of a new Filter,
> despite the presence of IsNull(X) as an existing Filter.
> This behaviour clashes with PushDownPred rule, resulting in a Push/Pull
> behaviour causing idempotency to not be achieved.
> So far the bug is suppressed because rule InferFiltersFromConstraint is
> invoked as ONCE only rule.
> But for fixing the bug <>, the rule InferFiltersFromConstraint should be part
> of the default rule set running a FIXED POINT iteration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
