[
https://issues.apache.org/jira/browse/SPARK-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust resolved SPARK-13995.
--------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Issue resolved by pull request 11809
[https://github.com/apache/spark/pull/11809]
> Extract correct IsNotNull constraints for Expression
> ----------------------------------------------------
>
> Key: SPARK-13995
> URL: https://issues.apache.org/jira/browse/SPARK-13995
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Liang-Chi Hsieh
> Fix For: 2.0.0
>
>
> We infer relative `IsNotNull` constraints from logical plan's expressions in
> `constructIsNotNullConstraints` now. However, we don't consider the case of
> (nested) `Cast`.
> For example:
> val tr = LocalRelation('a.int, 'b.long)
> val plan = tr.where('a.attr === 'b.attr).analyze
> Then, the plan's constraints will have `IsNotNull(Cast(resolveColumn(tr,
> "a"), LongType))`, instead of `IsNotNull(resolveColumn(tr, "a"))`. This PR
> fixes it.
> Besides, as `IsNotNull` constraints are most useful for `Attribute`, we
> should do recursing through any `Expression` that is null intolerant and
> construct `IsNotNull` constraints for all `Attribute`s under these
> Expressions.
> For example, consider the following constraints:
> val df = Seq((1,2,3)).toDF("a", "b", "c")
> df.where("a + b = c").queryExecution.analyzed.constraints
> The inferred isnotnull constraints should be isnotnull(a), isnotnull(b),
> isnotnull(c), instead of isnotnull(a + c) and isnotnull(c).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
