[
https://issues.apache.org/jira/browse/SPARK-18130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610410#comment-15610410
]
Apache Spark commented on SPARK-18130:
--------------------------------------
User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/15653
> Don't add inferred redundant isnotnull condition from constraints
> -----------------------------------------------------------------
>
> Key: SPARK-18130
> URL: https://issues.apache.org/jira/browse/SPARK-18130
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Liang-Chi Hsieh
> Priority: Minor
>
> In Optimizer, we will infer additional conditions from Filter's constraints
> which include IsNotNull. Some inferred IsNotNulls are useful because they can
> be short circuit for other predicates which refer the same attributes. But
> for an IsNotNull condition, we also infer other IsNotNull from it. Those
> IsNotNulls are redundant. We don't need to include them.
> E.g., for a condition like IsNotNull(a + Rand()), an additional IsNotNull(a)
> will be added into Filter's condition. So we need to evaluate IsNotNull(a +
> Rand()) && IsNotNull(a). Actually we don't need IsNotNull(a) because it is
> guaranteed by the original IsNotNull(a + Rand()).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
