Liang-Chi Hsieh created SPARK-18130:
---------------------------------------
Summary: Don't add inferred redundant isnotnull condition from
constraints
Key: SPARK-18130
URL: https://issues.apache.org/jira/browse/SPARK-18130
Project: Spark
Issue Type: Improvement
Components: SQL
Reporter: Liang-Chi Hsieh
Priority: Minor
In Optimizer, we will infer additional conditions from Filter's constraints
which include IsNotNull. Some inferred IsNotNulls are useful because they can
be short circuit for other predicates which refer the same attributes. But for
an IsNotNull condition, we also infer other IsNotNull from it. Those IsNotNulls
are redundant. We don't need to include them.
E.g., for a condition like IsNotNull(a + Rand()), an additional IsNotNull(a)
will be added into Filter's condition. So we need to evaluate IsNotNull(a +
Rand()) && IsNotNull(a). Actually we don't need IsNotNull(a) because it is
guaranteed by the original IsNotNull(a + Rand()).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
