Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21320#discussion_r197646687
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/QueryPlanConstraints.scala
---
@@ -99,27 +100,28 @@ trait ConstraintHelper {
}
/**
- * Infer the Attribute-specific IsNotNull constraints from the null
intolerant child expressions
- * of constraints.
+ * Infer the Attribute and ExtractValue-specific IsNotNull constraints
from the null intolerant
+ * child expressions of constraints.
*/
private def inferIsNotNullConstraints(constraint: Expression):
Seq[Expression] =
constraint match {
// When the root is IsNotNull, we can push IsNotNull through the
child null intolerant
// expressions
- case IsNotNull(expr) =>
scanNullIntolerantAttribute(expr).map(IsNotNull(_))
+ case IsNotNull(expr) =>
scanNullIntolerantField(expr).map(IsNotNull(_))
// Constraints always return true for all the inputs. That means,
null will never be returned.
// Thus, we can infer `IsNotNull(constraint)`, and also push
IsNotNull through the child
// null intolerant expressions.
- case _ => scanNullIntolerantAttribute(constraint).map(IsNotNull(_))
+ case _ => scanNullIntolerantField(constraint).map(IsNotNull(_))
}
/**
- * Recursively explores the expressions which are null intolerant and
returns all attributes
- * in these expressions.
+ * Recursively explores the expressions which are null intolerant and
returns all attributes and
+ * complex type extractors in these expressions.
*/
- private def scanNullIntolerantAttribute(expr: Expression):
Seq[Attribute] = expr match {
+ private def scanNullIntolerantField(expr: Expression): Seq[Expression] =
expr match {
+ case ev: ExtractValue => Seq(ev)
--- End diff --
Because of this change, we also need to change the code in
basicPhysicalOperators.scala. I do not think this is the right solution. More
importantly, the changes in basicPhysicalOperators.scala might break the
others. We need a separate PR for these changes. Please remove the changes made
in basicPhysicalOperators.scala and QueryPlanConstraints.scala
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
