Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r142119164
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/QueryPlanConstraints.scala
---
@@ -77,20 +77,21 @@ trait QueryPlanConstraints { self: LogicalPlan =>
constraint match {
// When the root is IsNotNull, we can push IsNotNull through the
child null intolerant
// expressions
- case IsNotNull(expr) =>
scanNullIntolerantAttribute(expr).map(IsNotNull(_))
+ case IsNotNull(expr) =>
scanNullIntolerantField(expr).map(IsNotNull(_))
// Constraints always return true for all the inputs. That means,
null will never be returned.
// Thus, we can infer `IsNotNull(constraint)`, and also push
IsNotNull through the child
// null intolerant expressions.
- case _ => scanNullIntolerantAttribute(constraint).map(IsNotNull(_))
+ case _ => scanNullIntolerantField(constraint).map(IsNotNull(_))
--- End diff --
> I don't believe this is covered in the current unit tests, so I will add
or modify a test to cover it.
I added the following test to cover this case:
https://github.com/apache/spark/blob/88786d3eaf9d3a3d2c80c2235c5014074ade3dc1/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala#L62-L79
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
