Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22357#discussion_r215866501
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala
---
@@ -196,6 +196,7 @@ private[sql] object ParquetSchemaPruning extends
Rule[LogicalPlan] {
*/
private def getRootFields(expr: Expression): Seq[RootField] = {
expr match {
+ case IsNotNull(_: Attribute) | IsNull(_: Attribute) => Seq.empty
--- End diff --
But the case mentioned here looks specific to the pushed filter itself. Can
we add a simple test for project case as well?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
