Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22357#discussion_r216256434
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala
---
@@ -199,6 +209,15 @@ private[sql] object ParquetSchemaPruning extends
Rule[LogicalPlan] {
case att: Attribute =>
RootField(StructField(att.name, att.dataType, att.nullable),
derivedFromAtt = true) :: Nil
case SelectedField(field) => RootField(field, derivedFromAtt =
false) :: Nil
+ // Root field accesses by `IsNotNull` and `IsNull` are special cases
as the expressions
+ // don't actually use any nested fields. These root field accesses
might be excluded later
+ // if there are any nested fields accesses in the query plan.
+ case IsNotNull(SelectedField(field)) =>
+ RootField(field, derivedFromAtt = false, contentAccessed = false)
:: Nil
+ case IsNull(SelectedField(field)) =>
+ RootField(field, derivedFromAtt = false, contentAccessed = false)
:: Nil
--- End diff --
@dbtsai The question you mentioned at
https://github.com/apache/spark/pull/22357/files#r216204022 was addressed by
this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
