Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22357#discussion_r215877729
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruning.scala
---
@@ -196,6 +196,7 @@ private[sql] object ParquetSchemaPruning extends
Rule[LogicalPlan] {
*/
private def getRootFields(expr: Expression): Seq[RootField] = {
expr match {
+ case IsNotNull(_: Attribute) | IsNull(_: Attribute) => Seq.empty
--- End diff --
I mean, for instance, this case `select address is not null, name.last from
contacts` it wouldn't work. I thought this is a quick bandaid fix to resolve a
basic case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
