Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15049#discussion_r147573054
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
---
@@ -555,6 +572,32 @@ class ParquetFilterSuite extends QueryTest with
ParquetTest with SharedSQLContex
}
}
}
+
+ test("Filters should be pushed down for Parquet readers at row group
level") {
--- End diff --
@jiangxb1987, here I added a test to make sure disabling
'spark.sql.parquet.recordFilter' still enables row group level filtering.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
