Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/21295#discussion_r190377736
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala
---
@@ -879,6 +879,18 @@ class ParquetQuerySuite extends QueryTest with
ParquetTest with SharedSQLContext
}
}
}
+
+ test("SPARK-24230: filter row group using dictionary") {
+ withSQLConf(("parquet.filter.dictionary.enabled", "true")) {
--- End diff --
Looks like the problem is a bug in Parquet. It is using the [stats property
instead of the dictionary
property](https://github.com/apache/parquet-mr/blob/8bbc6cb95fd9b4b9e86c924ca1e40fd555ecac1d/parquet-hadoop/src/main/java/org/apache/parquet/HadoopReadOptions.java#L83).
This is minor because there is almost no reason to turn either one off now
that we've built more confidence in the filters.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
