Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21295#discussion_r190251186
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala
---
@@ -879,6 +879,18 @@ class ParquetQuerySuite extends QueryTest with
ParquetTest with SharedSQLContext
}
}
}
+
+ test("SPARK-24230: filter row group using dictionary") {
+ withSQLConf(("parquet.filter.dictionary.enabled", "true")) {
--- End diff --
using SQLConf is OK, Spark will put all SQL configs to Hadoop conf, which
will be accessed by parquet writer at the executor side.
I'm also curious about why turning `parquet.filter.dictionary.enabled` off
can't avoid this bug.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
