Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21295#discussion_r190076831
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala
---
@@ -879,6 +879,18 @@ class ParquetQuerySuite extends QueryTest with
ParquetTest with SharedSQLContext
}
}
}
+
+ test("SPARK-24230: filter row group using dictionary") {
+ withSQLConf(("parquet.filter.dictionary.enabled", "true")) {
--- End diff --
We are unable to pass the parquet-specific parameter through `withSQLConf`.
Below shows the way to pass the parquet option.
```Scala
withTable("t1") {
spark.createDataFrame((0 until 100).map(i => ((i * 2) % 20,
s"data-$i"))).write
.option("parquet.filter.dictionary.enabled",
false).saveAsTable("t1")
checkAnswer(sql("SELECT _2 FROM t1 WHERE t1._1 = 5"), Seq.empty)
}
```
Could you help investigate why we still hit the error [without the fix]
when we set `parquet.filter.dictionary.enabled` to `false`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
