Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21603#discussion_r197382691
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -378,6 +378,17 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val PARQUET_FILTER_PUSHDOWN_INFILTERTHRESHOLD =
+ buildConf("spark.sql.parquet.pushdown.inFilterThreshold")
+ .doc("The maximum number of values to filter push-down optimization
for IN predicate. " +
+ "Large threshold will not provide much better performance. " +
+ "This configuration only has an effect when
'spark.sql.parquet.filterPushdown' is enabled.")
+ .internal()
+ .intConf
+ .checkValue(threshold => threshold > 0 && threshold <= 300,
--- End diff --
I also think to set a maximum value 300 is a bit weird.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
