Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21603#discussion_r202255923
  
    --- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
    @@ -386,6 +386,17 @@ object SQLConf {
         .booleanConf
         .createWithDefault(true)
     
    +  val PARQUET_FILTER_PUSHDOWN_INFILTERTHRESHOLD =
    +    buildConf("spark.sql.parquet.pushdown.inFilterThreshold")
    +      .doc("The maximum number of values to filter push-down optimization 
for IN predicate. " +
    +        "Large threshold won't necessarily provide much better 
performance. " +
    +        "The experiment argued that 300 is the limit threshold. " +
    +        "This configuration only has an effect when 
'spark.sql.parquet.filterPushdown' is enabled.")
    +      .internal()
    +      .intConf
    +      .checkValue(threshold => threshold > 0, "The threshold must be 
greater than 0.")
    --- End diff --
    
    Shell we allow, for example, `-1` here to disable this?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to