gengliangwang commented on a change in pull request #29642:
URL: https://github.com/apache/spark/pull/29642#discussion_r528452609
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -704,8 +704,8 @@ object SQLConf {
val PARQUET_FILTER_PUSHDOWN_INFILTERTHRESHOLD =
buildConf("spark.sql.parquet.pushdown.inFilterThreshold")
.doc("The maximum number of values to filter push-down optimization for
IN predicate. " +
- "Large threshold won't necessarily provide much better performance. " +
- "The experiment argued that 300 is the limit threshold. " +
+ "Spark will push-down a value greater than or equal to its minimum
value and " +
Review comment:
I think the default value `10` is too smaller here. What is the default
threshold in IMPLA?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
