maropu commented on a change in pull request #28424:
URL: https://github.com/apache/spark/pull/28424#discussion_r419018496
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1682,6 +1682,22 @@ object SQLConf {
.doubleConf
.createWithDefault(0.9)
+ val OPTIMIZE_INTERSECT_ENABLED =
+ buildConf("spark.sql.cbo.optimizeIntersect.enabled")
+ .internal()
+ .doc("Whether to use optimized Intersect implementation or not. " +
+ "Optimized Intersect logic tries to pushdown Distinct through Join
based on some stats")
+ .booleanConf
+ .createWithDefault(true)
+
+ val OPTIMIZE_INTERSECT_DISTINCT_REDUCTION_THRESHOLD =
+ buildConf("spark.sql.cbo.optimizeIntersect.distinctReductionThreshold")
+ .internal()
+ .doc("Ratio by which Distinct should reduce number of rows to qualify
for pushdown" +
+ " in Intersect optimization")
+ .intConf
+ .createWithDefault(100)
Review comment:
Yea, if we have the negative impact.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
