uros-b commented on code in PR #56777:
URL: https://github.com/apache/spark/pull/56777#discussion_r3476311153
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3869,6 +3869,17 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val BYPASS_PARTIAL_AGGREGATION =
buildConf("spark.sql.execution.bypassPartialAggregation")
+ .doc("When true, skips the pre-shuffle partial aggregation and runs a
single Complete-mode " +
+ "aggregation after the shuffle. Bypassing partial aggregation can
improve performance " +
+ "when group cardinality is high and the pre-shuffle reduction ratio is
low. " +
+ "When false (default), uses a two-phase Partial+Final aggregation across
a shuffle. " +
+ "This setting has no effect on queries containing DISTINCT aggregate
functions, where " +
+ "the partial aggregation phases are required for correctness and are
always applied.")
+ .version("3.3.1")
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
Should this be internal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
