ulysses-you commented on code in PR #57742:
URL: https://github.com/apache/spark/pull/57742#discussion_r3712194204
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4156,6 +4156,66 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ADAPTIVE_PARTIAL_AGGREGATION_ENABLED =
+
buildConf("spark.sql.execution.aggregate.adaptivePartialAggregation.enabled")
+ .doc("When true, hash aggregation adaptively bypasses the pre-shuffle
partial aggregation " +
+ "at runtime when it observes that the partial aggregation is not
reducing the number of " +
+ "rows enough to be worthwhile. Once bypassed, the remaining input rows
are passed " +
+ "through as single-row partial aggregation buffers for the final
aggregation to merge, " +
+ "which avoids the cost of maintaining and spilling a large aggregation
map with little " +
+ "reduction benefit. This applies only to hash aggregation with
grouping keys.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .booleanConf
+ .createWithDefault(true)
+
+ val ADAPTIVE_PARTIAL_AGGREGATION_SAMPLE_ROWS =
+
buildConf("spark.sql.execution.aggregate.adaptivePartialAggregation.sampleRows")
+ .doc("The number of input rows to sample before evaluating the reduction
ratio for the " +
+ s"no-spill tier of adaptive partial aggregation (see " +
+ s"'${ADAPTIVE_PARTIAL_AGGREGATION_ENABLED.key}'). From this many rows
on, if the ratio " +
+ "of distinct grouping keys to processed rows is at least " +
+ s"'spark.sql.execution.aggregate.adaptivePartialAggregation." +
+ "noSpillReductionRatioThreshold', partial aggregation is bypassed for
the rest of the " +
+ "input. When the ratio is below the threshold, the next evaluation
happens after twice " +
+ "as many rows, so low-cardinality input is re-checked only rarely.")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .intConf
+ .checkValue(_ > 0, "The sample row count must be positive.")
+ .createWithDefault(100000)
+
+ val ADAPTIVE_PARTIAL_AGGREGATION_NO_SPILL_REDUCTION_RATIO_THRESHOLD =
Review Comment:
Adding two configs is a trade-off, due to that it is difficult to accurately
determine how much data reduction rate can offset the performance loss caused
by sort and spill. There is a way to use the spill threshold, setting spill
thresholds to 0, so it can always passing through directly when spill happens.
I set both of these configs to the same default value to make their behavior
the same by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
