capistrant commented on code in PR #18958:
URL: https://github.com/apache/druid/pull/18958#discussion_r2732930750
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScaler.java:
##########
@@ -157,11 +164,17 @@ public int computeTaskCountForScaleAction()
final int optimalTaskCount = computeOptimalTaskCount(lastKnownMetrics);
final int currentTaskCount = lastKnownMetrics.getCurrentTaskCount();
- // Perform only scale-up actions
+ // Perform scale-up actions; scale-down actions only if configured.
int taskCount = -1;
if (optimalTaskCount > currentTaskCount) {
taskCount = optimalTaskCount;
- log.info("New task count [%d] on supervisor [%s]", taskCount,
supervisorId);
+ scaleDownCounter = 0; // Nullify the scaleDown counter after a
successful scaleup too.
+ log.info("New task count [%d] on supervisor [%s], scaling up",
taskCount, supervisorId);
+ } else if (!config.isScaleDownOnTaskRolloverOnly() && optimalTaskCount <
currentTaskCount &&
+ (config.getScaleDownBarrier() == 0 || ++scaleDownCounter ==
config.getScaleDownBarrier())) {
Review Comment:
nit suggestion: might be easier cognitively to drop the `||`?
```suggestion
} else if (!config.isScaleDownOnTaskRolloverOnly() && optimalTaskCount <
currentTaskCount && ++scaleDownCounter >= config.getScaleDownBarrier()) {
```
##########
docs/ingestion/supervisor.md:
##########
@@ -208,6 +208,8 @@ The following table outlines the configuration properties
related to the `costBa
|`lagWeight`|The weight of extracted lag value in cost function.| No| 0.25|
|`idleWeight`|The weight of extracted poll idle value in cost function. | No |
0.75 |
|`defaultProcessingRate`|A planned processing rate per task, required for
first cost estimations. | No | 1000 |
+|`scaleDownBarrier`| A number of successful scale down attempts which should
be skipped to prevent the auto-scaler from scaling down tasks immediately. |
No | 5 |
+|`scaleDownDuringTaskRolloverOnly`| Indicates whether task scaling down is
limited to periods during task rollovers only. | No | False |
Review Comment:
Is it possible that we want this to be `true` as default, if that is like
the production behavior we think will be preferred? I say `if that is like the
production behavior we think will be preferred` because of this in the
description:
> To test the cost-based autoscaler in real-world scenarios, we temporarily
allow scale-downs during task runtime
like instead of temporarily defaulting for testing, default for long term
productionalization plans and we can override for folks testing behavior more
rapidly?
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScalerConfig.java:
##########
@@ -43,11 +43,12 @@
@JsonInclude(JsonInclude.Include.NON_NULL)
public class CostBasedAutoScalerConfig implements AutoScalerConfig
{
- private static final long DEFAULT_SCALE_ACTION_PERIOD_MILLIS = 15 * 60 *
1000; // 15 minutes
- private static final long DEFAULT_MIN_TRIGGER_SCALE_ACTION_FREQUENCY_MILLIS
= 1200000; // 20 minutes
+ private static final long DEFAULT_SCALE_ACTION_PERIOD_MILLIS = 10 * 60 *
1000; // 10 minutes
Review Comment:
looks like this caused the default serde test to fail. I assume the one
below would also fail assertions if not fixed up as well.
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/CostBasedAutoScalerConfig.java:
##########
@@ -90,6 +105,8 @@ public CostBasedAutoScalerConfig(
this.lagWeight = Configs.valueOrDefault(lagWeight, DEFAULT_LAG_WEIGHT);
this.idleWeight = Configs.valueOrDefault(idleWeight, DEFAULT_IDLE_WEIGHT);
this.defaultProcessingRate = Configs.valueOrDefault(defaultProcessingRate,
DEFAULT_PROCESSING_RATE);
+ this.scaleDownBarrier = Configs.valueOrDefault(scaleDownBarrier,
DEFAULT_SCALE_DOWN_BARRIER);
+ this.scaleDownDuringTaskRolloverOnly =
Configs.valueOrDefault(scaleDownDuringTaskRolloverOnly, false);
Review Comment:
nit: should this follow pattern of default at top of file or are we only
doing that for magic numbers with context added in comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
