FrankChen021 commented on code in PR #19687:
URL: https://github.com/apache/druid/pull/19687#discussion_r3650319187
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorManager.java:
##########
@@ -644,6 +648,80 @@ public boolean isAnotherTaskGroupPublishingToPartitions(
}
}
+ /**
+ * Simulates the effects of the {@code costBased} auto-scaler by computing
the optimal
+ * task count under various values of aggregate lag.
+ */
+ public Map<String, Object> simulateAutoscaling(
+ String supervisorId,
+ CostBasedAutoScalerConfig config,
+ int criticalLag,
+ int maxProcessingRatePerTask,
+ @Nullable Integer requestedTaskCount
+ )
+ {
+ // Validate that this is a streaming supervisor
+ final StreamSupervisor supervisor = requireStreamSupervisor(supervisorId,
"simulateAutoscaling");
+
+ // Validate the inputs
+ InvalidInput.conditionalException(
+ criticalLag >= 1000,
+ "Value of critical lag[%d] must be 1000 or more",
+ criticalLag
+ );
+ InvalidInput.conditionalException(
+ maxProcessingRatePerTask >= 100,
+ "Value of maxProcessingRatePerTask[%d] must be 100 events per second
or more",
+ maxProcessingRatePerTask
+ );
+
+ // Simulate from the supervisor's live task count unless the caller pins
one.
+ final int currentTaskCount = Configs.valueOrDefault(
+ requestedTaskCount,
+ ((SeekableStreamSupervisor<?, ?, ?>)
supervisor).getIoConfig().getTaskCount()
+ );
+ InvalidInput.conditionalException(
+ requestedTaskCount == null
+ || (currentTaskCount >= config.getTaskCountMin() && currentTaskCount
<= config.getTaskCountMax()),
+ "Value of currentTaskCount[%d] must be within taskCountMin[%d] and
taskCountMax[%d]",
+ currentTaskCount, config.getTaskCountMin(), config.getTaskCountMax()
+ );
+ final int simulationTaskCount = Math.max(
+ config.getTaskCountMin(),
+ Math.min(currentTaskCount, config.getTaskCountMax())
+ );
+
+ // Assumption: enough partitions to reach taskCountMax.
+ final int partitionCount = config.getTaskCountMax();
Review Comment:
These values are not currently user-controlled: the panel/API do not expose
partition count or task duration; they derive partition count from
`taskCountMax` and fix duration at 3600 seconds. Thus a two-partition
supervisor can still show impossible recommendations up to 10 tasks, and
non-hour task durations produce a different curve. Please either read both from
the selected supervisor, or expose them as explicit simulator inputs and label
the assumptions.
Reviewed 11 of 11 changed files.
##########
indexing-service/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorManager.java:
##########
@@ -644,6 +651,90 @@ public boolean isAnotherTaskGroupPublishingToPartitions(
}
}
+ /**
+ * Simulates the effects of the {@code costBased} auto-scaler by computing
the optimal
+ * task count under various values of aggregate lag.
+ */
+ public Map<String, Object> simulateAutoscaling(
+ String supervisorId,
+ CostBasedAutoScalerConfig config,
+ int criticalLag,
+ int maxProcessingRatePerTask,
+ @Nullable Integer requestedTaskCount
+ )
+ {
+ // Validate that this is a streaming supervisor
+ final StreamSupervisor supervisor = requireStreamSupervisor(supervisorId,
"simulateAutoscaling");
+
+ // Validate the inputs
+ InvalidInput.conditionalException(
+ criticalLag >= 1000,
+ "Value of critical lag[%d] must be 1000 or more",
+ criticalLag
+ );
+ InvalidInput.conditionalException(
+ maxProcessingRatePerTask >= 100,
+ "Value of maxProcessingRatePerTask[%d] must be 100 events per second
or more",
+ maxProcessingRatePerTask
+ );
+ InvalidInput.conditionalException(
+ config.getTaskCountMin() >= 1,
+ "Value of taskCountMin[%d] must be 1 or more",
+ config.getTaskCountMin()
+ );
+ InvalidInput.conditionalException(
+ config.getTaskCountMax() <= MAX_SIMULATION_TASK_COUNT,
+ "Value of taskCountMax[%d] must be [%d] or less",
+ config.getTaskCountMax(), MAX_SIMULATION_TASK_COUNT
+ );
+
+ // Simulate from the supervisor's live task count unless the caller pins
one.
+ final int currentTaskCount = Configs.valueOrDefault(
+ requestedTaskCount,
+ ((SeekableStreamSupervisor<?, ?, ?>)
supervisor).getIoConfig().getTaskCount()
+ );
+ InvalidInput.conditionalException(
+ requestedTaskCount == null
+ || (currentTaskCount >= config.getTaskCountMin() && currentTaskCount
<= config.getTaskCountMax()),
+ "Value of currentTaskCount[%d] must be within taskCountMin[%d] and
taskCountMax[%d]",
+ currentTaskCount, config.getTaskCountMin(), config.getTaskCountMax()
+ );
+ final int simulationTaskCount = Math.max(
+ config.getTaskCountMin(),
+ Math.min(currentTaskCount, config.getTaskCountMax())
+ );
+
+ // Assumption: enough partitions to reach taskCountMax.
+ final int partitionCount = config.getTaskCountMax();
+ final int taskDurationSeconds = 3600;
+
+ // Assume that the tasks are fully used since there is some lag
+ final double avgProcessingRatePerTask = maxProcessingRatePerTask;
+ final double idleRatio = config.getOptimalTaskIdleRatio();
+
+ // Invoke the cost function for a variety of input values of lag
+ final Object[] rows = new Object[40];
+ final int lagStepSize = criticalLag / 20;
+ final CostBasedAutoScaler autoscaleSimulator =
CostBasedAutoScaler.createSimulator(config, supervisorId);
+ for (int i = 0; i < 40; ++i) {
+ final double observedAggregateLag = (double) lagStepSize * i;
+ final CostMetrics costMetrics = new CostMetrics(
+ observedAggregateLag / partitionCount,
+ simulationTaskCount,
+ partitionCount,
+ idleRatio,
+ taskDurationSeconds,
+ avgProcessingRatePerTask,
+ maxProcessingRatePerTask * 1.0
+ );
+ final int optimalTaskCount =
autoscaleSimulator.computeOptimalTaskCount(costMetrics);
Review Comment:
[P2] Suppress production logs for simulation samples
`computeOptimalTaskCount` emits an INFO metrics line on every call and often
a second candidate-table INFO line. This loop calls it 40 times per API
request, while the panel issues requests after debounced edits, so ordinary
simulator use floods Overlord logs with at least 40 entries per edit. Gate
these logs for simulator instances as metrics already are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
