kfaraz commented on code in PR #19646:
URL: https://github.com/apache/druid/pull/19646#discussion_r3505744847
##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -53,27 +53,28 @@ public class WeightedCostFunction
/**
* Minimum rate of processing for any task in records per second. This is
used
* as a placeholder if avg rate is not available to ensure that cost
computations
- * do not return infinitely large lag recovery times.
+ * do not return infinitely large lag recovery times, at the expense of
underestimating the lag cost.
*/
- static final double MIN_PROCESSING_RATE = 1_000;
+ static final double MIN_PROCESSING_RATE = 5_000;
Review Comment:
If we have not received metrics yet, auto-scaling would be skipped since
`CostBasedAutoScaler.validateMetricsForScaling` would return an error.
I have personally seen tasks in prod clusters maxing out at 5000 records/sec
when dealing with large records.
So, using a large `MIN_PROCESSING_RATE` would cause us to always
under-estimate `lagRecoveryTime`, irrespective of the actual
`avgProcessingRate`.
As such, let's keep a low value of `MIN_PROCESSING_RATE`(maybe even as low
as 100), since it is meant to be a safe-side measure that kicks in only when
`avgProcessingRate` is very low.
The penalty for scale-up is already driven by the optimal task idleness, and
can be controlled using the weights.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
