kfaraz commented on PR #20708: URL: https://github.com/apache/druid/pull/20708#issuecomment-6394634006
Thanks for the review, @Fly-Style ! > I am in doubt about the new approach (specifically, with the part where the idle is averaged across the whole task life). In my opinion, it over-conserves the task count at the end of task life. I guess (now I just guess, because I have not done any math) will have way worse decisions in the second part of the task life. In the last ~quarter of task life, we may theoretically be even worse at lag reaction, Yes, I did think about this when originally putting up this patch. It is true that if trends start changing towards the end of the `taskDuration` but the metrics do not change enough to affect the average, we would continue with the same task count after rollover. But the next set of tasks will always see the high lag (and a low average value of idle ratio) and the auto-scaler will react to it by scaling up at the next `minScaleUpDelay`. In short, we would have slightly delayed response in favor of smoother scaling. The driving philosophy behind this decision is as follows: - Once a `taskCount` has been chosen, commit to it and let it become stable. - At the end of the scaling cooldown (or `taskDuration` whichever comes first), evaluate the overall performance (hence plain average) of the current set of tasks and then choose the next `taskCount`. This way we look at the whole picture before making hasty scaling decisions. Frequent scaling also causes a lot of fragmentation since each task publishes a separate set of segments and we end up with a lot more segments in each interval. > As a good compromise, I think we can make the length of the moving average to 1/3 of task duration. WDYT? Hmm, I am not sure that I would want to tie it to a fraction of the task duration. Because then we have to choose a good value for that fraction. Instead, a good intuitive alternative would be to reset the running average every time the scaling cooldown elapses. Then every (actionable) scale up or scale down decision is not polluted by old metrics even if `taskDuration` is much longer `minScaleUpDelay` or `minScaleDownDelay`. But implementing that seems more complicated than necessary at the moment. As `costBased` auto-scaler gets further validated and becomes the de-facto choice for scaling, we will revisit this decision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
