kfaraz commented on PR #19549: URL: https://github.com/apache/druid/pull/19549#issuecomment-4628613612
@Fly-Style , IIUC, the partition assignment is all-or-nothing by design. Could you please elaborate on the benefits of the intermediate scaling? e.g. in the case of 400 partitions, what would be the benefit of scaling from 200 tasks to say 300 tasks (instead of directly to 400)? With 300 tasks, 100 tasks would be working on 2 partitions each and 200 tasks would be working on 1 partition each (and we cannot control which partitions get assigned to which tasks). So, the tasks working on 2 partitions have effectively not changed and would continue to be the ingestion bottleneck. Please note that in a non-uniform partition assignment, the idleness cost computation would be a little more involved, since the tasks working on fewer partitions would be more idle than others on average. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
