kfaraz commented on PR #13339: URL: https://github.com/apache/druid/pull/13339#issuecomment-1319968512
Yeah, #8139 needs some investigation. We should be able to support multiple partitions per task. As you said, 1 is the safest option, but I see your point. We could set it to the partition count, given we have enough worker slots. So, default could be something like `Math.min(1, totalWorkerCapacity/2)`. See `OverlordResource.getTotalWorkerCapacity()` for reference. > Future work: this patch forces a manual restart of the supervisor, but no spec changes are needed, or knowledge of how many partitions your message bus has. In the future we could have a thread that detects a change and automatically restarts the supervisor, but that might be overkill. For this, we already have a `LagBasedAutoscaler`, we could probably add a `LagAndPartitionBasedAutoscaler`? I have not looked into it but I think it should be doable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
