VladimirIordanov opened a new issue #8695: Support different types of MiddleManagers by Autoscaler URL: https://github.com/apache/incubator-druid/issues/8695 ### Motivation Since #7066 Druid supports different types of tasks which run on dedicated instance of MiddleManager. But Overlord still can spawn only one instance of MiddleManager via Autoscaler. That is not flexible when Druid should support different types of tasks and cluster parameter for each of the task is different. For instance realtime and batch tasks will require clusters of different parameters (workers count; CPU, memory, per each worker, etc.). In that case overlord could provision dedicated cluster per each task type. ### Proposed changes Change `DefaultWorkerBehaviorConfig` to support declaration of several autoscalers. Each autoscaler should have "tier" property which should link it to according "type" of `Task`. Change all current implementations of `ProvisioningStrategy` interface to select appropriate `AutoScaler` from `DefaultWorkerBehaviorConfig`. In that way overlord can spawn a corresponding cluster to submitted task. If there is no matched autoscaler found in the config then the first definition will be used as a default one. ### Rationale Currently DefaultWorkerBehaviorConfig has such format: ``` { "selectStrategy": { "type": "fillCapacity", ... }, "autoScaler": { "type": "ec2", ... } } ``` I propose to replace "autoScaler" section with "autoScalers". The new section will contain definitions of all supported autoscalers in key/value format: ``` { "selectStrategy": { "type": "fillCapacity", ... }, "autoScalers": { "_tier_1_": { "type": "ec2", ... }, "_tier_2_": { "type": "ec2", ... } }, ... } ``` Where "_tier_1_" and "_tier_2_" are tier names according to supported types of tasks (and middleManagers). ### Operational impact In scope of this proposal the format of DefaultWorkerBehaviorConfig will be changed significantly. This change will not be backward compatible with the previous versions of Druid. It should not impact the rolling update.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
