tomicooler opened a new pull request, #5783: URL: https://github.com/apache/hadoop/pull/5783
### Description of PR The new queue resource calculation logic is capable of calculating the queue resources based on mixed resource types. The queues under one parent can also mix their modes (e.g.: root.queue1=percentage root.queue2=weight root.queue3=absolute). Precedence is: absolute -> percentage -> weight. Note that the calculation is done by a single resource type at once. **Design doc:** [YARN-10888](https://issues.apache.org/jira/browse/YARN-10888) **Example** ``` yarn.scheduler.capacity.root.default.capacity = [memory=1w, vcores=4] yarn.scheduler.capacity.root.test_1.capacity = [memory=16384, vcores=100%] yarn.scheduler.capacity.root.test_2.capacity = [memory=3w, vcores=12] yarn.scheduler.capacity.root.test_1.test_1_1.capacity = [memory=1w, vcores=1w] yarn.scheduler.capacity.root.test_1.test_1_2.capacity = [memory=50%, vcores=2] yarn.scheduler.capacity.root.test_1.test_1_3.capacity = [memory=12288, vcores=86%] ``` Further examples: `hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySchedulerMixedMode.java` The feature is behind a **feature flag**, it is disabled by default. ``` yarn.scheduler.capacity.legacy-queue-mode.enabled = true/false ``` Note that the mixed mode is not yet supported for AQCv2 (or v1) templates. I'll file a Jira ticket for that. ### Key changes 1. The **capacity** and **absoluteCapacity** (and **maxApplications** and other derived properties) of a queue is calculated from the **effectiveMinResource** (= the current available resource for a given queue based on the queue hierarchy configuration and the available cluster resource). 2. When there is no cluster resources available yet, the maxApplications are set to the configured values, so applications can be submitted. 3. Changes in the scheduler-response: the capacity and absoluteCapacity resembles the effective values based on the queue configs and cluster resources. The normalizedWeight is not filled in the scheduler-response when queues are in weight mode. ### How was this patch tested? - I'll run all the unit test in both Legacy and the "Uniform" queue calculation mode in this PR. - I'll create a follow-up ticket to run some e2e like automatically on both mode. ### Acknowledges The current state of this PR was a team effort, thanks to: @9uapaw @brumi1024 @K0K0V0K ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'YARN-11000. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
