Hi All, We would like to bring up the discussion of merging “absolute min/max resources support in capacity scheduler” branch (YARN-5881) [2] into trunk in a few weeks. The goal is to get it in for Hadoop 3.1.
*Major work happened in this branch* - YARN-6471. Support to add min/max resource configuration for a queue - YARN-7332. Compute effectiveCapacity per each resource vector - YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources. *Regarding design details* Please refer [1] for detailed design document. *Regarding to testing:* We did extensive tests for the feature in the last couple of months. Comparing to latest trunk. - For SLS benchmark: We didn't see observable performance gap from simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+ containers allocated per second. - For microbenchmark: We use performance test cases added by YARN 6775, it did not show much performance regression comparing to trunk. *YARN-5881* <https://issues.apache.org/jira/browse/YARN-5881> #ResourceTypes = 2. Avg of fastest 20: 55294.52 #ResourceTypes = 2. Avg of fastest 20: 55401.66 *trunk* #ResourceTypes = 2. Avg of fastest 20: 55865.92 #ResourceTypes = 2. Avg of fastest 20: 55096.418 *Regarding to API stability:* All newly added @Public APIs are @Unstable. Documentation jira [3] could help to provide detailed configuration details. This feature works from end-to-end and we are running this in our development cluster for last couple of months and undergone good amount of testing. Branch code is run against trunk and tracked via [4]. We would love to get your thoughts before opening a voting thread. Special thanks to a team of folks who worked hard and contributed towards this efforts including design discussion / patch / reviews, etc.: Wangda Tan, Vinod Kumar Vavilappali, Rohith Sharma K S. [1] : https://issues.apache.org/jira/secure/attachment/12855984/YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf [2] : https://issues.apache.org/jira/browse/YARN-5881 [3] : https://issues.apache.org/jira/browse/YARN-7533 [4] : https://issues.apache.org/jira/browse/YARN-7510 Thanks, Sunil G and Wangda Tan
