Tao Yang created YARN-7004:
------------------------------
Summary: Add configs cache to optimize refreshQueues performance
for large scale queues
Key: YARN-7004
URL: https://issues.apache.org/jira/browse/YARN-7004
Project: Hadoop YARN
Issue Type: Improvement
Components: capacityscheduler
Affects Versions: 3.0.0-alpha4, 2.9.0
Reporter: Tao Yang
Assignee: Tao Yang
We have requirements for large scale queues in our production environment to
serve for many projects. So we did some tests for more than 5000 queues and
found that refreshQueues process took more than 1 minute. The refreshQueues
process costs most of time on iterating over all configurations to get
accessible-node-labels and ordering-policy configs for every queue.
Loading queue configs from cache should be beneficial to reduce time costs
(optimized from 1 minutes to 3 seconds for 5000 queues in our test) when
initializing/reinitializing queues. So I propose to load queue configs into
cache in CapacityScheduler#initializeQueues and
CapacityScheduler#reinitializeQueues. If cache has not be loaded on other
scenes, such as in test cases, it still can get queue configs by iterating over
all configurations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
