[
https://issues.apache.org/jira/browse/YARN-7004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16259019#comment-16259019
]
Tao Yang commented on YARN-7004:
--------------------------------
Thanks [~leftnoteasy] and [~jhung] for your comments.
This issue is irrelevant to the backing store of queue config or queue
managements. Most of time is cost on iterating over all configurations in
memory and execute {{String.startsWith}} to get accessible-node-labels and
ordering-policy configs for every queue. For example, if there are 20,000
configuration items, there will be 20,000 executions of {{String.startsWith}}
for every queue. So that it will cost minutes of time to refresh queues for a
cluster with large scale of queues.
Regarding YARN-5734, we already applied this feature in our product clusters
and have proposed YARN-7497 to add HDFS store for queue config. Please help to
review if you have time.
Regarding YARN-7117, currently we have no requirements about this, but perhaps
need it later. We will keep watching it.
> Add configs cache to optimize refreshQueues performance for large scale of
> queues
> ---------------------------------------------------------------------------------
>
> Key: YARN-7004
> URL: https://issues.apache.org/jira/browse/YARN-7004
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
> Affects Versions: 2.9.0, 3.0.0-alpha4
> Reporter: Tao Yang
> Assignee: Tao Yang
> Attachments: YARN-7004.001.patch
>
>
> We have requirements for large scale queues in our production environment to
> serve for many projects. So we did some tests for more than 5000 queues and
> found that refreshQueues process took more than 1 minute. The refreshQueues
> process costs most of time on iterating over all configurations to get
> accessible-node-labels and ordering-policy configs for every queue.
> Loading queue configs from cache should be beneficial to reduce time costs
> (optimized from 1 minutes to 3 seconds for 5000 queues in our test) when
> initializing/reinitializing queues. So I propose to load queue configs into
> cache in CapacityScheduler#initializeQueues and
> CapacityScheduler#reinitializeQueues. If cache has not be loaded on other
> scenes, such as in test cases, it still can get queue configs by iterating
> over all configurations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
