[
https://issues.apache.org/jira/browse/YARN-7005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-7005:
---------------------------
Attachment: YARN-7005.002.patch
Thanks [~leftnoteasy] for your suggestion.
Attaching v2 patch with performance test cases in TestCapacitySchedulerPerf
which prepare large scale queues (100/500/1000/5000) and make one allocation in
random queue for 1000 times. In my local test, this change can increase the
performance of 33% for 100 queues, 72% for 500 queues, 130% for 1000 queues and
300% for 5000 queues.
Testing result:
{noformat}
Before:
#QueueSize = 5000, testing times : 1000, total cost : 7353788602 ns, average
cost : 7353788.5 ns.
#QueueSize = 5000, testing times : 1000, total cost : 7677551118 ns, average
cost : 7677551.0 ns.
#QueueSize = 1000, testing times : 1000, total cost : 1873387351 ns, average
cost : 1873387.4 ns.
#QueueSize = 1000, testing times : 1000, total cost : 1858447758 ns, average
cost : 1858447.8 ns.
#QueueSize = 500, testing times : 1000, total cost : 1165215528 ns, average
cost : 1165215.5 ns.
#QueueSize = 500, testing times : 1000, total cost : 1188830091 ns, average
cost : 1188830.1 ns.
#QueueSize = 100, testing times : 1000, total cost : 591136755 ns, average cost
: 591136.75 ns.
#QueueSize = 100, testing times : 1000, total cost : 582527533 ns, average cost
: 582527.56 ns.
After:
#QueueSize = 5000, testing times : 1000, total cost : 1972288539 ns, average
cost : 1972288.5 ns.
#QueueSize = 5000, testing times : 1000, total cost : 1743286841 ns, average
cost : 1743286.8 ns.
#QueueSize = 1000, testing times : 1000, total cost : 727941445 ns, average
cost : 727941.44 ns.
#QueueSize = 1000, testing times : 1000, total cost : 831315133 ns, average
cost : 831315.1 ns.
#QueueSize = 500, testing times : 1000, total cost : 618290660 ns, average cost
: 618290.7 ns.
#QueueSize = 500, testing times : 1000, total cost : 677579089 ns, average cost
: 677579.06 ns.
#QueueSize = 100, testing times : 1000, total cost : 442223490 ns, average cost
: 442223.5 ns.
#QueueSize = 100, testing times : 1000, total cost : 470641208 ns, average cost
: 470641.22 ns.
{noformat}
> Skip unnecessary sorting and iterating process for child queues without
> pending resource to optimize schedule performance
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-7005
> URL: https://issues.apache.org/jira/browse/YARN-7005
> Project: Hadoop YARN
> Issue Type: Improvement
> Affects Versions: 2.9.0, 3.0.0-alpha4
> Reporter: Tao Yang
> Attachments: YARN-7005.001.patch, YARN-7005.002.patch
>
>
> Nowadays even if there is only one pending app in a queue, the scheduling
> process will go through all queues anyway and costs most of time on sorting
> and iterating child queues in ParentQueue#assignContainersToChildQueues.
> IIUIC, queues that have no pending resource can be skipped for sorting and
> iterating process to reduce time cost, obviously for a cluster with many
> queues. Please feel free to correct me if I ignore something else. Thanks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
