[
https://issues.apache.org/jira/browse/YARN-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982319#comment-15982319
]
JackZhou commented on YARN-6516:
--------------------------------
[~yufeigu] I am test in a real cluster which have about 2500 nodes. I have
already set continuous scheduling on but I set the
yarn.scheduler.fair.continuous-scheduling-sleep-ms to 500, so it is run per 500
ms. There is about 80 parent queues in my scheduler and about 200 queues total.
I think the scheduler assign a thousand containers per second is a pretty ideal
scenario, because if my queue is very empty
it will cost 1ms to assign a container for scheduler.
But in my test, I have two queues, the queue information as blow:
Used Resources: <memory:19456, vCores:19>
Num Active Applications: 19
Num Pending Applications: 1057
Min Resources: <memory:1024000, vCores:400>
Max Resources: <memory:3072000, vCores:1200>
Max Running Applications: 4000
Steady Fair Share: <memory:1024000, vCores:0>
Instantaneous Fair Share: <memory:3072000, vCores:0>
Used Resources: <memory:20480, vCores:20>
Num Active Applications: 20
Num Pending Applications: 781
Min Resources: <memory:1024000, vCores:400>
Max Resources: <memory:3072000, vCores:1200>
Max Running Applications: 4000
Steady Fair Share: <memory:1024000, vCores:0>
Instantaneous Fair Share: <memory:3072000, vCores:0>
The cost to assign a container up to about 3ms, and the scheduler only can
scheduler about 40 containers.
It is so slow!!!!!
Apps Submitted Apps Pending Apps Running Apps Completed Containers
Running Memory Used Memory Total Memory Reserved VCores Used
VCores Total VCores Reserved Active Nodes Decommissioned Nodes Lost
Nodes Unhealthy Nodes Rebooted Nodes
10268 1838 39 8391 39 39 GB 95 TB 0 B 39 97280
0 2432 2 64 0 0
User Metrics for hadoop
> FairScheduler:the algorithm of assignContainer is so slow for it only can
> assign a thousand containers per second
> -----------------------------------------------------------------------------------------------------------------
>
> Key: YARN-6516
> URL: https://issues.apache.org/jira/browse/YARN-6516
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 2.7.2
> Reporter: JackZhou
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
