[jira] [Commented] (YARN-2285) Preemption can cause capacity scheduler to show 5,000% queue capacity.
[ https://issues.apache.org/jira/browse/YARN-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063679#comment-14063679 ] Vinod Kumar Vavilapalli commented on YARN-2285: --- From the look of it, it sounds like this isn't tied to preemption. It looks like this was a bug that exists even when preemption is not enabled. Can we validate that? Preemption can cause capacity scheduler to show 5,000% queue capacity. -- Key: YARN-2285 URL: https://issues.apache.org/jira/browse/YARN-2285 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 2.5.0 Environment: Turn on CS Preemption. Reporter: Tassapol Athiapinya Assignee: Wangda Tan Priority: Minor Attachments: preemption_5000_percent.png I configure queue A, B to have 1%, 99% capacity respectively. There is no max capacity for each queue. Set high user limit factor. Submit app 1 to queue A. AM container takes 50% of cluster memory. Task containers take another 50%. Submit app 2 to queue B. Preempt task containers of app 1 out. Turns out capacity of queue B increases to 99% but queue A has 5000% used. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2285) Preemption can cause capacity scheduler to show 5,000% queue capacity.
[ https://issues.apache.org/jira/browse/YARN-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063710#comment-14063710 ] Sunil G commented on YARN-2285: --- I also got 106% in root queue with user-factor configured as 2. I observed reservation was happening in my scenario (2 nodes with 8GB). There were some preemption happened also in my test case, I will try collect logs share the same. Also will try without preemption as Vinod mentioned. Preemption can cause capacity scheduler to show 5,000% queue capacity. -- Key: YARN-2285 URL: https://issues.apache.org/jira/browse/YARN-2285 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 2.5.0 Environment: Turn on CS Preemption. Reporter: Tassapol Athiapinya Assignee: Wangda Tan Priority: Minor Attachments: preemption_5000_percent.png I configure queue A, B to have 1%, 99% capacity respectively. There is no max capacity for each queue. Set high user limit factor. Submit app 1 to queue A. AM container takes 50% of cluster memory. Task containers take another 50%. Submit app 2 to queue B. Preempt task containers of app 1 out. Turns out capacity of queue B increases to 99% but queue A has 5000% used. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2285) Preemption can cause capacity scheduler to show 5,000% queue capacity.
[ https://issues.apache.org/jira/browse/YARN-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063725#comment-14063725 ] Wangda Tan commented on YARN-2285: -- Thanks comments from Vinod and Sunil, bq. From the look of it, it sounds like this isn't tied to preemption. It looks like this was a bug that exists even when preemption is not enabled. Can we validate that? I'll validate this tomorrow The usage of root queue above 100% is caused by reserved container, currently the UI shows queue allocated+reserved, we may need change that for user easier understand what happened. Preemption can cause capacity scheduler to show 5,000% queue capacity. -- Key: YARN-2285 URL: https://issues.apache.org/jira/browse/YARN-2285 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 2.5.0 Environment: Turn on CS Preemption. Reporter: Tassapol Athiapinya Assignee: Wangda Tan Priority: Minor Attachments: preemption_5000_percent.png I configure queue A, B to have 1%, 99% capacity respectively. There is no max capacity for each queue. Set high user limit factor. Submit app 1 to queue A. AM container takes 50% of cluster memory. Task containers take another 50%. Submit app 2 to queue B. Preempt task containers of app 1 out. Turns out capacity of queue B increases to 99% but queue A has 5000% used. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2285) Preemption can cause capacity scheduler to show 5,000% queue capacity.
[ https://issues.apache.org/jira/browse/YARN-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064492#comment-14064492 ] Wangda Tan commented on YARN-2285: -- I've verified this will still happen even if preemption is not enabled, both for 5000% queue usage and above 100% root queue usage. Preemption can cause capacity scheduler to show 5,000% queue capacity. -- Key: YARN-2285 URL: https://issues.apache.org/jira/browse/YARN-2285 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 2.5.0 Environment: Turn on CS Preemption. Reporter: Tassapol Athiapinya Assignee: Wangda Tan Priority: Minor Attachments: preemption_5000_percent.png I configure queue A, B to have 1%, 99% capacity respectively. There is no max capacity for each queue. Set high user limit factor. Submit app 1 to queue A. AM container takes 50% of cluster memory. Task containers take another 50%. Submit app 2 to queue B. Preempt task containers of app 1 out. Turns out capacity of queue B increases to 99% but queue A has 5000% used. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2285) Preemption can cause capacity scheduler to show 5,000% queue capacity.
[ https://issues.apache.org/jira/browse/YARN-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062697#comment-14062697 ] Tassapol Athiapinya commented on YARN-2285: --- Also it is not major but percentage shown is not right. In attached screenshot, root queue used is 146.5%. Preemption can cause capacity scheduler to show 5,000% queue capacity. -- Key: YARN-2285 URL: https://issues.apache.org/jira/browse/YARN-2285 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 2.5.0 Environment: Turn on CS Preemption. Reporter: Tassapol Athiapinya Priority: Minor Attachments: preemption_5000_percent.png I configure queue A, B to have 1%, 99% capacity respectively. There is no max capacity for each queue. Set high user limit factor. Submit app 1 to queue A. AM container takes 50% of cluster memory. Task containers take another 50%. Submit app 2 to queue B. Preempt task containers of app 1 out. Turns out capacity of queue B increases to 99% but queue A has 5000% used. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2285) Preemption can cause capacity scheduler to show 5,000% queue capacity.
[ https://issues.apache.org/jira/browse/YARN-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062983#comment-14062983 ] Wangda Tan commented on YARN-2285: -- [~tassapola], one question, is the root queue usage over 100% reproduce-able? If yes, could you provide info about how to reproduce this issue? Thanks, Preemption can cause capacity scheduler to show 5,000% queue capacity. -- Key: YARN-2285 URL: https://issues.apache.org/jira/browse/YARN-2285 Project: Hadoop YARN Issue Type: Improvement Components: capacityscheduler Affects Versions: 2.5.0 Environment: Turn on CS Preemption. Reporter: Tassapol Athiapinya Assignee: Wangda Tan Priority: Minor Attachments: preemption_5000_percent.png I configure queue A, B to have 1%, 99% capacity respectively. There is no max capacity for each queue. Set high user limit factor. Submit app 1 to queue A. AM container takes 50% of cluster memory. Task containers take another 50%. Submit app 2 to queue B. Preempt task containers of app 1 out. Turns out capacity of queue B increases to 99% but queue A has 5000% used. -- This message was sent by Atlassian JIRA (v6.2#6252)
