[jira] [Commented] (YARN-7769) FS QueueManager should not create default queue at init

2021-04-14 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320963#comment-17320963
 ] 

Benjamin Teke commented on YARN-7769:
-

And I fixed the checkstyle issues in the latest patch.

> FS QueueManager should not create default queue at init
> ---
>
> Key: YARN-7769
> URL: https://issues.apache.org/jira/browse/YARN-7769
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-7769.001.patch, YARN-7769.002.patch, 
> YARN-7769.003.patch
>
>
> Currently the FairScheduler QueueManager automatically creates the default 
> queue. However the default queue does not need to exist. We have two possible 
> cases which we should handle:
> * Based on the placement rule "Default" the name for the default queue might 
> not be default and it should be created with a different name
> * There might not be a "Default" placement rule at all which removes the need 
> to create the queue.
> We should leave the creation of the default queue to the point in time that 
> we can assess if it is needed or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7769) FS QueueManager should not create default queue at init

2021-04-14 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-7769:

Attachment: YARN-7769.003.patch

> FS QueueManager should not create default queue at init
> ---
>
> Key: YARN-7769
> URL: https://issues.apache.org/jira/browse/YARN-7769
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-7769.001.patch, YARN-7769.002.patch, 
> YARN-7769.003.patch
>
>
> Currently the FairScheduler QueueManager automatically creates the default 
> queue. However the default queue does not need to exist. We have two possible 
> cases which we should handle:
> * Based on the placement rule "Default" the name for the default queue might 
> not be default and it should be created with a different name
> * There might not be a "Default" placement rule at all which removes the need 
> to create the queue.
> We should leave the creation of the default queue to the point in time that 
> we can assess if it is needed or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7769) FS QueueManager should not create default queue at init

2021-04-14 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320806#comment-17320806
 ] 

Benjamin Teke commented on YARN-7769:
-

Yes, I missed those, thanks. Uploaded a new patch, let's see the CI results now.

> FS QueueManager should not create default queue at init
> ---
>
> Key: YARN-7769
> URL: https://issues.apache.org/jira/browse/YARN-7769
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-7769.001.patch, YARN-7769.002.patch
>
>
> Currently the FairScheduler QueueManager automatically creates the default 
> queue. However the default queue does not need to exist. We have two possible 
> cases which we should handle:
> * Based on the placement rule "Default" the name for the default queue might 
> not be default and it should be created with a different name
> * There might not be a "Default" placement rule at all which removes the need 
> to create the queue.
> We should leave the creation of the default queue to the point in time that 
> we can assess if it is needed or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7769) FS QueueManager should not create default queue at init

2021-04-14 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-7769:

Attachment: YARN-7769.002.patch

> FS QueueManager should not create default queue at init
> ---
>
> Key: YARN-7769
> URL: https://issues.apache.org/jira/browse/YARN-7769
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-7769.001.patch, YARN-7769.002.patch
>
>
> Currently the FairScheduler QueueManager automatically creates the default 
> queue. However the default queue does not need to exist. We have two possible 
> cases which we should handle:
> * Based on the placement rule "Default" the name for the default queue might 
> not be default and it should be created with a different name
> * There might not be a "Default" placement rule at all which removes the need 
> to create the queue.
> We should leave the creation of the default queue to the point in time that 
> we can assess if it is needed or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-7769) FS QueueManager should not create default queue at init

2021-04-13 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-7769:

Attachment: YARN-7769.001.patch

> FS QueueManager should not create default queue at init
> ---
>
> Key: YARN-7769
> URL: https://issues.apache.org/jira/browse/YARN-7769
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-7769.001.patch
>
>
> Currently the FairScheduler QueueManager automatically creates the default 
> queue. However the default queue does not need to exist. We have two possible 
> cases which we should handle:
> * Based on the placement rule "Default" the name for the default queue might 
> not be default and it should be created with a different name
> * There might not be a "Default" placement rule at all which removes the need 
> to create the queue.
> We should leave the creation of the default queue to the point in time that 
> we can assess if it is needed or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-7769) FS QueueManager should not create default queue at init

2021-04-12 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke reassigned YARN-7769:
---

Assignee: Benjamin Teke  (was: Wilfred Spiegelenburg)

> FS QueueManager should not create default queue at init
> ---
>
> Key: YARN-7769
> URL: https://issues.apache.org/jira/browse/YARN-7769
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Benjamin Teke
>Priority: Major
>
> Currently the FairScheduler QueueManager automatically creates the default 
> queue. However the default queue does not need to exist. We have two possible 
> cases which we should handle:
> * Based on the placement rule "Default" the name for the default queue might 
> not be default and it should be created with a different name
> * There might not be a "Default" placement rule at all which removes the need 
> to create the queue.
> We should leave the creation of the default queue to the point in time that 
> we can assess if it is needed or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-7769) FS QueueManager should not create default queue at init

2021-04-12 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319537#comment-17319537
 ] 

Benjamin Teke commented on YARN-7769:
-

I see no activity on this for a while now, so I'll take this over, if you don't 
have any objections [~wilfreds].

> FS QueueManager should not create default queue at init
> ---
>
> Key: YARN-7769
> URL: https://issues.apache.org/jira/browse/YARN-7769
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 3.1.0
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Major
>
> Currently the FairScheduler QueueManager automatically creates the default 
> queue. However the default queue does not need to exist. We have two possible 
> cases which we should handle:
> * Based on the placement rule "Default" the name for the default queue might 
> not be default and it should be created with a different name
> * There might not be a "Default" placement rule at all which removes the need 
> to create the queue.
> We should leave the creation of the default queue to the point in time that 
> we can assess if it is needed or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10676) Improve code quality in TestTimelineAuthenticationFilterForV1

2021-03-05 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296138#comment-17296138
 ] 

Benjamin Teke commented on YARN-10676:
--

Thanks [~snemeth] for the patch. LGTM (non-binding) from my side as well.

> Improve code quality in TestTimelineAuthenticationFilterForV1
> -
>
> Key: YARN-10676
> URL: https://issues.apache.org/jira/browse/YARN-10676
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-10676.001.patch
>
>
> - In testcase "testDelegationTokenOperations", the exception message is 
> checked but in case it does not match the assertion, the exception is not 
> printed. This happens 3 times.
> - Assertion messages can be added
> - Fields called "httpSpnegoKeytabFile" and "httpSpnegoPrincipal" can be 
> static final.
> - There's a typo in comment "avaiable" (happens 2 times)
> - There are some Assert.fail() calls, without messages.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10642) AsyncDispatcher will stuck introduced by YARN-8995.

2021-03-03 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294633#comment-17294633
 ] 

Benjamin Teke edited comment on YARN-10642 at 3/3/21, 4:17 PM:
---

[~zhengchenyu], looked at the patch. Can you please tend to the checkstyle 
issue? Otherwise LGTM (non-binding).

[~pbacsko] or [~snemeth] if you have the time can you take a look at it?


was (Author: bteke):
[~zhengchenyu], looked at the patch, LGTM (non-binding).

[~pbacsko] or [~snemeth] if you have the time can you take a look at it?

> AsyncDispatcher will stuck introduced by YARN-8995.
> ---
>
> Key: YARN-10642
> URL: https://issues.apache.org/jira/browse/YARN-10642
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Critical
> Attachments: MockForDeadLoop.java, YARN-10642.001.patch, 
> YARN-10642.002.patch, YARN-10642.003.patch, deadloop.png, debugfornode.png, 
> put.png, take.png
>
>
> In our cluster, ResouceManager stuck twice within twenty days. Yarn client 
> can't submit application. I got jstack info at second time, then found the 
> reason.
> I analyze all the jstack, I found many thread stuck because can't get 
> LinkedBlockingQueue.putLock. (Note: Sorry for limited space , omit the 
> analytical process)
> The reason is that one thread hold the putLock all the time, 
> printEventQueueDetails will called forEachRemaining, then hold putLock and 
> readLock. The AsyncDispatcher will stuck.
> {code}
> Thread 6526 (IPC Server handler 454 on default port 8030):
>   State: RUNNABLE
>   Blocked count: 29988
>   Waited count: 2035029
>   Stack:
> 
> java.util.concurrent.LinkedBlockingQueue$LBQSpliterator.forEachRemaining(LinkedBlockingQueue.java:926)
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> 
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.printEventQueueDetails(AsyncDispatcher.java:270)
> 
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:295)
> 
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:408)
> 
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:215)
> 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> 
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:432)
> 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1040)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:958)
> java.security.AccessController.doPrivileged(Native Method)
> {code}
> I analyze LinkedBlockingQueue's source code. I found forEachRemaining in 
> LinkedBlockingQueue.LBQSpliterator may stuck, when forEachRemaining and take 
> are called in different thread. 
> YARN-8995 introduce printEventQueueDetails method, 
> "eventQueue.stream().collect" will called forEachRemaining method.
> Let's see why? "put.png" shows that how to put("a"), "take.png" shows that 
> how to take()。Specical Node: The removed Node will point itself for help gc!!!
> The key point code is in forEachRemaining, we see LBQSpliterator use 
> forEachRemaining to visit all Node. But when got item value from Node, will 
> release the lock. If at this time, take() will be called. 
> The variable 'p' in forEachRemaining may point a Node which point itself, 
> then forEachRemaining will be in dead loop. You can see it in "deadloop.png"
> Let's see a simple uni-test, Let's forEachRemaining called more slow than 
> take, the problem will reproduction。uni-test is MockForDeadLoop.java.
> I debug MockForDeadLoop.java, and see a Node 

[jira] [Commented] (YARN-10642) AsyncDispatcher will stuck introduced by YARN-8995.

2021-03-03 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294633#comment-17294633
 ] 

Benjamin Teke commented on YARN-10642:
--

[~zhengchenyu], looked at the patch, LGTM (non-binding).

[~pbacsko] or [~snemeth] if you have the time can you take a look at it?

> AsyncDispatcher will stuck introduced by YARN-8995.
> ---
>
> Key: YARN-10642
> URL: https://issues.apache.org/jira/browse/YARN-10642
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Critical
> Attachments: MockForDeadLoop.java, YARN-10642.001.patch, 
> YARN-10642.002.patch, YARN-10642.003.patch, deadloop.png, debugfornode.png, 
> put.png, take.png
>
>
> In our cluster, ResouceManager stuck twice within twenty days. Yarn client 
> can't submit application. I got jstack info at second time, then found the 
> reason.
> I analyze all the jstack, I found many thread stuck because can't get 
> LinkedBlockingQueue.putLock. (Note: Sorry for limited space , omit the 
> analytical process)
> The reason is that one thread hold the putLock all the time, 
> printEventQueueDetails will called forEachRemaining, then hold putLock and 
> readLock. The AsyncDispatcher will stuck.
> {code}
> Thread 6526 (IPC Server handler 454 on default port 8030):
>   State: RUNNABLE
>   Blocked count: 29988
>   Waited count: 2035029
>   Stack:
> 
> java.util.concurrent.LinkedBlockingQueue$LBQSpliterator.forEachRemaining(LinkedBlockingQueue.java:926)
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> 
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.printEventQueueDetails(AsyncDispatcher.java:270)
> 
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:295)
> 
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:408)
> 
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:215)
> 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> 
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:432)
> 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1040)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:958)
> java.security.AccessController.doPrivileged(Native Method)
> {code}
> I analyze LinkedBlockingQueue's source code. I found forEachRemaining in 
> LinkedBlockingQueue.LBQSpliterator may stuck, when forEachRemaining and take 
> are called in different thread. 
> YARN-8995 introduce printEventQueueDetails method, 
> "eventQueue.stream().collect" will called forEachRemaining method.
> Let's see why? "put.png" shows that how to put("a"), "take.png" shows that 
> how to take()。Specical Node: The removed Node will point itself for help gc!!!
> The key point code is in forEachRemaining, we see LBQSpliterator use 
> forEachRemaining to visit all Node. But when got item value from Node, will 
> release the lock. If at this time, take() will be called. 
> The variable 'p' in forEachRemaining may point a Node which point itself, 
> then forEachRemaining will be in dead loop. You can see it in "deadloop.png"
> Let's see a simple uni-test, Let's forEachRemaining called more slow than 
> take, the problem will reproduction。uni-test is MockForDeadLoop.java.
> I debug MockForDeadLoop.java, and see a Node point itself. You can see pic 
> "debugfornode.png"
> Environment:
>   OS: CentOS Linux release 7.5.1804 (Core) 
>   JDK: jdk1.8.0_281



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Commented] (YARN-10642) AsyncDispatcher will stuck introduced by YARN-8995.

2021-03-02 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293863#comment-17293863
 ] 

Benjamin Teke commented on YARN-10642:
--

[~zhengchenyu] thanks working on this. Moved it to Submit Patch state to 
trigger CI. Also I'll look at it tomorrow.

> AsyncDispatcher will stuck introduced by YARN-8995.
> ---
>
> Key: YARN-10642
> URL: https://issues.apache.org/jira/browse/YARN-10642
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Critical
> Attachments: MockForDeadLoop.java, YARN-10642.001.patch, 
> YARN-10642.002.patch, YARN-10642.003.patch, deadloop.png, debugfornode.png, 
> put.png, take.png
>
>
> In our cluster, ResouceManager stuck twice within twenty days. Yarn client 
> can't submit application. I got jstack info at second time, then found the 
> reason.
> I analyze all the jstack, I found many thread stuck because can't get 
> LinkedBlockingQueue.putLock. (Note: Sorry for limited space , omit the 
> analytical process)
> The reason is that one thread hold the putLock all the time, 
> printEventQueueDetails will called forEachRemaining, then hold putLock and 
> readLock. The AsyncDispatcher will stuck.
> {code}
> Thread 6526 (IPC Server handler 454 on default port 8030):
>   State: RUNNABLE
>   Blocked count: 29988
>   Waited count: 2035029
>   Stack:
> 
> java.util.concurrent.LinkedBlockingQueue$LBQSpliterator.forEachRemaining(LinkedBlockingQueue.java:926)
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> 
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> 
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.printEventQueueDetails(AsyncDispatcher.java:270)
> 
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:295)
> 
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:408)
> 
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:215)
> 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
> 
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
> 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:432)
> 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1040)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:958)
> java.security.AccessController.doPrivileged(Native Method)
> {code}
> I analyze LinkedBlockingQueue's source code. I found forEachRemaining in 
> LinkedBlockingQueue.LBQSpliterator may stuck, when forEachRemaining and take 
> are called in different thread. 
> YARN-8995 introduce printEventQueueDetails method, 
> "eventQueue.stream().collect" will called forEachRemaining method.
> Let's see why? "put.png" shows that how to put("a"), "take.png" shows that 
> how to take()。Specical Node: The removed Node will point itself for help gc!!!
> The key point code is in forEachRemaining, we see LBQSpliterator use 
> forEachRemaining to visit all Node. But when got item value from Node, will 
> release the lock. If at this time, take() will be called. 
> The variable 'p' in forEachRemaining may point a Node which point itself, 
> then forEachRemaining will be in dead loop. You can see it in "deadloop.png"
> Let's see a simple uni-test, Let's forEachRemaining called more slow than 
> take, the problem will reproduction。uni-test is MockForDeadLoop.java.
> I debug MockForDeadLoop.java, and see a Node point itself. You can see pic 
> "debugfornode.png"
> Environment:
>   OS: CentOS Linux release 7.5.1804 (Core) 
>   JDK: jdk1.8.0_281



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To 

[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used

2021-03-01 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293008#comment-17293008
 ] 

Benjamin Teke commented on YARN-10532:
--

[~zhuqi],

Thanks for the patch. Generally looks good to me, but the latest patch has a 
whitespace error in TestAutoCreatedQueueDeletionPolicy.java, the CI will most 
likely complain about that. Otherwise if you upload a new patch, can you please 
also remove the unused Supplier imports (where applicable, 
TestAutoCreatedQueueDeletionPolicy and 
TestCapacitySchedulerNewQueueAutoCreation have it) and close the mockRM in 
TestSchedulingMonitor.testRMUpdateAutoCreatedQueueDeletionPolicy?

Thanks!

> Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is 
> not being used
> 
>
> Key: YARN-10532
> URL: https://issues.apache.org/jira/browse/YARN-10532
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10532.001.patch, YARN-10532.002.patch, 
> YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, 
> YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, 
> YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, 
> YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, 
> YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, 
> YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, 
> YARN-10532.021.patch, YARN-10532.022.patch, YARN-10532.023.patch, 
> YARN-10532.024.patch, YARN-10532.025.patch, image-2021-02-12-21-32-02-267.png
>
>
> It's better if we can delete auto-created queues when they are not in use for 
> a period of time (like 5 mins). It will be helpful when we have a large 
> number of auto-created queues (e.g. from 500 users), but only a small subset 
> of queues are actively used.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode

2021-02-26 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.006.patch

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-26 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291759#comment-17291759
 ] 

Benjamin Teke commented on YARN-10627:
--

[~pbacsko] sure, fixed it.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-25 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290894#comment-17290894
 ] 

Benjamin Teke commented on YARN-10627:
--

Thanks [~pbacsko] and [~gandras] for the review. Fixed the checkstyle issues, 
added the rm close methods, and removed the unnecessary GB string. The current 
tests include cases for capacity related log string generation (the used 
configurations are mixed mode, they have both capacity and weight in the 
hierarchy), this is why I didn't feel the need to introduce separate cases. Do 
you think it should be done?

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + 

[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode

2021-02-25 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.005.patch

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch, 
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   + 

[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode

2021-02-25 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.004.patch

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, YARN-10627.004.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   + " [= configuredState ]" + 

[jira] [Updated] (YARN-10640) Adjust the queue Configured capacity to Configured weight number for weight mode in UI.

2021-02-24 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10640:
-
Summary: Adjust the queue Configured capacity to  Configured weight number 
for weight mode in UI.  (was: Ajust the queue Configured capacity to  
Configured weight number for weight mode in UI.)

> Adjust the queue Configured capacity to  Configured weight number for weight 
> mode in UI.
> 
>
> Key: YARN-10640
> URL: https://issues.apache.org/jira/browse/YARN-10640
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10640.001.patch, YARN-10640.002.patch, 
> image-2021-02-20-11-21-50-306.png, image-2021-02-20-14-18-56-261.png, 
> image-2021-02-20-14-19-30-767.png
>
>
> In weight mode:
> Both the weight mode static queue and the dynamic queue will show the 
> Configured Capacity to 0. I think this should change to Configured Weight if 
> we use weight mode, this will be helpful.
> Such as in dynamic weight mode queue:
> !image-2021-02-20-11-21-50-306.png|width=528,height=374!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10640) Ajust the queue Configured capacity to Configured weight number for weight mode in UI.

2021-02-24 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289902#comment-17289902
 ] 

Benjamin Teke commented on YARN-10640:
--

[~zhuqi], thanks for working on this. LGTM, +1 (non-binding)

> Ajust the queue Configured capacity to  Configured weight number for weight 
> mode in UI.
> ---
>
> Key: YARN-10640
> URL: https://issues.apache.org/jira/browse/YARN-10640
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10640.001.patch, YARN-10640.002.patch, 
> image-2021-02-20-11-21-50-306.png, image-2021-02-20-14-18-56-261.png, 
> image-2021-02-20-14-19-30-767.png
>
>
> In weight mode:
> Both the weight mode static queue and the dynamic queue will show the 
> Configured Capacity to 0. I think this should change to Configured Weight if 
> we use weight mode, this will be helpful.
> Such as in dynamic weight mode queue:
> !image-2021-02-20-11-21-50-306.png|width=528,height=374!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-22 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288509#comment-17288509
 ] 

Benjamin Teke commented on YARN-10627:
--

[~zhuqi]

Uploaded a new patchset with the added UTs.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   

[jira] [Commented] (YARN-10641) Refactor the max app related update, and fix maxApllications update error when add new queues.

2021-02-22 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288456#comment-17288456
 ] 

Benjamin Teke commented on YARN-10641:
--

[~zhuqi], thanks for working on this, this indeed seems to be an issue.

I've checked the latest patch, and I've a question. Consider the following code 
snippet (AbstractCSQueue#updateMaxAppRelatedField):

{code:java}
if (maxApplications < 0) {
  int maxGlobalPerQueueApps = conf.getGlobalMaximumApplicationsPerQueue();
  if (maxGlobalPerQueueApps > 0) {
// In absolute mode ,should shrink when change to corresponding 
capacity.
maxApplications = this.capacityConfigType != 
CapacityConfigType.ABSOLUTE_RESOURCE ?
  maxGlobalPerQueueApps : Math.max(maxApplications,
(int) (conf.getGlobalMaximumApplicationsPerQueue() * queueCapacities
.getAbsoluteCapacity(label)));
  } else{
//Here we should get the max of label abs capacites.
maxApplications = Math.max(maxApplications,
(int) (conf.getMaximumSystemApplications() * queueCapacities
.getAbsoluteCapacity(label)));
  }
}
{code}

In the if block is the second part (the Math.max call) actually necessary in 
the ternary operator? Because in this block the *maxApplications < 0* condition 
is already true.

Also in the same method at the log.info call there seems to be a debug message 
(_update max app related_). Is it intentional?


> Refactor the max app related update, and fix maxApllications update error 
> when add new queues.
> --
>
> Key: YARN-10641
> URL: https://issues.apache.org/jira/browse/YARN-10641
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Critical
> Attachments: YARN-10641.001.patch, YARN-10641.002.patch, 
> YARN-10641.003.patch, image-2021-02-20-15-49-58-677.png, 
> image-2021-02-20-15-53-51-099.png, image-2021-02-20-15-55-44-780.png, 
> image-2021-02-20-16-29-18-519.png, image-2021-02-20-16-31-13-714.png
>
>
> When refactor the update logic in YARN-10504 .
> The update max applications based abs/cap is wrong, this should be fixed, 
> because the max applications is key part to limit applications in CS.
> For example: 
> When adding a dynamic queue, the other children's max app of parent queue are 
> not updated correctly:
> !image-2021-02-20-15-53-51-099.png|width=639,height=509!  
> The new added queue's max app will updated correctly:
> !image-2021-02-20-15-55-44-780.png|width=542,height=426!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10646) TestCapacitySchedulerWeightMode test descriptor comments doesn't reflect the correct scenario

2021-02-22 Thread Benjamin Teke (Jira)
Benjamin Teke created YARN-10646:


 Summary: TestCapacitySchedulerWeightMode test descriptor comments 
doesn't reflect the correct scenario
 Key: YARN-10646
 URL: https://issues.apache.org/jira/browse/YARN-10646
 Project: Hadoop YARN
  Issue Type: Sub-task
 Environment: 

Reporter: Benjamin Teke
Assignee: Benjamin Teke


There is a mixup in TestCapacitySchedulerWeightMode in the configuration 
creator method comments and the test case descriptor comments. See the 
following code:


{code:java}
/*
   * Queue structure:
   *  root (*)
   *  ___
   * /   \
   *   a x(=100%), y(50%)   b y(=50%), z(=100%)
   *    __
   *  /   /  \
   * a1 ([x,y]: w=100)b1(no)  b2([y,z]: w=100)
   *
   * Parent uses weight, child uses percentage
   */
  public static Configuration getCSConfWithLabelsParentUsePctChildUseWeight(
  Configuration config) {
{code}

While inside the method all the queues (including the second level ones) are 
configured with capacity, only some labels are configured with weights:

{code:java}
conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "x", 100);
conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "y", 100);
conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "z", 100);

...

conf.setQueues(A, new String[] { "a1" });
conf.setCapacityByLabel(A1, RMNodeLabelsManager.NO_LABEL, 100);
conf.setMaximumCapacity(A1, 100);
conf.setAccessibleNodeLabels(A1, toSet("x", "y"));
conf.setDefaultNodeLabelExpression(A1, "x");
conf.setCapacityByLabel(A1, "x", 100);
conf.setCapacityByLabel(A1, "y", 100);

conf.setQueues(B, new String[] { "b1", "b2" });
conf.setCapacityByLabel(B1, RMNodeLabelsManager.NO_LABEL, 50);
conf.setMaximumCapacity(B1, 50);
conf.setAccessibleNodeLabels(B1, RMNodeLabelsManager.EMPTY_STRING_SET);

conf.setCapacityByLabel(B2, RMNodeLabelsManager.NO_LABEL, 50);
conf.setMaximumCapacity(B2, 50);
conf.setAccessibleNodeLabels(B2, toSet("y", "z"));
conf.setCapacityByLabel(B2, "y", 100);
conf.setCapacityByLabel(B2, "z", 100);
{code}





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode

2021-02-22 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.003.patch

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> YARN-10627.003.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   + " [= configuredState ]" + "\n" + "acls = " + 

[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode

2021-02-19 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287213#comment-17287213
 ] 

Benjamin Teke commented on YARN-10627:
--

[~zhuqi], indeed, thanks for finding it. Fixed it with the second patch, will 
introduce some tests for it in patch 003.

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= 

[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode

2021-02-19 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.002.patch

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch, 
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   + " [= configuredState ]" + "\n" + "acls = " + aclsString
> 

[jira] [Commented] (YARN-10636) CS Auto Queue creation should reject submissions with empty path parts

2021-02-19 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286949#comment-17286949
 ] 

Benjamin Teke commented on YARN-10636:
--

Thanks [~shuzirra]. I too think the new test case should be introduced in 
TestCapacitySchedulerNewQueueAutoCreation. If you fix the checkstyle issue 
could you move it to that class?

Thanks!

> CS Auto Queue creation should reject submissions with empty path parts
> --
>
> Key: YARN-10636
> URL: https://issues.apache.org/jira/browse/YARN-10636
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10636.001.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10635) CSMapping rule can return paths with empty parts

2021-02-19 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286947#comment-17286947
 ] 

Benjamin Teke commented on YARN-10635:
--

[~shuzirra] Thanks for the patch. I agree with [~gandras]. LGTM with that minor 
thing.

> CSMapping rule can return paths with empty parts
> 
>
> Key: YARN-10635
> URL: https://issues.apache.org/jira/browse/YARN-10635
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10635.001.patch, YARN-10635.002.patch, 
> YARN-10635.003.patch
>
>
> When a variable to be substituted evaluates to empty string, we might result 
> with paths where one of the parts is empty, these paths are obviously 
> problematic, but sometimes (when the path includes a dynamicParent) we accept 
> them as valid paths instead of getting the fallback action of the rule.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications

2021-02-18 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286547#comment-17286547
 ] 

Benjamin Teke commented on YARN-10513:
--

[~gandras],

Thanks for the update. This revision is LGTM as well.

> CS Flexible Auto Queue Creation RM UIv2 modifications
> -
>
> Key: YARN-10513
> URL: https://issues.apache.org/jira/browse/YARN-10513
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: Screenshot 2021-02-04 at 12.54.25.png, Screenshot 
> 2021-02-04 at 12.54.52.png, Screenshot 2021-02-04 at 12.55.10.png, Screenshot 
> 2021-02-08 at 10.34.32.png, Screenshot 2021-02-17 at 15.22.30.png, 
> YARN-10513.001.patch, YARN-10513.002.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-18 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286354#comment-17286354
 ] 

Benjamin Teke commented on YARN-10609:
--

Thanks [~zhuqi], the latest patch looks good to me.

> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch, YARN-10609.002.patch, 
> YARN-10609.003.patch, YARN-10609.004.patch, YARN-10609.005.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode

2021-02-17 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.001.patch

> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch
>
>
> In YARN-10504 weight mode was added, however the logged information about the 
> created queues or the toString methods weren't updated accordingly. Some 
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
>  LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
>   + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
>   + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
>   + ", absoluteMaxCapacity=" + this.queueCapacities
>   .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
>   + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
>   + ", reservationsContinueLooking=" + reservationsContinueLooking
>   + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
>   + ", priority=" + priority
>   + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " + 
> "capacity=" + queueCapacities.getCapacity() + ", " +  
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() + 
> "usedCapacity=" + getUsedCapacity() + ", " + 
> "numApps=" + getNumApplications() + ", " + 
> "numContainers=" + getNumContainers();
>  }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
>   LOG.info(
>   "Initializing " + getQueuePath() + "\n" + "capacity = "
>   + queueCapacities.getCapacity()
>   + " [= (float) configuredCapacity / 100 ]" + "\n"
>   + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
>   + " [= parentAbsoluteCapacity * capacity ]" + "\n"
>   + "maxCapacity = " + queueCapacities.getMaximumCapacity()
>   + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = 
> "
>   + queueCapacities.getAbsoluteMaximumCapacity()
>   + " [= 1.0 maximumCapacity undefined, "
>   + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 
> otherwise ]"
>   + "\n" + "effectiveMinResource=" +
>   getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
>   + " , effectiveMaxResource=" +
>   getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
>   + "\n" + "userLimit = " + usersManager.getUserLimit()
>   + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
>   + usersManager.getUserLimitFactor()
>   + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = 
> "
>   + maxApplications
>   + " [= configuredMaximumSystemApplicationsPerQueue or"
>   + " (int)(configuredMaximumSystemApplications * 
> absoluteCapacity)]"
>   + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
>   + " [= (int)(maxApplications * (userLimit / 100.0f) * "
>   + "userLimitFactor) ]" + "\n"
>   + "maxParallelApps = " + getMaxParallelApps() + "\n"
>   + "usedCapacity = " +
>   + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory 
> / "
>   + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
>   + "absoluteUsedCapacity = " + absoluteUsedCapacity
>   + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
>   + "maxAMResourcePerQueuePercent = " + 
> maxAMResourcePerQueuePercent
>   + " [= configuredMaximumAMResourcePercent ]" + "\n"
>   + "minimumAllocationFactor = " + minimumAllocationFactor
>   + " [= (float)(maximumAllocationMemory - 
> minimumAllocationMemory) / "
>   + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
>   + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
>   + "numContainers = " + numContainers
>   + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>   + " [= configuredState ]" + "\n" + "acls = " + aclsString
>   + " [= configuredAcls ]" + "\n"
>   + 

[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-17 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285900#comment-17285900
 ] 

Benjamin Teke commented on YARN-10609:
--

[~zhuqi] thanks for the update. There may have been a mixup as the sentences 
with the "default value" and the "specified as float" are duplicated. If you 
remove those it's +1 from my side. Thanks!

User limit factor provides a way to control the max amount of resources that a 
single user can consume. It is the multiple of the queue's capacity. By default 
this is set to 1 which ensures that a single user can never take more than the 
queue's configured capacity irrespective of how idle the cluster is. Increasing 
it means a single user can use more than the minimum capacity of the cluster, 
while decreasing it results in lower maximum resources. -By default this is set 
to 1 which ensures that a single user can never take more than the queue's 
configured capacity irrespective of how idle the cluster is. Value is specified 
as a float.- Setting this to -1 will disable the feature. Value is specified as 
a float. Note: using the flexible auto queue creation 
(yarn.scheduler.capacity..auto-queue-creation-v2) with weights will 
automatically set this property to -1, as the dynamic queues will be created 
with the hardcoded weight of 1 and in idle cluster scenarios they should be 
able to use more resources than calculated.

> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch, YARN-10609.002.patch, 
> YARN-10609.003.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications

2021-02-16 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285222#comment-17285222
 ] 

Benjamin Teke edited comment on YARN-10513 at 2/16/21, 2:15 PM:


[~gandras], thanks for working on this. LGTM as well.

cc: [~pbacsko] [~snemeth] if you have the time for a review.


was (Author: bteke):
[~gandras], thanks for working on this. LGTM as well.

cc: [~snemeth] if you have the time for a review.

> CS Flexible Auto Queue Creation RM UIv2 modifications
> -
>
> Key: YARN-10513
> URL: https://issues.apache.org/jira/browse/YARN-10513
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: Screenshot 2021-02-04 at 12.54.25.png, Screenshot 
> 2021-02-04 at 12.54.52.png, Screenshot 2021-02-04 at 12.55.10.png, Screenshot 
> 2021-02-08 at 10.34.32.png, YARN-10513.001.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications

2021-02-16 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285222#comment-17285222
 ] 

Benjamin Teke commented on YARN-10513:
--

[~gandras], thanks for working on this. LGTM as well.

cc: [~snemeth] if you have the time for a review.

> CS Flexible Auto Queue Creation RM UIv2 modifications
> -
>
> Key: YARN-10513
> URL: https://issues.apache.org/jira/browse/YARN-10513
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: Screenshot 2021-02-04 at 12.54.25.png, Screenshot 
> 2021-02-04 at 12.54.52.png, Screenshot 2021-02-04 at 12.55.10.png, Screenshot 
> 2021-02-08 at 10.34.32.png, YARN-10513.001.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-16 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285211#comment-17285211
 ] 

Benjamin Teke commented on YARN-10609:
--

[~zhuqi] Thanks! Sorry, I have one more thing :) Maybe if we touch this we can 
rephrase the first sentence a bit:

_The multiple of the queue capacity which can be configured to allow a single 
user to acquire more resources. _ 

To me this misses the point that setting this to below 1 limits the users 
resources. I think a phrasing like the following would be clearer:

_User limit factor provides a way to control the max amount of resources that a 
single user can consume. It is the multiple of the queue's capacity. By default 
this is set to 1 which ensures that a single user can never take more than the 
queue's configured capacity irrespective of how idle the cluster is. Increasing 
it means a single user can use more than the minimum capacity of the cluster, 
while decreasing it results in lower maximum resources._

What's your opinion about this?


> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch, YARN-10609.002.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10627) Extend logging to give more information about weight mode

2021-02-15 Thread Benjamin Teke (Jira)
Benjamin Teke created YARN-10627:


 Summary: Extend logging to give more information about weight mode
 Key: YARN-10627
 URL: https://issues.apache.org/jira/browse/YARN-10627
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn
Reporter: Benjamin Teke
Assignee: Benjamin Teke


In YARN-10504 weight mode was added, however the logged information about the 
created queues or the toString methods weren't updated accordingly. Some 
examples:

ParentQueue#setupQueueConfigs:

{code:java}
 LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
  + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
  + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
  + ", absoluteMaxCapacity=" + this.queueCapacities
  .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
  + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
  + ", reservationsContinueLooking=" + reservationsContinueLooking
  + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
  + ", priority=" + priority
  + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
{code}


ParentQueue#toString:

{code:java}
public String toString() {
return queueName + ": " +
"numChildQueue= " + childQueues.size() + ", " + 
"capacity=" + queueCapacities.getCapacity() + ", " +  
"absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
"usedResources=" + queueUsage.getUsed() + 
"usedCapacity=" + getUsedCapacity() + ", " + 
"numApps=" + getNumApplications() + ", " + 
"numContainers=" + getNumContainers();
 }
{code}

LeafQueue#setupQueueConfigs:

{code:java}
  LOG.info(
  "Initializing " + getQueuePath() + "\n" + "capacity = "
  + queueCapacities.getCapacity()
  + " [= (float) configuredCapacity / 100 ]" + "\n"
  + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
  + " [= parentAbsoluteCapacity * capacity ]" + "\n"
  + "maxCapacity = " + queueCapacities.getMaximumCapacity()
  + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = "
  + queueCapacities.getAbsoluteMaximumCapacity()
  + " [= 1.0 maximumCapacity undefined, "
  + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise 
]"
  + "\n" + "effectiveMinResource=" +
  getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
  + " , effectiveMaxResource=" +
  getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
  + "\n" + "userLimit = " + usersManager.getUserLimit()
  + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
  + usersManager.getUserLimitFactor()
  + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = "
  + maxApplications
  + " [= configuredMaximumSystemApplicationsPerQueue or"
  + " (int)(configuredMaximumSystemApplications * 
absoluteCapacity)]"
  + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
  + " [= (int)(maxApplications * (userLimit / 100.0f) * "
  + "userLimitFactor) ]" + "\n"
  + "maxParallelApps = " + getMaxParallelApps() + "\n"
  + "usedCapacity = " +
  + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory / "
  + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
  + "absoluteUsedCapacity = " + absoluteUsedCapacity
  + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
  + "maxAMResourcePerQueuePercent = " + maxAMResourcePerQueuePercent
  + " [= configuredMaximumAMResourcePercent ]" + "\n"
  + "minimumAllocationFactor = " + minimumAllocationFactor
  + " [= (float)(maximumAllocationMemory - minimumAllocationMemory) 
/ "
  + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
  + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
  + "numContainers = " + numContainers
  + " [= currentNumContainers ]" + "\n" + "state = " + getState()
  + " [= configuredState ]" + "\n" + "acls = " + aclsString
  + " [= configuredAcls ]" + "\n"
  + "nodeLocalityDelay = " + nodeLocalityDelay + "\n"
  + "rackLocalityAdditionalDelay = "
  + rackLocalityAdditionalDelay + "\n"
  + "labels=" + labelStrBuilder.toString() + "\n"
  + "reservationsContinueLooking = "
  + reservationsContinueLooking + "\n" + "preemptionDisabled = "
  + getPreemptionDisabled() + "\n" + "defaultAppPriorityPerQueue = "
  + 

[jira] [Commented] (YARN-10625) FairScheduler: add global flag to disable AM-preemption

2021-02-15 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284777#comment-17284777
 ] 

Benjamin Teke commented on YARN-10625:
--

[~pbacsko] thanks for the patch. LGTM +1 (non-binding). I agree with the doc 
update, as the global property doesn't overwrite the queue level ones, which 
can be non-trivial.

> FairScheduler: add global flag to disable AM-preemption
> ---
>
> Key: YARN-10625
> URL: https://issues.apache.org/jira/browse/YARN-10625
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: fairscheduler
>Affects Versions: 3.3.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10625-001.patch
>
>
> YARN-9537 added a feature to disable AM preemption on a per queue basis.
> This is a nice enhancement, but it's very inconvenient if the cluster has a 
> lot of queues or queues dynamically created/deleted regularly (static queue 
> configuration changes).
> It's a legitimate use-case to have AM preemption turned off completely. To 
> make it easier, add property which acts as a global flag for this feature.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.

2021-02-13 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284225#comment-17284225
 ] 

Benjamin Teke commented on YARN-10624:
--

Hey [~zhuqi],

 

I agree, this should be supported, thanks for working on this. The patch is 
clean, looks good to me apart from one minor nit: in ParentQueue can you please 
delete the spaces before the colons in the instantiation of 
SchedulerDynamicEditException? Otherwise +1 (non-binding) from me.

> Support max queues limit configuration in new auto created queue, consistent 
> with old auto created.
> ---
>
> Key: YARN-10624
> URL: https://issues.apache.org/jira/browse/YARN-10624
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10624.001.patch
>
>
> Since old created leaf queue has the max leaf queues limit, i think we also 
> should support this in new auto created queue, both the auto created leaf and 
> the auto created parent need limits.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10618) RM UI2 Application page shows the AM preempted containers instead of the nonAM ones

2021-02-08 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10618:
-
Attachment: YARN-10618.001.patch

> RM UI2 Application page shows the AM preempted containers instead of the 
> nonAM ones
> ---
>
> Key: YARN-10618
> URL: https://issues.apache.org/jira/browse/YARN-10618
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Minor
> Attachments: YARN-10618.001.patch
>
>
> YARN RM UIv2 application page shows the AM preempted containers under both 
> the _Num Non-AM container preempted_ and _Num AM container preempted_.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10618) RM UI2 Application page shows the AM preempted containers instead of the nonAM ones

2021-02-08 Thread Benjamin Teke (Jira)
Benjamin Teke created YARN-10618:


 Summary: RM UI2 Application page shows the AM preempted containers 
instead of the nonAM ones
 Key: YARN-10618
 URL: https://issues.apache.org/jira/browse/YARN-10618
 Project: Hadoop YARN
  Issue Type: Bug
  Components: yarn-ui-v2
Reporter: Benjamin Teke
Assignee: Benjamin Teke


YARN RM UIv2 application page shows the AM preempted containers under both the 
_Num Non-AM container preempted_ and _Num AM container preempted_.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)

2021-02-08 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281017#comment-17281017
 ] 

Benjamin Teke commented on YARN-10609:
--

Hi [~zhuqi],

Thanks for the patch! Generally the content seems to be OK to me, however I 
would rephrase it a little to be more akin to the rest of the documentation:

_The multiple of the queue capacity which can be configured to allow a single 
user to acquire more resources. By default this is set to 1 which ensures that 
a single user can never take more than the queue's configured capacity 
irrespective of how idle the cluster is._ Setting this to -1 will disable the 
feature. _Value is specified as a float._ Note: using the flexible auto queue 
creation (yarn.scheduler.capacity..auto-queue-creation-v2) with 
weights will automatically set this property to -1, as the dynamic queues will 
be created with the hardcoded weight of 1 and in idle cluster scenarios they 
should be able to use more resources than calculated.

 

What do you think about is?

cc: [~gandras], [~snemeth], [~pbacsko], [~shuzirra]

> Update the document for YARN-10531(Be able to disable user limit factor for 
> CapacityScheduler Leaf Queue)
> -
>
> Key: YARN-10609
> URL: https://issues.apache.org/jira/browse/YARN-10609
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10609.001.patch
>
>
> Since we have finished YARN-10531.
> We should update the corresponding document.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Resolved] (YARN-10614) Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion

2021-02-04 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke resolved YARN-10614.
--
Resolution: Invalid

Marking it as invalid, it was already implemented at the time of the creation.

> Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion
> 
>
> Key: YARN-10614
> URL: https://issues.apache.org/jira/browse/YARN-10614
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Fix For: 3.4.0
>
>
> Weight mode conversion was added to fs2cs in YARN-10525 and in YARN-10600 
> changes were made to convert the root queue to weight mode. However root 
> queue doesn't get the auto-create-v2 enabled flag, which again can be 
> misleading to former FS users. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10614) Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion

2021-02-04 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10614:
-
Description: Weight mode conversion was added to fs2cs in YARN-10525 and in 
YARN-10600 changes were made to convert the root queue to weight mode. However 
root queue doesn't get the auto-create-v2 enabled flag, which again can be 
misleading to former FS users.   (was: Weight mode conversion was added to 
fs2cs in YARN-10525. However root queue doesn't get converted to weight mode, 
which can be misleading to former FS users, so it should be added to fs2cs.)

> Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion
> 
>
> Key: YARN-10614
> URL: https://issues.apache.org/jira/browse/YARN-10614
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Fix For: 3.4.0
>
>
> Weight mode conversion was added to fs2cs in YARN-10525 and in YARN-10600 
> changes were made to convert the root queue to weight mode. However root 
> queue doesn't get the auto-create-v2 enabled flag, which again can be 
> misleading to former FS users. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10614) Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion

2021-02-04 Thread Benjamin Teke (Jira)
Benjamin Teke created YARN-10614:


 Summary: Add auto-create-v2-enabled to root queue in fs2cs weight 
mode conversion
 Key: YARN-10614
 URL: https://issues.apache.org/jira/browse/YARN-10614
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Benjamin Teke
Assignee: Benjamin Teke
 Fix For: 3.4.0


Weight mode conversion was added to fs2cs in YARN-10525. However root queue 
doesn't get converted to weight mode, which can be misleading to former FS 
users, so it should be added to fs2cs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10605) Add queue-mappings-override.enable property in FS2CS conversions

2021-02-01 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276356#comment-17276356
 ] 

Benjamin Teke commented on YARN-10605:
--

[~gandras] thanks for the patch. I have one minor nit about it: in 
TestFSConfigToCSConfigConverter you introduced an unused import 
{{org.junit.Assert.assertFalse}}. Otherwise the patch LGTM, +1 (non-binding).

> Add queue-mappings-override.enable property in FS2CS conversions
> 
>
> Key: YARN-10605
> URL: https://issues.apache.org/jira/browse/YARN-10605
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10605.001.patch
>
>
> In Capacity Scheduler the
> {noformat}
> queue-mappings-override.enable
> {noformat}
> property is false by default. As this is not set during an FS2CS conversion, 
> the converted placement rules (aka. mapping rules in CS) are ignored during 
> application submission. We should enable this property in the conversion 
> logic if there are placement rules to be converted.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10604) Support auto queue creation without mapping rules

2021-02-01 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276344#comment-17276344
 ] 

Benjamin Teke commented on YARN-10604:
--

[~gandras] thanks for the patch. Looks good to me, +1 (non-binding).

> Support auto queue creation without mapping rules
> -
>
> Key: YARN-10604
> URL: https://issues.apache.org/jira/browse/YARN-10604
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10604.001.patch
>
>
> Currently, the Capacity Scheduler skips auto queue creation entirely, if the 
> ApplicationPlacementContext is null, which happens, when the mapping rules 
> are turned off by:
> {noformat}
> 
> yarn.scheduler.capacity.queue-mappings-override.enable
> false
> {noformat}
> We should allow the auto queue creation to be taken into consideration 
> without disrupting the application submission flow.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'

2021-01-28 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273883#comment-17273883
 ] 

Benjamin Teke commented on YARN-10178:
--

[~zhuqi] thanks for the patch. The logic looks good to me, +1 (non-binding) 
from my side. 

> Global Scheduler async thread crash caused by 'Comparison method violates its 
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch, YARN-10178.002.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received 
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread, 
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException: 
> Comparison method violates its general contract!  
>at 
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has  few require 
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw 
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know 
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use 
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign 
> container,and construct a CSAssignment struct, and use 
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService  will tryCommit this CSAssignment,look tryCommit 
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
>   long commitStart = System.nanoTime();
>   ResourceCommitRequest request =
>   (ResourceCommitRequest) r;
>  
>   ...
>   boolean isSuccess = false;
>   if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for attemptId - when async-scheduling enabled,
> // proposal might be outdated if AM failover just finished
> // and proposal queue was not be consumed in time
> if (app != null && attemptId.equals(app.getApplicationAttemptId())) {
>   if (app.accept(cluster, request, updatePending)
>   && app.apply(cluster, request, updatePending)) { 

[jira] [Commented] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.

2021-01-28 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273860#comment-17273860
 ] 

Benjamin Teke commented on YARN-10590:
--

Thanks [~zhuqi], for working on this. Can you please elaborate on the part 
where you talk about the validity of the current logic? Because as we discussed 
in YARN-10504 the initialization of auto created queues from template was 
changed (see 
[comment|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549]
 and 
[comment|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17262562=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17262562]).

Also once we decide that we follow with the test modifications we probably 
should modify the jiras description to contain the original logic change 
introduced in YARN-10504.

> Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute 
> caculation loss.
> -
>
> Key: YARN-10590
> URL: https://issues.apache.org/jira/browse/YARN-10590
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: zhuqi
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10590.001.patch, YARN-10590.002.patch
>
>
> Because we have introduced YARN-10504.
> In auto created leaf queue with absolute mode, the caculation of effective 
> min resource will different from other mode. 
> We should fix the related test in  TestCapacitySchedulerAutoCreatedQueueBase.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10600) Convert root queue in fs2cs weight mode conversion

2021-01-28 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10600:
-
Attachment: YARN-10600.001.patch

> Convert root queue in fs2cs weight mode conversion
> --
>
> Key: YARN-10600
> URL: https://issues.apache.org/jira/browse/YARN-10600
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10600.001.patch
>
>
> Weight mode conversion was added to fs2cs in YARN-10525. However root queue 
> doesn't get converted to weight mode, which can be misleading to former FS 
> users, so it should be added to fs2cs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10600) Convert root queue in fs2cs weight mode conversion

2021-01-28 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10600:
-
Description: Weight mode conversion was added to fs2cs in YARN-10525. 
However root queue doesn't get converted to weight mode, which can be 
misleading to former FS users, so it should be added to fs2cs.  (was: Weight 
mode conversion was added to fs2cs in YARN-10525. However root queue doesn't 
get converted to weight mode, which can be misleading to former FS users.

Currently, we convert FS weights to percentages, however, it will be more 
useful to keep those values and use them in CS as well.)

> Convert root queue in fs2cs weight mode conversion
> --
>
> Key: YARN-10600
> URL: https://issues.apache.org/jira/browse/YARN-10600
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Fix For: 3.4.0
>
>
> Weight mode conversion was added to fs2cs in YARN-10525. However root queue 
> doesn't get converted to weight mode, which can be misleading to former FS 
> users, so it should be added to fs2cs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10600) Convert root queue in fs2cs weight mode conversion

2021-01-28 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10600:
-
Description: 
Weight mode conversion was added to fs2cs in YARN-10525. However root queue 
doesn't get converted to weight mode, which can be misleading to former FS 
users.

Currently, we convert FS weights to percentages, however, it will be more 
useful to keep those values and use them in CS as well.

  was:
Weight mode will be added to Capacity Scheduler.

Currently, we convert FS weights to percentages, however, it will be more 
useful to keep those values and use them in CS as well.


> Convert root queue in fs2cs weight mode conversion
> --
>
> Key: YARN-10600
> URL: https://issues.apache.org/jira/browse/YARN-10600
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0
>
>
> Weight mode conversion was added to fs2cs in YARN-10525. However root queue 
> doesn't get converted to weight mode, which can be misleading to former FS 
> users.
> Currently, we convert FS weights to percentages, however, it will be more 
> useful to keep those values and use them in CS as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-10600) Convert root queue in fs2cs weight mode conversion

2021-01-28 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke reassigned YARN-10600:


Assignee: Benjamin Teke  (was: Peter Bacsko)

> Convert root queue in fs2cs weight mode conversion
> --
>
> Key: YARN-10600
> URL: https://issues.apache.org/jira/browse/YARN-10600
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Fix For: 3.4.0
>
>
> Weight mode conversion was added to fs2cs in YARN-10525. However root queue 
> doesn't get converted to weight mode, which can be misleading to former FS 
> users.
> Currently, we convert FS weights to percentages, however, it will be more 
> useful to keep those values and use them in CS as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10600) Convert root queue in fs2cs weight mode conversion

2021-01-28 Thread Benjamin Teke (Jira)
Benjamin Teke created YARN-10600:


 Summary: Convert root queue in fs2cs weight mode conversion
 Key: YARN-10600
 URL: https://issues.apache.org/jira/browse/YARN-10600
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Benjamin Teke
Assignee: Peter Bacsko
 Fix For: 3.4.0


Weight mode will be added to Capacity Scheduler.

Currently, we convert FS weights to percentages, however, it will be more 
useful to keep those values and use them in CS as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-27 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272997#comment-17272997
 ] 

Benjamin Teke commented on YARN-10598:
--

[~snemeth] Thanks for the review, both of them are valid points. Uploaded a new 
patch.

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the 
> creation type with additional information
> --
>
> Key: YARN-10598
> URL: https://issues.apache.org/jira/browse/YARN-10598
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10598.001.patch, YARN-10598.002.patch, 
> YARN-10598.003.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also implemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> To extend/modify the values added in YARN-10581 these 3 fields will describe 
> a queue:
>  * queueType : *parent/leaf*
>  * creationMethod : *static/dynamicLegacy/dynamicFlexible*
>  * autoCreationEligibility : *off/legacy/flexible*
> After this change here are some example cases:
>  * Static parent queue which has the auto-creation-enabled-v2 false:
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *off*
>  * Static managed parent (can have dynamic children):
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *legacy*
>  * Legacy auto-created leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicLegacy*
>  ** autoCreationEligibility : *off*
>  * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 
> true: 
>  ** queueType : *parent*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *flexible*
>  * Auto-created (v2) leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *off*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-27 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10598:
-
Attachment: YARN-10598.003.patch

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the 
> creation type with additional information
> --
>
> Key: YARN-10598
> URL: https://issues.apache.org/jira/browse/YARN-10598
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10598.001.patch, YARN-10598.002.patch, 
> YARN-10598.003.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also implemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> To extend/modify the values added in YARN-10581 these 3 fields will describe 
> a queue:
>  * queueType : *parent/leaf*
>  * creationMethod : *static/dynamicLegacy/dynamicFlexible*
>  * autoCreationEligibility : *off/legacy/flexible*
> After this change here are some example cases:
>  * Static parent queue which has the auto-creation-enabled-v2 false:
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *off*
>  * Static managed parent (can have dynamic children):
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *legacy*
>  * Legacy auto-created leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicLegacy*
>  ** autoCreationEligibility : *off*
>  * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 
> true: 
>  ** queueType : *parent*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *flexible*
>  * Auto-created (v2) leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *off*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-27 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272881#comment-17272881
 ] 

Benjamin Teke commented on YARN-10598:
--

Thank [~gandras] for the review. Fixed the suggestions and the checkstyle 
issues that doesn't need a larger refactor.

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the 
> creation type with additional information
> --
>
> Key: YARN-10598
> URL: https://issues.apache.org/jira/browse/YARN-10598
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10598.001.patch, YARN-10598.002.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also implemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> To extend/modify the values added in YARN-10581 these 3 fields will describe 
> a queue:
>  * queueType : *parent/leaf*
>  * creationMethod : *static/dynamicLegacy/dynamicFlexible*
>  * autoCreationEligibility : *off/legacy/flexible*
> After this change here are some example cases:
>  * Static parent queue which has the auto-creation-enabled-v2 false:
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *off*
>  * Static managed parent (can have dynamic children):
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *legacy*
>  * Legacy auto-created leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicLegacy*
>  ** autoCreationEligibility : *off*
>  * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 
> true: 
>  ** queueType : *parent*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *flexible*
>  * Auto-created (v2) leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *off*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-27 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10598:
-
Attachment: YARN-10598.002.patch

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the 
> creation type with additional information
> --
>
> Key: YARN-10598
> URL: https://issues.apache.org/jira/browse/YARN-10598
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10598.001.patch, YARN-10598.002.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also implemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> To extend/modify the values added in YARN-10581 these 3 fields will describe 
> a queue:
>  * queueType : *parent/leaf*
>  * creationMethod : *static/dynamicLegacy/dynamicFlexible*
>  * autoCreationEligibility : *off/legacy/flexible*
> After this change here are some example cases:
>  * Static parent queue which has the auto-creation-enabled-v2 false:
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *off*
>  * Static managed parent (can have dynamic children):
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *legacy*
>  * Legacy auto-created leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicLegacy*
>  ** autoCreationEligibility : *off*
>  * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 
> true: 
>  ** queueType : *parent*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *flexible*
>  * Auto-created (v2) leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *off*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-26 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10598:
-
Attachment: YARN-10598.001.patch

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the 
> creation type with additional information
> --
>
> Key: YARN-10598
> URL: https://issues.apache.org/jira/browse/YARN-10598
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10598.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also implemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> To extend/modify the values added in YARN-10581 these 3 fields will describe 
> a queue:
>  * queueType : *parent/leaf*
>  * creationMethod : *static/dynamicLegacy/dynamicFlexible*
>  * autoCreationEligibility : *off/legacy/flexible*
> After this change here are some example cases:
>  * Static parent queue which has the auto-creation-enabled-v2 false:
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *off*
>  * Static managed parent (can have dynamic children):
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *legacy*
>  * Legacy auto-created leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicLegacy*
>  ** autoCreationEligibility : *off*
>  * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 
> true: 
>  ** queueType : *parent*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *flexible*
>  * Auto-created (v2) leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *off*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-26 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10598:
-
Description: 
Under this umbrella (YARN-10496), weight-mode has been implemented for CS with 
YARN-10504.

Auto-queue creation has been also implemented with YARN-10506.

Connected to this effort, we would like to expose the type of the queue with 
the RM's /scheduler REST endpoint.

To extend/modify the values added in YARN-10581 these 3 fields will describe a 
queue:
 * queueType : *parent/leaf*
 * creationMethod : *static/dynamicLegacy/dynamicFlexible*
 * autoCreationEligibility : *off/legacy/flexible*

After this change here are some example cases:
 * Static parent queue which has the auto-creation-enabled-v2 false:
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *off*
 * Static managed parent (can have dynamic children):
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *legacy*
 * Legacy auto-created leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicLegacy*
 ** autoCreationEligibility : *off*
 * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 true: 
 ** queueType : *parent*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *flexible*
 * Auto-created (v2) leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *off*

  was:
Under this umbrella (YARN-10496), weight-mode has been implemented for CS with 
YARN-10504.

Auto-queue creation has been also implemented with YARN-10506.

Connected to this effort, we would like to expose the type of the queue with 
the RM's /scheduler REST endpoint.

To extend/modify the values added in YARN-10581 these 3 fields will describe a 
queue:
 * queueType : *parent/leaf*
 * creationMethod : *static/dynamicLegacy/dynamicFlexible*
 * autoCreationEligibility : *off/legacy/flexible*

After this change here are some example cases:
 * Static parent queue which has the auto-creation-enabled-v2 false:
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *off*
 * Static managed parent (can have dynamic children):
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *legacy*
 * Legacy auto-created leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicLegacy*
 ** autoCreationEligibility : *off*
 * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: 
 ** queueType : *parent*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *flexible*
 * Auto-created (v2) leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *off*


> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the 
> creation type with additional information
> --
>
> Key: YARN-10598
> URL: https://issues.apache.org/jira/browse/YARN-10598
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also implemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> To extend/modify the values added in YARN-10581 these 3 fields will describe 
> a queue:
>  * queueType : *parent/leaf*
>  * creationMethod : *static/dynamicLegacy/dynamicFlexible*
>  * autoCreationEligibility : *off/legacy/flexible*
> After this change here are some example cases:
>  * Static parent queue which has the auto-creation-enabled-v2 false:
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *off*
>  * Static managed parent (can have dynamic children):
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *legacy*
>  * Legacy auto-created leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicLegacy*
>  ** autoCreationEligibility : *off*
>  * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 
> true: 
>  ** queueType : *parent*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *flexible*
>  * Auto-created (v2) leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *off*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: 

[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-26 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10598:
-
Description: 
Under this umbrella (YARN-10496), weight-mode has been implemented for CS with 
YARN-10504.

Auto-queue creation has been also implemented with YARN-10506.

Connected to this effort, we would like to expose the type of the queue with 
the RM's /scheduler REST endpoint.

To extend/modify the values added in YARN-10581 these 3 fields will describe a 
queue:
 * queueType : *parent/leaf*
 * creationMethod : *static/dynamicLegacy/dynamicFlexible*
 * autoCreationEligibility : *off/legacy/flexible*

After this change here are some example cases:
 * Static parent queue which has the auto-creation-enabled-v2 false:
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *off*
 * **Static managed parent (can have dynamic children):
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *legacy*
 * Legacy auto-created leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicLegacy*
 ** autoCreationEligibility : *off*
 * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: 
 ** queueType : *parent*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *flexible*
 * Auto-created (v2) leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *off*

  was:
Under this umbrella (YARN-10496), weight-mode has been implemented for CS with 
YARN-10504.

Auto-queue creation has been also imlemented with YARN-10506.

Connected to this effort, we would like to expose the type of the queue with 
the RM's /scheduler REST endpoint.

The queue type should hold these values: 
 * Auto-created parent queue: *autoCreatedParent*
 * Auto-created leaf queue: *autoCreatedLeaf*
 * Static parent: *staticParent*
 * Static leaf: *staticLeaf* 


> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the 
> creation type with additional information
> --
>
> Key: YARN-10598
> URL: https://issues.apache.org/jira/browse/YARN-10598
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also implemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> To extend/modify the values added in YARN-10581 these 3 fields will describe 
> a queue:
>  * queueType : *parent/leaf*
>  * creationMethod : *static/dynamicLegacy/dynamicFlexible*
>  * autoCreationEligibility : *off/legacy/flexible*
> After this change here are some example cases:
>  * Static parent queue which has the auto-creation-enabled-v2 false:
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *off*
>  * **Static managed parent (can have dynamic children):
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *legacy*
>  * Legacy auto-created leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicLegacy*
>  ** autoCreationEligibility : *off*
>  * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: 
>  ** queueType : *parent*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *flexible*
>  * Auto-created (v2) leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *off*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-26 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10598:
-
Description: 
Under this umbrella (YARN-10496), weight-mode has been implemented for CS with 
YARN-10504.

Auto-queue creation has been also implemented with YARN-10506.

Connected to this effort, we would like to expose the type of the queue with 
the RM's /scheduler REST endpoint.

To extend/modify the values added in YARN-10581 these 3 fields will describe a 
queue:
 * queueType : *parent/leaf*
 * creationMethod : *static/dynamicLegacy/dynamicFlexible*
 * autoCreationEligibility : *off/legacy/flexible*

After this change here are some example cases:
 * Static parent queue which has the auto-creation-enabled-v2 false:
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *off*
 * Static managed parent (can have dynamic children):
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *legacy*
 * Legacy auto-created leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicLegacy*
 ** autoCreationEligibility : *off*
 * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: 
 ** queueType : *parent*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *flexible*
 * Auto-created (v2) leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *off*

  was:
Under this umbrella (YARN-10496), weight-mode has been implemented for CS with 
YARN-10504.

Auto-queue creation has been also implemented with YARN-10506.

Connected to this effort, we would like to expose the type of the queue with 
the RM's /scheduler REST endpoint.

To extend/modify the values added in YARN-10581 these 3 fields will describe a 
queue:
 * queueType : *parent/leaf*
 * creationMethod : *static/dynamicLegacy/dynamicFlexible*
 * autoCreationEligibility : *off/legacy/flexible*

After this change here are some example cases:
 * Static parent queue which has the auto-creation-enabled-v2 false:
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *off*
 * **Static managed parent (can have dynamic children):
 ** queueType : *parent*
 ** creationMethod : *static*
 ** autoCreationEligibility : *legacy*
 * Legacy auto-created leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicLegacy*
 ** autoCreationEligibility : *off*
 * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: 
 ** queueType : *parent*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *flexible*
 * Auto-created (v2) leaf queue (cannot have children):
 ** queueType : *leaf*
 ** creationMethod : *dynamicFlexible*
 ** autoCreationEligibility : *off*


> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the 
> creation type with additional information
> --
>
> Key: YARN-10598
> URL: https://issues.apache.org/jira/browse/YARN-10598
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also implemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> To extend/modify the values added in YARN-10581 these 3 fields will describe 
> a queue:
>  * queueType : *parent/leaf*
>  * creationMethod : *static/dynamicLegacy/dynamicFlexible*
>  * autoCreationEligibility : *off/legacy/flexible*
> After this change here are some example cases:
>  * Static parent queue which has the auto-creation-enabled-v2 false:
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *off*
>  * Static managed parent (can have dynamic children):
>  ** queueType : *parent*
>  ** creationMethod : *static*
>  ** autoCreationEligibility : *legacy*
>  * Legacy auto-created leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicLegacy*
>  ** autoCreationEligibility : *off*
>  * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: 
>  ** queueType : *parent*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *flexible*
>  * Auto-created (v2) leaf queue (cannot have children):
>  ** queueType : *leaf*
>  ** creationMethod : *dynamicFlexible*
>  ** autoCreationEligibility : *off*



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: 

[jira] [Assigned] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-26 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke reassigned YARN-10598:


Assignee: Benjamin Teke  (was: Szilard Nemeth)

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the 
> creation type with additional information
> --
>
> Key: YARN-10598
> URL: https://issues.apache.org/jira/browse/YARN-10598
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also imlemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> The queue type should hold these values: 
>  * Auto-created parent queue: *autoCreatedParent*
>  * Auto-created leaf queue: *autoCreatedLeaf*
>  * Static parent: *staticParent*
>  * Static leaf: *staticLeaf* 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

2021-01-26 Thread Benjamin Teke (Jira)
Benjamin Teke created YARN-10598:


 Summary: CS Flexible Auto Queue Creation: Modify RM /scheduler 
endpoint to extend the creation type with additional information
 Key: YARN-10598
 URL: https://issues.apache.org/jira/browse/YARN-10598
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Benjamin Teke
Assignee: Szilard Nemeth


Under this umbrella (YARN-10496), weight-mode has been implemented for CS with 
YARN-10504.

Auto-queue creation has been also imlemented with YARN-10506.

Connected to this effort, we would like to expose the type of the queue with 
the RM's /scheduler REST endpoint.

The queue type should hold these values: 
 * Auto-created parent queue: *autoCreatedParent*
 * Auto-created leaf queue: *autoCreatedLeaf*
 * Static parent: *staticParent*
 * Static leaf: *staticLeaf* 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled

2021-01-25 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271419#comment-17271419
 ] 

Benjamin Teke commented on YARN-10596:
--

[~gandras] thanks for the patch. I have a suggestion for making the condition a 
bit more readable (for me at least). Consider the following part:

{code:java}
boolean isEligibleForAutoQueueCreation = false;

// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
  isEligibleForAutoQueueCreation = ((ParentQueue) 
oldQueue).isDynamicQueue();
}

// if a queue is eligible for auto queue creation v2
// it must be a ParentQueue
if (!isEligibleForAutoQueueCreation) {
  isEligibleForAutoQueueCreation =
  conf.isAutoQueueCreationV2Enabled(fullQueueName);
}
{code}

Basically the condition is: if the queue is a dynamic parent, or it has the the 
AutoQueueCreationV2 enabled then it can be an empty parent. As the old 
autoQueueEnabled flag is parsed and stored as a boolean I think it would be 
nicer if you got the conf.isAutoQueueCreationV2Enabled there and kept the 
isDynamicParent intact (as isDynamicParent induces the eligibility for auto 
creation, however it is not the same). You could skip the (bit) misleading  if 
(!isEligibleForAutoQueueCreation) part, and instead of the if 
(childQueueNames.size() == 0 && !isEligibleForAutoQueueCreation) condition you 
could say that the queue should not have 0 childQueues, should not be a dynamic 
parent and should not be eligible for auto creation based on the config.

> Allow static definition of childless ParentQueues with auto-queue-creation-v2 
> enabled
> -
>
> Key: YARN-10596
> URL: https://issues.apache.org/jira/browse/YARN-10596
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10596.001.patch
>
>
> The old auto queue creation/managed queue logic allowed the definition of 
> childless parents to be created statically, if the auto-create-child-queue 
> flag was turned on the parent (thus making it a ManagedParentQueue).
> Since it is not an edge case, we also need to support the creation of a 
> ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, 
> even when no child queue is defined under the parent.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format

2021-01-25 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271371#comment-17271371
 ] 

Benjamin Teke edited comment on YARN-10585 at 1/25/21, 3:34 PM:


Thank [~shuzirra] for working on this. Based on your answers to [~gandras]'s 
comments the patch looks good to me, so generally +1 (non-binding) from my 
side. Another patch might be needed because the 002 generates license warnings 
and has a checkstyle issue.


was (Author: bteke):
Thank [~shuzirra] for working on this. Based on your answers to [~gandras]'s 
comments the patch looks good to me, so +1 (non-binding) from my side.

> Create a class which can convert from legacy mapping rule format to the new 
> JSON format
> ---
>
> Key: YARN-10585
> URL: https://issues.apache.org/jira/browse/YARN-10585
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10585.001.patch, YARN-10585.002.patch
>
>
> To make transition easier we need to create tooling to support the migration 
> effort. The first step is to create a class which can migrate from legacy to 
> the new JSON format.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format

2021-01-25 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271371#comment-17271371
 ] 

Benjamin Teke commented on YARN-10585:
--

Thank [~shuzirra] for working on this. Based on your answers to [~gandras]'s 
comments the patch looks good to me, so +1 (non-binding) from my side.

> Create a class which can convert from legacy mapping rule format to the new 
> JSON format
> ---
>
> Key: YARN-10585
> URL: https://issues.apache.org/jira/browse/YARN-10585
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10585.001.patch, YARN-10585.002.patch
>
>
> To make transition easier we need to create tooling to support the migration 
> effort. The first step is to create a class which can migrate from legacy to 
> the new JSON format.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation

2021-01-20 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10565:
-
Description: 
In YARN-10504 weight mode support was introduced to CS. This jira is a followup 
to simplify and restructure the initialization, so that the weight 
calculation/absolute/percentage mode is easier to understand and modify.

To be refactored:
* In ParentQueue.java#1099 the error message should be more specific, instead 
of the {{LOG.error("Fatal issue found: e", e);}}
* AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT 
just to be on the safe side
* Uncomment the commented assertions in 
TestCapacitySchedulerAutoCreatedQueueBase.validateEffectiveMinResource
* Check whether the assertion modification in TestRMWebServices is absolutely 
necessary or could be hiding a bug.
* Same for TestRMWebServicesForCSWithPartitions.java


  was:
In YARN-10504 weight mode support was introduced to CS. This jira is a followup 
to simplify and restructure the initialization, so that the weight 
calculation/absolute/percentage mode is easier to understand and modify.

To be refactored:
* In ParentQueue.java#1099 the error message should be more specific, instead 
of the {{LOG.error("Fatal issue found: e", e);}}
* AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT 
just to be on the safe side
* 



> Refactor CS queue initialization to simplify weight mode calculation
> 
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch, YARN-10565.002.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a 
> followup to simplify and restructure the initialization, so that the weight 
> calculation/absolute/percentage mode is easier to understand and modify.
> To be refactored:
> * In ParentQueue.java#1099 the error message should be more specific, instead 
> of the {{LOG.error("Fatal issue found: e", e);}}
> * AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT 
> just to be on the safe side
> * Uncomment the commented assertions in 
> TestCapacitySchedulerAutoCreatedQueueBase.validateEffectiveMinResource
> * Check whether the assertion modification in TestRMWebServices is absolutely 
> necessary or could be hiding a bug.
> * Same for TestRMWebServicesForCSWithPartitions.java



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation

2021-01-20 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10565:
-
Description: 
In YARN-10504 weight mode support was introduced to CS. This jira is a followup 
to simplify and restructure the initialization, so that the weight 
calculation/absolute/percentage mode is easier to understand and modify.

To be refactored:
* In ParentQueue.java#1099 the error message should be more specific, instead 
of the {{LOG.error("Fatal issue found: e", e);}}
* AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT 
just to be on the safe side
* 


  was:In YARN-10504 weight mode support was introduced to CS. This jira is a 
followup to simplify and restructure the initialization, so that the weight 
calculation/absolute/percentage mode is easier to understand and modify.


> Refactor CS queue initialization to simplify weight mode calculation
> 
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch, YARN-10565.002.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a 
> followup to simplify and restructure the initialization, so that the weight 
> calculation/absolute/percentage mode is easier to understand and modify.
> To be refactored:
> * In ParentQueue.java#1099 the error message should be more specific, instead 
> of the {{LOG.error("Fatal issue found: e", e);}}
> * AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT 
> just to be on the safe side
> * 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode and enable it by default

2021-01-18 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267417#comment-17267417
 ] 

Benjamin Teke commented on YARN-10573:
--

The findbugs issue should be solved by now (see YARN-10574) so retriggering the 
CI job should get rid of it.

> Enhance placement rule conversion in fs2cs in weight mode and enable it by 
> default
> --
>
> Key: YARN-10573
> URL: https://issues.apache.org/jira/browse/YARN-10573
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>  Labels: fs2cs
> Attachments: YARN-10573-001.patch, YARN-10573-002.patch, 
> YARN-10573-003.patch
>
>
> If we're using weight mode, we have much more freedom when it comes to 
> placement rules.
> We need to do three things in this ticket:
> #1
> In YARN-10525, weight conversion became the default in {{fs2cs}}. This also 
> means that we can support nested rules properly and also queues can be 
> created under {{root}}. 
> Therefore, a lot of warnings and validations inside 
> {{QueuePlacementConverter}} are not necessary and only relevant if the user 
> chose percentage-based conversion in the command line.
> #2
> Remove unnecessary stuff in the RuleHandler which are already supported (plus 
> the test code which affected by this):
> * SPECIFIED_NOT_FIRST
> * USER_MAX_APPS_DEFAULT
> * USER_MAX_RUNNING_APPS
> #3
> Currently, users have to use "\-m" or "\-\-convert\-placement\-rules" switch 
> to convert the placement rules from FS.
> Initially, we converted to the old mapping rule format, which has serious 
> limitations, so we disabled the automatic conversion.
> With the new JSON-based format and placement engine, this conversion should 
> happen automatically.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode and enable it by default

2021-01-18 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267388#comment-17267388
 ] 

Benjamin Teke edited comment on YARN-10573 at 1/18/21, 4:11 PM:


[~pbacsko] thanks for the patch. Went through it, it looks good to me. +1 
(non-binding). Let's see the CI results.


was (Author: bteke):
[~pbacsko] thanks for the patch. Went through it, it looks good to me. +1 
(non-binding).

> Enhance placement rule conversion in fs2cs in weight mode and enable it by 
> default
> --
>
> Key: YARN-10573
> URL: https://issues.apache.org/jira/browse/YARN-10573
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>  Labels: fs2cs
> Attachments: YARN-10573-001.patch, YARN-10573-002.patch, 
> YARN-10573-003.patch
>
>
> If we're using weight mode, we have much more freedom when it comes to 
> placement rules.
> We need to do three things in this ticket:
> #1
> In YARN-10525, weight conversion became the default in {{fs2cs}}. This also 
> means that we can support nested rules properly and also queues can be 
> created under {{root}}. 
> Therefore, a lot of warnings and validations inside 
> {{QueuePlacementConverter}} are not necessary and only relevant if the user 
> chose percentage-based conversion in the command line.
> #2
> Remove unnecessary stuff in the RuleHandler which are already supported (plus 
> the test code which affected by this):
> * SPECIFIED_NOT_FIRST
> * USER_MAX_APPS_DEFAULT
> * USER_MAX_RUNNING_APPS
> #3
> Currently, users have to use "\-m" or "\-\-convert\-placement\-rules" switch 
> to convert the placement rules from FS.
> Initially, we converted to the old mapping rule format, which has serious 
> limitations, so we disabled the automatic conversion.
> With the new JSON-based format and placement engine, this conversion should 
> happen automatically.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode and enable it by default

2021-01-18 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267388#comment-17267388
 ] 

Benjamin Teke commented on YARN-10573:
--

[~pbacsko] thanks for the patch. Went through it, it looks good to me. +1 
(non-binding).

> Enhance placement rule conversion in fs2cs in weight mode and enable it by 
> default
> --
>
> Key: YARN-10573
> URL: https://issues.apache.org/jira/browse/YARN-10573
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>  Labels: fs2cs
> Attachments: YARN-10573-001.patch, YARN-10573-002.patch, 
> YARN-10573-003.patch
>
>
> If we're using weight mode, we have much more freedom when it comes to 
> placement rules.
> We need to do three things in this ticket:
> #1
> In YARN-10525, weight conversion became the default in {{fs2cs}}. This also 
> means that we can support nested rules properly and also queues can be 
> created under {{root}}. 
> Therefore, a lot of warnings and validations inside 
> {{QueuePlacementConverter}} are not necessary and only relevant if the user 
> chose percentage-based conversion in the command line.
> #2
> Remove unnecessary stuff in the RuleHandler which are already supported (plus 
> the test code which affected by this):
> * SPECIFIED_NOT_FIRST
> * USER_MAX_APPS_DEFAULT
> * USER_MAX_RUNNING_APPS
> #3
> Currently, users have to use "\-m" or "\-\-convert\-placement\-rules" switch 
> to convert the placement rules from FS.
> Initially, we converted to the old mapping rule format, which has serious 
> limitations, so we disabled the automatic conversion.
> With the new JSON-based format and placement engine, this conversion should 
> happen automatically.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation

2021-01-15 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266036#comment-17266036
 ] 

Benjamin Teke commented on YARN-10506:
--

Thanks for working on this! Checked the latest patch, generally seems good to 
me. I have two minor nits about it:
* In AbstractCSQueue the field  {{lastSubmittedTimestamp}} is unused. It has a 
"setter" and a getter, but neither of them are called. Shouldn't this be 
removed to avoid confusion?
* The same goes for CapacitySchedulerAutoQueueHandler's conf field.



> Update queue creation logic to use weight mode and allow the flexible 
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch, 
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, 
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, 
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, 
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, 
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, 
> YARN-10506.014.patch
>
>
> The queue creation logic should be updated to use weight mode and support the 
> flexible creation. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation

2021-01-15 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10565:
-
Attachment: YARN-10565.002.patch

> Refactor CS queue initialization to simplify weight mode calculation
> 
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch, YARN-10565.002.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a 
> followup to simplify and restructure the initialization, so that the weight 
> calculation/absolute/percentage mode is easier to understand and modify.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation

2021-01-15 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke reassigned YARN-10565:


Assignee: Benjamin Teke

> Refactor CS queue initialization to simplify weight mode calculation
> 
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a 
> followup to simplify and restructure the initialization, so that the weight 
> calculation/absolute/percentage mode is easier to understand and modify.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS

2021-01-15 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266010#comment-17266010
 ] 

Benjamin Teke commented on YARN-10512:
--

Thank you [~snemeth] for working on this. 

Regarding the 2nd point in [~gandras]'s comment: as I see it 
{{getCapacityConfigurationTypeForQueues}} basically uses the same approach 
(check whether the QueueCapacities' weight or capacity is set), except it 
doesn't consider the queue's {{getCapacityConfigType()}} method. That method 
should be sufficient for differentiating ABSOLUTE and PERCENTAGE+WEIGHT mode, 
as {{AbstractCSQueue.updateEffectiveResources}} uses it as well. And the 
dynamic queues will have their weight set in their respective QueueCapacities 
object, as it holds the calculated values, not the ones read from the 
configuration. All in all I suggest to keep this patch as simple as possible 
and consider refactoring this after the YARN-10504 refactor is finished 
(YARN-10565).


> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include 
> mode of operation for CS
> --
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10512.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> We would like to expose the mode of operation with the RM's /scheduler REST 
> endpoint.
> The field name will be 'mode'.
> All queue representations in the response will be uniformly hold any of the 
> mode values of: "percentage", "absolute", "weight".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10496) [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler

2021-01-13 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264106#comment-17264106
 ] 

Benjamin Teke commented on YARN-10496:
--

[~pbacsko], regarding the max capacity: as of now YARN-10504 disabled the 
validation for the absolute and absolute max capacity of a queue. I think we 
should allow some flexibility by either introducing a flag or a special format 
like you mentioned. Couple of concerns/questions:
 * Should we allow the max capacity to be lower than the capacity?
 ** In "relative to the cluster" mode this can be straightforward, especially 
with weight mode, I can setup a quite large queue hierarchy with weights and 
not worry about any queue eating up large part of the cluster resources.
** In "relative to the parent" mode this can allow an option where the weights 
are basically disabled, and the queues are configured with the max capacity. 
Not necessarily a problem, but this can lead to hard to read configurations.
* If we keep/reintroduce the capacity < max capacity constraint in weight mode 
the user might have to calculate the percentages from weight manually. For 
example child1 and child2 are the only child queues under a parent with weights 
3 and 1. In this setup child1 has to have the configured max capacity as 75% 
while child2 can have anything above 25%. This is ok for a static parent, but 
if/when auto-create templates/wildcard configs will be supported the capacity 
can greatly change based on the number of dynamic queues. If I want to express 
the max capacity of any child as 33% of the parent's resources I will need to 
define at least 3 static queues with the same weight, I can't allow these to be 
auto created (because 1 queue with weight 1 will have the capacity 100%, 2 
queues with weight 1 will have 50%). This is another reason to let this 
constraint go.

> [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
> -
>
> Key: YARN-10496
> URL: https://issues.apache.org/jira/browse/YARN-10496
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: capacity scheduler
>Reporter: Wangda Tan
>Priority: Major
>
> CapacityScheduler today doesn’t support an auto queue creation which is 
> flexible enough. The current constraints: 
>  * Only leaf queues can be auto-created
>  * A parent can only have either static queues or dynamic ones. This causes 
> multiple constraints. For example:
>  * It isn’t possible to have a VIP user like Alice with a static queue 
> root.user.alice with 50% capacity while the other user queues (under 
> root.user) are created dynamically and they share the remaining 50% of 
> resources.
>  
>  * In comparison, FairScheduler allows the following scenarios, Capacity 
> Scheduler doesn’t:
>  ** This implies that there is no possibility to have both dynamically 
> created and static queues at the same time under root
>  * A new queue needs to be created under an existing parent, while the parent 
> already has static queues
>  * Nested queue mapping policy, like in the following example: 
> |
> 
> |
>  * Here two levels of queues may need to be created 
> If an application belongs to user _alice_ (who has the primary_group of 
> _engineering_), the scheduler checks whether _root.engineering_ exists, if it 
> doesn’t,  it’ll be created. Then scheduler checks whether 
> _root.engineering.alice_ exists, and creates it if it doesn't.
>  
> When we try to move users from FairScheduler to CapacityScheduler, we face 
> feature gaps which blocks users migrate from FS to CS.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation

2021-01-13 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264003#comment-17264003
 ] 

Benjamin Teke commented on YARN-10565:
--

Added most of the changes from YARN-10506.011 patch, as it was abandoned there.

> Refactor CS queue initialization to simplify weight mode calculation
> 
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a 
> followup to simplify and restructure the initialization, so that the weight 
> calculation/absolute/percentage mode is easier to understand and modify.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation

2021-01-13 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10565:
-
Attachment: YARN-10565.001.patch

> Refactor CS queue initialization to simplify weight mode calculation
> 
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a 
> followup to simplify and restructure the initialization, so that the weight 
> calculation/absolute/percentage mode is easier to understand and modify.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-10512) CS Flexible Auto Queue Creation Check RM REST API impact

2021-01-13 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke reassigned YARN-10512:


Assignee: Benjamin Teke

> CS Flexible Auto Queue Creation Check RM REST API impact
> 
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10525) Add weight mode conversion to fs2cs

2021-01-12 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263566#comment-17263566
 ] 

Benjamin Teke commented on YARN-10525:
--

Thanks [~pbacsko] for working on this. The patch LGTM (non-binding), and the 
tests are passing locally. [~snemeth] can you please retrigger the CI?

> Add weight mode conversion to fs2cs
> ---
>
> Key: YARN-10525
> URL: https://issues.apache.org/jira/browse/YARN-10525
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: zhuqi
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10525-001.patch, YARN-10525-002.patch, 
> YARN-10525-003.patch, YARN-10525-004.patch, YARN-10525-005.patch
>
>
> Weight mode will be added to Capacity Scheduler.
> Currently, we convert FS weights to percentages, however, it will be more 
> useful to keep those values and use them in CS as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262884#comment-17262884
 ] 

Benjamin Teke commented on YARN-10504:
--

I agree with your comment, so if it got an LGTM lets fix the remaining issues 
in a later patch. 

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.011.patch, 
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262720#comment-17262720
 ] 

Benjamin Teke commented on YARN-10504:
--

Uploaded patch 11 containing the said modification, however it causes an issue 
in 
TestAbsoluteResourceConfiguration.testSimpleMinMaxResourceConfigurartionPerQueue.
 I'll look into it, in the meantime patch 11 can be considered obsolete.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.011.patch, 
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10504:
-
Attachment: YARN-10504.011.patch

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.011.patch, 
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262597#comment-17262597
 ] 

Benjamin Teke commented on YARN-10565:
--

[~sunilg] created this item based on your 
[comment|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17262580=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17262580]
 on YARN-10504

> Refactor CS queue initialization to simplify weight mode calculation
> 
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Priority: Major
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a 
> followup to simplify and restructure the initialization, so that the weight 
> calculation/absolute/percentage mode is easier to understand and modify.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation

2021-01-11 Thread Benjamin Teke (Jira)
Benjamin Teke created YARN-10565:


 Summary: Refactor CS queue initialization to simplify weight mode 
calculation
 Key: YARN-10565
 URL: https://issues.apache.org/jira/browse/YARN-10565
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Benjamin Teke


In YARN-10504 weight mode support was introduced to CS. This jira is a followup 
to simplify and restructure the initialization, so that the weight 
calculation/absolute/percentage mode is easier to understand and modify.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
 ] 

Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:33 AM:
-

[~zhuqi],

Thanks for the detailed explanation on the root cause of the Absolute mode - 
auto creation test issue. I came to the same conclusion in my comment 
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
 so I agree with your suggestion. Will upload a patch containing it, if no 
objections.

Thanks [~sunilg] for the review. I'll create the followup jira.


was (Author: bteke):
[~zhuqi],

As for the root cause of the Absolute mode - auto creation test issue I came to 
the same conclusion in my comment 
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
 so I agree with your suggestion. Will upload a patch containing it, if no 
objections.

Thanks [~sunilg] for the review. I'll create the followup jira.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261549#comment-17261549
 ] 

Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:32 AM:
-

Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure: 
{{AutoCreatedLeafQueue#reinitializeFromTemplate}} was refactored, now the 
getting and merging the QueueCapacities happens *before* calling the 
{{ParentQueue#updateClusterResource}} (and 
{{LeafQueue#updateClusterResource}}). In {{LeafQueue#updateClusterResource}} 
the {{AbstractCSQueue#updateEffectiveResources}} is called where the 
effectiveMinResource of the created queue is overridden with the template's 
effectiveMinResources which is exactly the same the test is getting in the 
asserts.
{code:java}
  void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
  Resource resourceByLabel = labelManager.getResourceByLabel(label,
  clusterResource);  
 Resource minResource = queueResourceQuotas.getConfiguredMinResource(
  label);  // Update effective resource (min/max) to each child 
queue.
  if (getCapacityConfigType().equals(
  CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));


...{code}


was (Author: bteke):
Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure: 
{{AutoCreatedLeafQueue#reinitializeFromTemplate }}was refactored, now the 
getting and merging the QueueCapacities happens{{ *before* }}calling the{{ 
ParentQueue#updateClusterResource}} (and {{LeafQueue#updateClusterResource}}). 
In {{LeafQueue#updateClusterResource}} the 
AbstractCSQueue#updateEffectiveResources is called where the 
effectiveMinResource of the created queue is overridden with the template's 
effectiveMinResources which is exactly the same the test is getting in the 
asserts.
{code:java}
  void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
  Resource resourceByLabel = labelManager.getResourceByLabel(label,
  clusterResource);  Resource minResource = 
queueResourceQuotas.getConfiguredMinResource(
  label);  // Update effective resource (min/max) to each child 
queue.
  if (getCapacityConfigType().equals(
  CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));


...{code}

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261549#comment-17261549
 ] 

Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:31 AM:
-

Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure: 
{{AutoCreatedLeafQueue#reinitializeFromTemplate }}was refactored, now the 
getting and merging the QueueCapacities happens{{ *before* }}calling the{{ 
ParentQueue#updateClusterResource}} (and {{LeafQueue#updateClusterResource}}). 
In {{LeafQueue#updateClusterResource}} the 
AbstractCSQueue#updateEffectiveResources is called where the 
effectiveMinResource of the created queue is overridden with the template's 
effectiveMinResources which is exactly the same the test is getting in the 
asserts.
{code:java}
  void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
  Resource resourceByLabel = labelManager.getResourceByLabel(label,
  clusterResource);  Resource minResource = 
queueResourceQuotas.getConfiguredMinResource(
  label);  // Update effective resource (min/max) to each child 
queue.
  if (getCapacityConfigType().equals(
  CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));


...{code}


was (Author: bteke):
Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure: 
{{AutoCreatedLeafQueue#reinitializeFromTemplate }}was refactored, now the 
getting and merging the QueueCapacities happens *before* calling the 
{{ParentQueue#updateClusterResource}} (and 
{{LeafQueue#updateClusterResource}}). In {{LeafQueue#updateClusterResource 
}}the {{AbstractCSQueue#updateEffectiveResources }}is called where the 
effectiveMinResource of the created queue is overridden with the template's 
effectiveMinResources which is exactly the same the test is getting in the 
asserts.


{code:java}
  void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
  Resource resourceByLabel = labelManager.getResourceByLabel(label,
  clusterResource);  Resource minResource = 
queueResourceQuotas.getConfiguredMinResource(
  label);  // Update effective resource (min/max) to each child 
queue.
  if (getCapacityConfigType().equals(
  CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));


...{code}

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
 ] 

Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:29 AM:
-

[~zhuqi],

As for the root cause of the Absolute mode - auto creation test issue I came to 
the same conclusion in my comment 
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
 so I agree with your suggestion. Will upload a patch containing it, if no 
objections.

Thanks [~sunilg] for the review. I'll create the followup jira.


was (Author: bteke):
[~zhuqi],

As for the root cause of the Absolute mode - auto creation test issue I came to 
the same conclusion in my comment above, so I agree with your suggestion. Will 
upload a patch containing it, if no objections.

Thanks [~sunilg] for the review. I'll create the followup jira.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
 ] 

Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:29 AM:
-

[~zhuqi],

As for the root cause of the Absolute mode - auto creation test issue I came to 
the same conclusion in my comment above, so I agree with your suggestion. Will 
upload a patch containing it, if no objections.

Thanks [~sunilg] for the review. I'll create the followup jira.


was (Author: bteke):
[~zhuqi],

I came to the same conclusion in my comment 
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
 so I agree with your suggestion. Will upload a patch containing it, if no 
objections.

Thanks [~sunilg] for the review. I'll create the followup jira.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
 ] 

Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:28 AM:
-

[~zhuqi],

I came to the same conclusion in my comment 
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
 so I agree with your suggestion. Will upload a patch containing it, if no 
objections.

Thanks [~sunilg] for the review. I'll create the followup jira.


was (Author: bteke):
[~zhuqi],

I came to the same conclusion in my comment above, so I agree with your 
suggestion. Will upload a patch containing it, if no objections.

Thanks [~sunilg] for the review. I'll create the followup jira.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
 ] 

Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:27 AM:
-

[~zhuqi],

I came to the same conclusion in my comment above, so I agree with your 
suggestion. Will upload a patch containing it, if no objections.

Thanks [~sunilg] for the review. I'll create the followup jira.


was (Author: bteke):
[~zhuqi],

I came to the same conclusion in my comment 
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
 so I agree with your suggestion. Will upload a patch containing it, if no 
objections.

 

Thanks [~sunilg] for the review! I'll create the followup jira.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-11 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
 ] 

Benjamin Teke commented on YARN-10504:
--

[~zhuqi],

I came to the same conclusion in my comment 
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
 so I agree with your suggestion. Will upload a patch containing it, if no 
objections.

 

Thanks [~sunilg] for the review! I'll create the followup jira.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch, 
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-09 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261961#comment-17261961
 ] 

Benjamin Teke commented on YARN-10504:
--

[~wangda], [~zhuqi]

 

Added a small new patch. ver.6 had a small issue in 
ParentQueue.getCapacityConfigurationTypeForQueues, where if the passed queues 
collection was empty the iterator where the mixed mode check was implemented 
threw a NoSuchElementException, causing all of the AutoCreatedTests to fail. 
Also setChildQueues has a similar mixed mode check, where the root queue is 
included in the check, causing an unnecessary exception. So I extended that if 
as well. Additionally I added [~zhuqi]'s suggestion to 
LeafQueue.updateClusterResource.

There is one more issue: now 2 of the TestAbsoluteResourceWithAutoQueue tests 
are failing, because the apps are stuck in SUBMITTED state. [~wangda] do you 
have an idea on this one?

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-09 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10504:
-
Attachment: YARN-10504.007.patch

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.ver-1.patch, 
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-09 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261954#comment-17261954
 ] 

Benjamin Teke commented on YARN-10504:
--

[~wangda]/[~zhuqi]/[~gandras],

 

Let me take care of the suggestions in [~zhuqi]'s comment above. I'll base it 
on the ver.6 patch.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.006.patch, YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, 
> YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-08 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261549#comment-17261549
 ] 

Benjamin Teke commented on YARN-10504:
--

Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure: 
{{AutoCreatedLeafQueue#reinitializeFromTemplate }}was refactored, now the 
getting and merging the QueueCapacities happens *before* calling the 
{{ParentQueue#updateClusterResource}} (and 
{{LeafQueue#updateClusterResource}}). In {{LeafQueue#updateClusterResource 
}}the {{AbstractCSQueue#updateEffectiveResources }}is called where the 
effectiveMinResource of the created queue is overridden with the template's 
effectiveMinResources which is exactly the same the test is getting in the 
asserts.


{code:java}
  void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
  Resource resourceByLabel = labelManager.getResourceByLabel(label,
  clusterResource);  Resource minResource = 
queueResourceQuotas.getConfiguredMinResource(
  label);  // Update effective resource (min/max) to each child 
queue.
  if (getCapacityConfigType().equals(
  CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));


...{code}

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-08 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261381#comment-17261381
 ] 

Benjamin Teke commented on YARN-10504:
--

Hey [~zhuqi],

Thanks for your updates. Took the liberty to refactor/include your changes in 
my latest patch. Also based on our offline discussion with [~gandras] I removed 
some parts of his 002 patch, as it will be handled differently. As you have 
mentioned the AutoQueueCreation should now work, because the leafQueueTemplates 
should now have the correct absolute capacities. I will now turn to the 
AbsoluteResourceWithAutoQueue issue and handle the maxApplications in auto 
created cases.

If you upload a new patch please use the 005 patch as base, because that 
contains the latest from our side and also includes your changes from 003/004 
patch.

Thanks!

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-08 Thread Benjamin Teke (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Teke updated YARN-10504:
-
Attachment: YARN-10504.005.patch

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch, 
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-06 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260292#comment-17260292
 ] 

Benjamin Teke edited comment on YARN-10504 at 1/7/21, 7:53 AM:
---

[~wangda],

 

Regarding the AutoCreatedLeafQueue failures: we are looking into the issue. 
Currently it seems like the way the mock setup is structured the 
updateClusterResource is not called at the correct time. We'll update the patch 
once the issue is verified and fixed.


was (Author: bteke):
[~wangda],

 

Regarding the AutoCreatedLeafQueue: we are looking into the issue. Cuurently it 
seems like the way the mock setup is structured the updateClusterResource is 
not called at the correct time. We'll update the patch once the issue is fixed.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler

2021-01-06 Thread Benjamin Teke (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260292#comment-17260292
 ] 

Benjamin Teke commented on YARN-10504:
--

[~wangda],

 

Regarding the AutoCreatedLeafQueue: we are looking into the issue. Cuurently it 
seems like the way the mock setup is structured the updateClusterResource is 
not called at the correct time. We'll update the patch once the issue is fixed.

> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch, 
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a 
> weight mode should be introduced. The existing \{{capacity }}property should 
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the 
> weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



  1   2   3   >