[jira] [Commented] (YARN-7769) FS QueueManager should not create default queue at init
[ https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320963#comment-17320963 ] Benjamin Teke commented on YARN-7769: - And I fixed the checkstyle issues in the latest patch. > FS QueueManager should not create default queue at init > --- > > Key: YARN-7769 > URL: https://issues.apache.org/jira/browse/YARN-7769 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-7769.001.patch, YARN-7769.002.patch, > YARN-7769.003.patch > > > Currently the FairScheduler QueueManager automatically creates the default > queue. However the default queue does not need to exist. We have two possible > cases which we should handle: > * Based on the placement rule "Default" the name for the default queue might > not be default and it should be created with a different name > * There might not be a "Default" placement rule at all which removes the need > to create the queue. > We should leave the creation of the default queue to the point in time that > we can assess if it is needed or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7769) FS QueueManager should not create default queue at init
[ https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-7769: Attachment: YARN-7769.003.patch > FS QueueManager should not create default queue at init > --- > > Key: YARN-7769 > URL: https://issues.apache.org/jira/browse/YARN-7769 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-7769.001.patch, YARN-7769.002.patch, > YARN-7769.003.patch > > > Currently the FairScheduler QueueManager automatically creates the default > queue. However the default queue does not need to exist. We have two possible > cases which we should handle: > * Based on the placement rule "Default" the name for the default queue might > not be default and it should be created with a different name > * There might not be a "Default" placement rule at all which removes the need > to create the queue. > We should leave the creation of the default queue to the point in time that > we can assess if it is needed or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7769) FS QueueManager should not create default queue at init
[ https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320806#comment-17320806 ] Benjamin Teke commented on YARN-7769: - Yes, I missed those, thanks. Uploaded a new patch, let's see the CI results now. > FS QueueManager should not create default queue at init > --- > > Key: YARN-7769 > URL: https://issues.apache.org/jira/browse/YARN-7769 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-7769.001.patch, YARN-7769.002.patch > > > Currently the FairScheduler QueueManager automatically creates the default > queue. However the default queue does not need to exist. We have two possible > cases which we should handle: > * Based on the placement rule "Default" the name for the default queue might > not be default and it should be created with a different name > * There might not be a "Default" placement rule at all which removes the need > to create the queue. > We should leave the creation of the default queue to the point in time that > we can assess if it is needed or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7769) FS QueueManager should not create default queue at init
[ https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-7769: Attachment: YARN-7769.002.patch > FS QueueManager should not create default queue at init > --- > > Key: YARN-7769 > URL: https://issues.apache.org/jira/browse/YARN-7769 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-7769.001.patch, YARN-7769.002.patch > > > Currently the FairScheduler QueueManager automatically creates the default > queue. However the default queue does not need to exist. We have two possible > cases which we should handle: > * Based on the placement rule "Default" the name for the default queue might > not be default and it should be created with a different name > * There might not be a "Default" placement rule at all which removes the need > to create the queue. > We should leave the creation of the default queue to the point in time that > we can assess if it is needed or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7769) FS QueueManager should not create default queue at init
[ https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-7769: Attachment: YARN-7769.001.patch > FS QueueManager should not create default queue at init > --- > > Key: YARN-7769 > URL: https://issues.apache.org/jira/browse/YARN-7769 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-7769.001.patch > > > Currently the FairScheduler QueueManager automatically creates the default > queue. However the default queue does not need to exist. We have two possible > cases which we should handle: > * Based on the placement rule "Default" the name for the default queue might > not be default and it should be created with a different name > * There might not be a "Default" placement rule at all which removes the need > to create the queue. > We should leave the creation of the default queue to the point in time that > we can assess if it is needed or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7769) FS QueueManager should not create default queue at init
[ https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-7769: --- Assignee: Benjamin Teke (was: Wilfred Spiegelenburg) > FS QueueManager should not create default queue at init > --- > > Key: YARN-7769 > URL: https://issues.apache.org/jira/browse/YARN-7769 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Benjamin Teke >Priority: Major > > Currently the FairScheduler QueueManager automatically creates the default > queue. However the default queue does not need to exist. We have two possible > cases which we should handle: > * Based on the placement rule "Default" the name for the default queue might > not be default and it should be created with a different name > * There might not be a "Default" placement rule at all which removes the need > to create the queue. > We should leave the creation of the default queue to the point in time that > we can assess if it is needed or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7769) FS QueueManager should not create default queue at init
[ https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319537#comment-17319537 ] Benjamin Teke commented on YARN-7769: - I see no activity on this for a while now, so I'll take this over, if you don't have any objections [~wilfreds]. > FS QueueManager should not create default queue at init > --- > > Key: YARN-7769 > URL: https://issues.apache.org/jira/browse/YARN-7769 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg >Priority: Major > > Currently the FairScheduler QueueManager automatically creates the default > queue. However the default queue does not need to exist. We have two possible > cases which we should handle: > * Based on the placement rule "Default" the name for the default queue might > not be default and it should be created with a different name > * There might not be a "Default" placement rule at all which removes the need > to create the queue. > We should leave the creation of the default queue to the point in time that > we can assess if it is needed or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10676) Improve code quality in TestTimelineAuthenticationFilterForV1
[ https://issues.apache.org/jira/browse/YARN-10676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296138#comment-17296138 ] Benjamin Teke commented on YARN-10676: -- Thanks [~snemeth] for the patch. LGTM (non-binding) from my side as well. > Improve code quality in TestTimelineAuthenticationFilterForV1 > - > > Key: YARN-10676 > URL: https://issues.apache.org/jira/browse/YARN-10676 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-10676.001.patch > > > - In testcase "testDelegationTokenOperations", the exception message is > checked but in case it does not match the assertion, the exception is not > printed. This happens 3 times. > - Assertion messages can be added > - Fields called "httpSpnegoKeytabFile" and "httpSpnegoPrincipal" can be > static final. > - There's a typo in comment "avaiable" (happens 2 times) > - There are some Assert.fail() calls, without messages. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10642) AsyncDispatcher will stuck introduced by YARN-8995.
[
https://issues.apache.org/jira/browse/YARN-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294633#comment-17294633
]
Benjamin Teke edited comment on YARN-10642 at 3/3/21, 4:17 PM:
---
[~zhengchenyu], looked at the patch. Can you please tend to the checkstyle
issue? Otherwise LGTM (non-binding).
[~pbacsko] or [~snemeth] if you have the time can you take a look at it?
was (Author: bteke):
[~zhengchenyu], looked at the patch, LGTM (non-binding).
[~pbacsko] or [~snemeth] if you have the time can you take a look at it?
> AsyncDispatcher will stuck introduced by YARN-8995.
> ---
>
> Key: YARN-10642
> URL: https://issues.apache.org/jira/browse/YARN-10642
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Critical
> Attachments: MockForDeadLoop.java, YARN-10642.001.patch,
> YARN-10642.002.patch, YARN-10642.003.patch, deadloop.png, debugfornode.png,
> put.png, take.png
>
>
> In our cluster, ResouceManager stuck twice within twenty days. Yarn client
> can't submit application. I got jstack info at second time, then found the
> reason.
> I analyze all the jstack, I found many thread stuck because can't get
> LinkedBlockingQueue.putLock. (Note: Sorry for limited space , omit the
> analytical process)
> The reason is that one thread hold the putLock all the time,
> printEventQueueDetails will called forEachRemaining, then hold putLock and
> readLock. The AsyncDispatcher will stuck.
> {code}
> Thread 6526 (IPC Server handler 454 on default port 8030):
> State: RUNNABLE
> Blocked count: 29988
> Waited count: 2035029
> Stack:
>
> java.util.concurrent.LinkedBlockingQueue$LBQSpliterator.forEachRemaining(LinkedBlockingQueue.java:926)
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
>
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.printEventQueueDetails(AsyncDispatcher.java:270)
>
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:295)
>
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:408)
>
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:215)
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
>
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
>
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:432)
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
>
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1040)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:958)
> java.security.AccessController.doPrivileged(Native Method)
> {code}
> I analyze LinkedBlockingQueue's source code. I found forEachRemaining in
> LinkedBlockingQueue.LBQSpliterator may stuck, when forEachRemaining and take
> are called in different thread.
> YARN-8995 introduce printEventQueueDetails method,
> "eventQueue.stream().collect" will called forEachRemaining method.
> Let's see why？ "put.png" shows that how to put("a"), "take.png" shows that
> how to take()。Specical Node: The removed Node will point itself for help gc!!!
> The key point code is in forEachRemaining, we see LBQSpliterator use
> forEachRemaining to visit all Node. But when got item value from Node, will
> release the lock. If at this time, take() will be called.
> The variable 'p' in forEachRemaining may point a Node which point itself,
> then forEachRemaining will be in dead loop. You can see it in "deadloop.png"
> Let's see a simple uni-test, Let's forEachRemaining called more slow than
> take, the problem will reproduction。uni-test is MockForDeadLoop.java.
> I debug MockForDeadLoop.java, and see a Node
[jira] [Commented] (YARN-10642) AsyncDispatcher will stuck introduced by YARN-8995.
[
https://issues.apache.org/jira/browse/YARN-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294633#comment-17294633
]
Benjamin Teke commented on YARN-10642:
--
[~zhengchenyu], looked at the patch, LGTM (non-binding).
[~pbacsko] or [~snemeth] if you have the time can you take a look at it?
> AsyncDispatcher will stuck introduced by YARN-8995.
> ---
>
> Key: YARN-10642
> URL: https://issues.apache.org/jira/browse/YARN-10642
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Critical
> Attachments: MockForDeadLoop.java, YARN-10642.001.patch,
> YARN-10642.002.patch, YARN-10642.003.patch, deadloop.png, debugfornode.png,
> put.png, take.png
>
>
> In our cluster, ResouceManager stuck twice within twenty days. Yarn client
> can't submit application. I got jstack info at second time, then found the
> reason.
> I analyze all the jstack, I found many thread stuck because can't get
> LinkedBlockingQueue.putLock. (Note: Sorry for limited space , omit the
> analytical process)
> The reason is that one thread hold the putLock all the time,
> printEventQueueDetails will called forEachRemaining, then hold putLock and
> readLock. The AsyncDispatcher will stuck.
> {code}
> Thread 6526 (IPC Server handler 454 on default port 8030):
> State: RUNNABLE
> Blocked count: 29988
> Waited count: 2035029
> Stack:
>
> java.util.concurrent.LinkedBlockingQueue$LBQSpliterator.forEachRemaining(LinkedBlockingQueue.java:926)
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
>
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.printEventQueueDetails(AsyncDispatcher.java:270)
>
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:295)
>
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:408)
>
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:215)
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
>
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
>
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:432)
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
>
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1040)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:958)
> java.security.AccessController.doPrivileged(Native Method)
> {code}
> I analyze LinkedBlockingQueue's source code. I found forEachRemaining in
> LinkedBlockingQueue.LBQSpliterator may stuck, when forEachRemaining and take
> are called in different thread.
> YARN-8995 introduce printEventQueueDetails method,
> "eventQueue.stream().collect" will called forEachRemaining method.
> Let's see why？ "put.png" shows that how to put("a"), "take.png" shows that
> how to take()。Specical Node: The removed Node will point itself for help gc!!!
> The key point code is in forEachRemaining, we see LBQSpliterator use
> forEachRemaining to visit all Node. But when got item value from Node, will
> release the lock. If at this time, take() will be called.
> The variable 'p' in forEachRemaining may point a Node which point itself,
> then forEachRemaining will be in dead loop. You can see it in "deadloop.png"
> Let's see a simple uni-test, Let's forEachRemaining called more slow than
> take, the problem will reproduction。uni-test is MockForDeadLoop.java.
> I debug MockForDeadLoop.java, and see a Node point itself. You can see pic
> "debugfornode.png"
> Environment:
> OS: CentOS Linux release 7.5.1804 (Core)
> JDK: jdk1.8.0_281
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (YARN-10642) AsyncDispatcher will stuck introduced by YARN-8995.
[
https://issues.apache.org/jira/browse/YARN-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293863#comment-17293863
]
Benjamin Teke commented on YARN-10642:
--
[~zhengchenyu] thanks working on this. Moved it to Submit Patch state to
trigger CI. Also I'll look at it tomorrow.
> AsyncDispatcher will stuck introduced by YARN-8995.
> ---
>
> Key: YARN-10642
> URL: https://issues.apache.org/jira/browse/YARN-10642
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Critical
> Attachments: MockForDeadLoop.java, YARN-10642.001.patch,
> YARN-10642.002.patch, YARN-10642.003.patch, deadloop.png, debugfornode.png,
> put.png, take.png
>
>
> In our cluster, ResouceManager stuck twice within twenty days. Yarn client
> can't submit application. I got jstack info at second time, then found the
> reason.
> I analyze all the jstack, I found many thread stuck because can't get
> LinkedBlockingQueue.putLock. (Note: Sorry for limited space , omit the
> analytical process)
> The reason is that one thread hold the putLock all the time,
> printEventQueueDetails will called forEachRemaining, then hold putLock and
> readLock. The AsyncDispatcher will stuck.
> {code}
> Thread 6526 (IPC Server handler 454 on default port 8030):
> State: RUNNABLE
> Blocked count: 29988
> Waited count: 2035029
> Stack:
>
> java.util.concurrent.LinkedBlockingQueue$LBQSpliterator.forEachRemaining(LinkedBlockingQueue.java:926)
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
>
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
>
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.printEventQueueDetails(AsyncDispatcher.java:270)
>
> org.apache.hadoop.yarn.event.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:295)
>
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.handleProgress(DefaultAMSProcessor.java:408)
>
> org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:215)
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor.allocate(DisabledPlacementProcessor.java:75)
>
> org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92)
>
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:432)
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
>
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1040)
> org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:958)
> java.security.AccessController.doPrivileged(Native Method)
> {code}
> I analyze LinkedBlockingQueue's source code. I found forEachRemaining in
> LinkedBlockingQueue.LBQSpliterator may stuck, when forEachRemaining and take
> are called in different thread.
> YARN-8995 introduce printEventQueueDetails method,
> "eventQueue.stream().collect" will called forEachRemaining method.
> Let's see why？ "put.png" shows that how to put("a"), "take.png" shows that
> how to take()。Specical Node: The removed Node will point itself for help gc!!!
> The key point code is in forEachRemaining, we see LBQSpliterator use
> forEachRemaining to visit all Node. But when got item value from Node, will
> release the lock. If at this time, take() will be called.
> The variable 'p' in forEachRemaining may point a Node which point itself,
> then forEachRemaining will be in dead loop. You can see it in "deadloop.png"
> Let's see a simple uni-test, Let's forEachRemaining called more slow than
> take, the problem will reproduction。uni-test is MockForDeadLoop.java.
> I debug MockForDeadLoop.java, and see a Node point itself. You can see pic
> "debugfornode.png"
> Environment:
> OS: CentOS Linux release 7.5.1804 (Core)
> JDK: jdk1.8.0_281
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293008#comment-17293008 ] Benjamin Teke commented on YARN-10532: -- [~zhuqi], Thanks for the patch. Generally looks good to me, but the latest patch has a whitespace error in TestAutoCreatedQueueDeletionPolicy.java, the CI will most likely complain about that. Otherwise if you upload a new patch, can you please also remove the unused Supplier imports (where applicable, TestAutoCreatedQueueDeletionPolicy and TestCapacitySchedulerNewQueueAutoCreation have it) and close the mockRM in TestSchedulingMonitor.testRMUpdateAutoCreatedQueueDeletionPolicy? Thanks! > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch, YARN-10532.008.patch, > YARN-10532.009.patch, YARN-10532.010.patch, YARN-10532.011.patch, > YARN-10532.012.patch, YARN-10532.013.patch, YARN-10532.014.patch, > YARN-10532.015.patch, YARN-10532.016.patch, YARN-10532.017.patch, > YARN-10532.018.patch, YARN-10532.019.patch, YARN-10532.020.patch, > YARN-10532.021.patch, YARN-10532.022.patch, YARN-10532.023.patch, > YARN-10532.024.patch, YARN-10532.025.patch, image-2021-02-12-21-32-02-267.png > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.006.patch
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch,
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " +
[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291759#comment-17291759
]
Benjamin Teke commented on YARN-10627:
--
[~pbacsko] sure, fixed it.
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch,
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers
[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290894#comment-17290894
]
Benjamin Teke commented on YARN-10627:
--
Thanks [~pbacsko] and [~gandras] for the review. Fixed the checkstyle issues,
added the rm close methods, and removed the unnecessary GB string. The current
tests include cases for capacity related log string generation (the used
configurations are mixed mode, they have both capacity and weight in the
hierarchy), this is why I didn't feel the need to introduce separate cases. Do
you think it should be done?
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch,
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " +
[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.005.patch
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch,
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " + getState()
> +
[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.004.patch
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> YARN-10627.003.patch, YARN-10627.004.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " + getState()
> + " [= configuredState ]" +
[jira] [Updated] (YARN-10640) Adjust the queue Configured capacity to Configured weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10640: - Summary: Adjust the queue Configured capacity to Configured weight number for weight mode in UI. (was: Ajust the queue Configured capacity to Configured weight number for weight mode in UI.) > Adjust the queue Configured capacity to Configured weight number for weight > mode in UI. > > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10640.001.patch, YARN-10640.002.patch, > image-2021-02-20-11-21-50-306.png, image-2021-02-20-14-18-56-261.png, > image-2021-02-20-14-19-30-767.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0. I think this should change to Configured Weight if > we use weight mode, this will be helpful. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10640) Ajust the queue Configured capacity to Configured weight number for weight mode in UI.
[ https://issues.apache.org/jira/browse/YARN-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289902#comment-17289902 ] Benjamin Teke commented on YARN-10640: -- [~zhuqi], thanks for working on this. LGTM, +1 (non-binding) > Ajust the queue Configured capacity to Configured weight number for weight > mode in UI. > --- > > Key: YARN-10640 > URL: https://issues.apache.org/jira/browse/YARN-10640 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10640.001.patch, YARN-10640.002.patch, > image-2021-02-20-11-21-50-306.png, image-2021-02-20-14-18-56-261.png, > image-2021-02-20-14-19-30-767.png > > > In weight mode: > Both the weight mode static queue and the dynamic queue will show the > Configured Capacity to 0. I think this should change to Configured Weight if > we use weight mode, this will be helpful. > Such as in dynamic weight mode queue: > !image-2021-02-20-11-21-50-306.png|width=528,height=374! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288509#comment-17288509
]
Benjamin Teke commented on YARN-10627:
--
[~zhuqi]
Uploaded a new patchset with the added UTs.
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> YARN-10627.003.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " + getState()
>
[jira] [Commented] (YARN-10641) Refactor the max app related update, and fix maxApllications update error when add new queues.
[
https://issues.apache.org/jira/browse/YARN-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288456#comment-17288456
]
Benjamin Teke commented on YARN-10641:
--
[~zhuqi], thanks for working on this, this indeed seems to be an issue.
I've checked the latest patch, and I've a question. Consider the following code
snippet (AbstractCSQueue#updateMaxAppRelatedField):
{code:java}
if (maxApplications < 0) {
int maxGlobalPerQueueApps = conf.getGlobalMaximumApplicationsPerQueue();
if (maxGlobalPerQueueApps > 0) {
// In absolute mode ,should shrink when change to corresponding
capacity.
maxApplications = this.capacityConfigType !=
CapacityConfigType.ABSOLUTE_RESOURCE ?
maxGlobalPerQueueApps : Math.max(maxApplications,
(int) (conf.getGlobalMaximumApplicationsPerQueue() * queueCapacities
.getAbsoluteCapacity(label)));
} else{
//Here we should get the max of label abs capacites.
maxApplications = Math.max(maxApplications,
(int) (conf.getMaximumSystemApplications() * queueCapacities
.getAbsoluteCapacity(label)));
}
}
{code}
In the if block is the second part (the Math.max call) actually necessary in
the ternary operator? Because in this block the *maxApplications < 0* condition
is already true.
Also in the same method at the log.info call there seems to be a debug message
(_update max app related_). Is it intentional?
> Refactor the max app related update, and fix maxApllications update error
> when add new queues.
> --
>
> Key: YARN-10641
> URL: https://issues.apache.org/jira/browse/YARN-10641
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Critical
> Attachments: YARN-10641.001.patch, YARN-10641.002.patch,
> YARN-10641.003.patch, image-2021-02-20-15-49-58-677.png,
> image-2021-02-20-15-53-51-099.png, image-2021-02-20-15-55-44-780.png,
> image-2021-02-20-16-29-18-519.png, image-2021-02-20-16-31-13-714.png
>
>
> When refactor the update logic in YARN-10504 .
> The update max applications based abs/cap is wrong, this should be fixed,
> because the max applications is key part to limit applications in CS.
> For example:
> When adding a dynamic queue, the other children's max app of parent queue are
> not updated correctly:
> !image-2021-02-20-15-53-51-099.png|width=639,height=509!
> The new added queue's max app will updated correctly:
> !image-2021-02-20-15-55-44-780.png|width=542,height=426!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10646) TestCapacitySchedulerWeightMode test descriptor comments doesn't reflect the correct scenario
Benjamin Teke created YARN-10646:
Summary: TestCapacitySchedulerWeightMode test descriptor comments
doesn't reflect the correct scenario
Key: YARN-10646
URL: https://issues.apache.org/jira/browse/YARN-10646
Project: Hadoop YARN
Issue Type: Sub-task
Environment:
Reporter: Benjamin Teke
Assignee: Benjamin Teke
There is a mixup in TestCapacitySchedulerWeightMode in the configuration
creator method comments and the test case descriptor comments. See the
following code:
{code:java}
/*
* Queue structure:
* root (*)
* ___
* / \
* a x(=100%), y(50%) b y(=50%), z(=100%)
* __
* / / \
* a1 ([x,y]: w=100)b1(no) b2([y,z]: w=100)
*
* Parent uses weight, child uses percentage
*/
public static Configuration getCSConfWithLabelsParentUsePctChildUseWeight(
Configuration config) {
{code}
While inside the method all the queues (including the second level ones) are
configured with capacity, only some labels are configured with weights:
{code:java}
conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "x", 100);
conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "y", 100);
conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "z", 100);
...
conf.setQueues(A, new String[] { "a1" });
conf.setCapacityByLabel(A1, RMNodeLabelsManager.NO_LABEL, 100);
conf.setMaximumCapacity(A1, 100);
conf.setAccessibleNodeLabels(A1, toSet("x", "y"));
conf.setDefaultNodeLabelExpression(A1, "x");
conf.setCapacityByLabel(A1, "x", 100);
conf.setCapacityByLabel(A1, "y", 100);
conf.setQueues(B, new String[] { "b1", "b2" });
conf.setCapacityByLabel(B1, RMNodeLabelsManager.NO_LABEL, 50);
conf.setMaximumCapacity(B1, 50);
conf.setAccessibleNodeLabels(B1, RMNodeLabelsManager.EMPTY_STRING_SET);
conf.setCapacityByLabel(B2, RMNodeLabelsManager.NO_LABEL, 50);
conf.setMaximumCapacity(B2, 50);
conf.setAccessibleNodeLabels(B2, toSet("y", "z"));
conf.setCapacityByLabel(B2, "y", 100);
conf.setCapacityByLabel(B2, "z", 100);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.003.patch
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> YARN-10627.003.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " + getState()
> + " [= configuredState ]" + "\n" + "acls = " +
[jira] [Commented] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287213#comment-17287213
]
Benjamin Teke commented on YARN-10627:
--
[~zhuqi], indeed, thanks for finding it. Fixed it with the second patch, will
introduce some tests for it in patch 003.
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [=
[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.002.patch
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " + getState()
> + " [= configuredState ]" + "\n" + "acls = " + aclsString
>
[jira] [Commented] (YARN-10636) CS Auto Queue creation should reject submissions with empty path parts
[ https://issues.apache.org/jira/browse/YARN-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286949#comment-17286949 ] Benjamin Teke commented on YARN-10636: -- Thanks [~shuzirra]. I too think the new test case should be introduced in TestCapacitySchedulerNewQueueAutoCreation. If you fix the checkstyle issue could you move it to that class? Thanks! > CS Auto Queue creation should reject submissions with empty path parts > -- > > Key: YARN-10636 > URL: https://issues.apache.org/jira/browse/YARN-10636 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10636.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10635) CSMapping rule can return paths with empty parts
[ https://issues.apache.org/jira/browse/YARN-10635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286947#comment-17286947 ] Benjamin Teke commented on YARN-10635: -- [~shuzirra] Thanks for the patch. I agree with [~gandras]. LGTM with that minor thing. > CSMapping rule can return paths with empty parts > > > Key: YARN-10635 > URL: https://issues.apache.org/jira/browse/YARN-10635 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10635.001.patch, YARN-10635.002.patch, > YARN-10635.003.patch > > > When a variable to be substituted evaluates to empty string, we might result > with paths where one of the parts is empty, these paths are obviously > problematic, but sometimes (when the path includes a dynamicParent) we accept > them as valid paths instead of getting the fallback action of the rule. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications
[ https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286547#comment-17286547 ] Benjamin Teke commented on YARN-10513: -- [~gandras], Thanks for the update. This revision is LGTM as well. > CS Flexible Auto Queue Creation RM UIv2 modifications > - > > Key: YARN-10513 > URL: https://issues.apache.org/jira/browse/YARN-10513 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: Screenshot 2021-02-04 at 12.54.25.png, Screenshot > 2021-02-04 at 12.54.52.png, Screenshot 2021-02-04 at 12.55.10.png, Screenshot > 2021-02-08 at 10.34.32.png, Screenshot 2021-02-17 at 15.22.30.png, > YARN-10513.001.patch, YARN-10513.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)
[ https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286354#comment-17286354 ] Benjamin Teke commented on YARN-10609: -- Thanks [~zhuqi], the latest patch looks good to me. > Update the document for YARN-10531(Be able to disable user limit factor for > CapacityScheduler Leaf Queue) > - > > Key: YARN-10609 > URL: https://issues.apache.org/jira/browse/YARN-10609 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10609.001.patch, YARN-10609.002.patch, > YARN-10609.003.patch, YARN-10609.004.patch, YARN-10609.005.patch > > > Since we have finished YARN-10531. > We should update the corresponding document. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10627:
-
Attachment: YARN-10627.001.patch
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10627.001.patch
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [= currentNumContainers ]" + "\n" + "state = " + getState()
> + " [= configuredState ]" + "\n" + "acls = " + aclsString
> + " [= configuredAcls ]" + "\n"
> +
[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)
[ https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285900#comment-17285900 ] Benjamin Teke commented on YARN-10609: -- [~zhuqi] thanks for the update. There may have been a mixup as the sentences with the "default value" and the "specified as float" are duplicated. If you remove those it's +1 from my side. Thanks! User limit factor provides a way to control the max amount of resources that a single user can consume. It is the multiple of the queue's capacity. By default this is set to 1 which ensures that a single user can never take more than the queue's configured capacity irrespective of how idle the cluster is. Increasing it means a single user can use more than the minimum capacity of the cluster, while decreasing it results in lower maximum resources. -By default this is set to 1 which ensures that a single user can never take more than the queue's configured capacity irrespective of how idle the cluster is. Value is specified as a float.- Setting this to -1 will disable the feature. Value is specified as a float. Note: using the flexible auto queue creation (yarn.scheduler.capacity..auto-queue-creation-v2) with weights will automatically set this property to -1, as the dynamic queues will be created with the hardcoded weight of 1 and in idle cluster scenarios they should be able to use more resources than calculated. > Update the document for YARN-10531(Be able to disable user limit factor for > CapacityScheduler Leaf Queue) > - > > Key: YARN-10609 > URL: https://issues.apache.org/jira/browse/YARN-10609 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10609.001.patch, YARN-10609.002.patch, > YARN-10609.003.patch > > > Since we have finished YARN-10531. > We should update the corresponding document. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications
[ https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285222#comment-17285222 ] Benjamin Teke edited comment on YARN-10513 at 2/16/21, 2:15 PM: [~gandras], thanks for working on this. LGTM as well. cc: [~pbacsko] [~snemeth] if you have the time for a review. was (Author: bteke): [~gandras], thanks for working on this. LGTM as well. cc: [~snemeth] if you have the time for a review. > CS Flexible Auto Queue Creation RM UIv2 modifications > - > > Key: YARN-10513 > URL: https://issues.apache.org/jira/browse/YARN-10513 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: Screenshot 2021-02-04 at 12.54.25.png, Screenshot > 2021-02-04 at 12.54.52.png, Screenshot 2021-02-04 at 12.55.10.png, Screenshot > 2021-02-08 at 10.34.32.png, YARN-10513.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications
[ https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285222#comment-17285222 ] Benjamin Teke commented on YARN-10513: -- [~gandras], thanks for working on this. LGTM as well. cc: [~snemeth] if you have the time for a review. > CS Flexible Auto Queue Creation RM UIv2 modifications > - > > Key: YARN-10513 > URL: https://issues.apache.org/jira/browse/YARN-10513 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: Screenshot 2021-02-04 at 12.54.25.png, Screenshot > 2021-02-04 at 12.54.52.png, Screenshot 2021-02-04 at 12.55.10.png, Screenshot > 2021-02-08 at 10.34.32.png, YARN-10513.001.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)
[ https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285211#comment-17285211 ] Benjamin Teke commented on YARN-10609: -- [~zhuqi] Thanks! Sorry, I have one more thing :) Maybe if we touch this we can rephrase the first sentence a bit: _The multiple of the queue capacity which can be configured to allow a single user to acquire more resources. _ To me this misses the point that setting this to below 1 limits the users resources. I think a phrasing like the following would be clearer: _User limit factor provides a way to control the max amount of resources that a single user can consume. It is the multiple of the queue's capacity. By default this is set to 1 which ensures that a single user can never take more than the queue's configured capacity irrespective of how idle the cluster is. Increasing it means a single user can use more than the minimum capacity of the cluster, while decreasing it results in lower maximum resources._ What's your opinion about this? > Update the document for YARN-10531(Be able to disable user limit factor for > CapacityScheduler Leaf Queue) > - > > Key: YARN-10609 > URL: https://issues.apache.org/jira/browse/YARN-10609 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10609.001.patch, YARN-10609.002.patch > > > Since we have finished YARN-10531. > We should update the corresponding document. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10627) Extend logging to give more information about weight mode
Benjamin Teke created YARN-10627:
Summary: Extend logging to give more information about weight mode
Key: YARN-10627
URL: https://issues.apache.org/jira/browse/YARN-10627
Project: Hadoop YARN
Issue Type: Sub-task
Components: yarn
Reporter: Benjamin Teke
Assignee: Benjamin Teke
In YARN-10504 weight mode was added, however the logged information about the
created queues or the toString methods weren't updated accordingly. Some
examples:
ParentQueue#setupQueueConfigs:
{code:java}
LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
+ ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
+ ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
+ ", absoluteMaxCapacity=" + this.queueCapacities
.getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
+ aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
+ ", reservationsContinueLooking=" + reservationsContinueLooking
+ ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
+ ", priority=" + priority
+ ", allowZeroCapacitySum=" + allowZeroCapacitySum);
{code}
ParentQueue#toString:
{code:java}
public String toString() {
return queueName + ": " +
"numChildQueue= " + childQueues.size() + ", " +
"capacity=" + queueCapacities.getCapacity() + ", " +
"absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
"usedResources=" + queueUsage.getUsed() +
"usedCapacity=" + getUsedCapacity() + ", " +
"numApps=" + getNumApplications() + ", " +
"numContainers=" + getNumContainers();
}
{code}
LeafQueue#setupQueueConfigs:
{code:java}
LOG.info(
"Initializing " + getQueuePath() + "\n" + "capacity = "
+ queueCapacities.getCapacity()
+ " [= (float) configuredCapacity / 100 ]" + "\n"
+ "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
+ " [= parentAbsoluteCapacity * capacity ]" + "\n"
+ "maxCapacity = " + queueCapacities.getMaximumCapacity()
+ " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity = "
+ queueCapacities.getAbsoluteMaximumCapacity()
+ " [= 1.0 maximumCapacity undefined, "
+ "(parentAbsoluteMaxCapacity * maximumCapacity) / 100 otherwise
]"
+ "\n" + "effectiveMinResource=" +
getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
+ " , effectiveMaxResource=" +
getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
+ "\n" + "userLimit = " + usersManager.getUserLimit()
+ " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
+ usersManager.getUserLimitFactor()
+ " [= configuredUserLimitFactor ]" + "\n" + "maxApplications = "
+ maxApplications
+ " [= configuredMaximumSystemApplicationsPerQueue or"
+ " (int)(configuredMaximumSystemApplications *
absoluteCapacity)]"
+ "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
+ " [= (int)(maxApplications * (userLimit / 100.0f) * "
+ "userLimitFactor) ]" + "\n"
+ "maxParallelApps = " + getMaxParallelApps() + "\n"
+ "usedCapacity = " +
+ queueCapacities.getUsedCapacity() + " [= usedResourcesMemory / "
+ "(clusterResourceMemory * absoluteCapacity)]" + "\n"
+ "absoluteUsedCapacity = " + absoluteUsedCapacity
+ " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
+ "maxAMResourcePerQueuePercent = " + maxAMResourcePerQueuePercent
+ " [= configuredMaximumAMResourcePercent ]" + "\n"
+ "minimumAllocationFactor = " + minimumAllocationFactor
+ " [= (float)(maximumAllocationMemory - minimumAllocationMemory)
/ "
+ "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
+ maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
+ "numContainers = " + numContainers
+ " [= currentNumContainers ]" + "\n" + "state = " + getState()
+ " [= configuredState ]" + "\n" + "acls = " + aclsString
+ " [= configuredAcls ]" + "\n"
+ "nodeLocalityDelay = " + nodeLocalityDelay + "\n"
+ "rackLocalityAdditionalDelay = "
+ rackLocalityAdditionalDelay + "\n"
+ "labels=" + labelStrBuilder.toString() + "\n"
+ "reservationsContinueLooking = "
+ reservationsContinueLooking + "\n" + "preemptionDisabled = "
+ getPreemptionDisabled() + "\n" + "defaultAppPriorityPerQueue = "
+
[jira] [Commented] (YARN-10625) FairScheduler: add global flag to disable AM-preemption
[ https://issues.apache.org/jira/browse/YARN-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284777#comment-17284777 ] Benjamin Teke commented on YARN-10625: -- [~pbacsko] thanks for the patch. LGTM +1 (non-binding). I agree with the doc update, as the global property doesn't overwrite the queue level ones, which can be non-trivial. > FairScheduler: add global flag to disable AM-preemption > --- > > Key: YARN-10625 > URL: https://issues.apache.org/jira/browse/YARN-10625 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.3.0 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10625-001.patch > > > YARN-9537 added a feature to disable AM preemption on a per queue basis. > This is a nice enhancement, but it's very inconvenient if the cluster has a > lot of queues or queues dynamically created/deleted regularly (static queue > configuration changes). > It's a legitimate use-case to have AM preemption turned off completely. To > make it easier, add property which acts as a global flag for this feature. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10624) Support max queues limit configuration in new auto created queue, consistent with old auto created.
[ https://issues.apache.org/jira/browse/YARN-10624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17284225#comment-17284225 ] Benjamin Teke commented on YARN-10624: -- Hey [~zhuqi], I agree, this should be supported, thanks for working on this. The patch is clean, looks good to me apart from one minor nit: in ParentQueue can you please delete the spaces before the colons in the instantiation of SchedulerDynamicEditException? Otherwise +1 (non-binding) from me. > Support max queues limit configuration in new auto created queue, consistent > with old auto created. > --- > > Key: YARN-10624 > URL: https://issues.apache.org/jira/browse/YARN-10624 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10624.001.patch > > > Since old created leaf queue has the max leaf queues limit, i think we also > should support this in new auto created queue, both the auto created leaf and > the auto created parent need limits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10618) RM UI2 Application page shows the AM preempted containers instead of the nonAM ones
[ https://issues.apache.org/jira/browse/YARN-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10618: - Attachment: YARN-10618.001.patch > RM UI2 Application page shows the AM preempted containers instead of the > nonAM ones > --- > > Key: YARN-10618 > URL: https://issues.apache.org/jira/browse/YARN-10618 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Minor > Attachments: YARN-10618.001.patch > > > YARN RM UIv2 application page shows the AM preempted containers under both > the _Num Non-AM container preempted_ and _Num AM container preempted_. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10618) RM UI2 Application page shows the AM preempted containers instead of the nonAM ones
Benjamin Teke created YARN-10618: Summary: RM UI2 Application page shows the AM preempted containers instead of the nonAM ones Key: YARN-10618 URL: https://issues.apache.org/jira/browse/YARN-10618 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Reporter: Benjamin Teke Assignee: Benjamin Teke YARN RM UIv2 application page shows the AM preempted containers under both the _Num Non-AM container preempted_ and _Num AM container preempted_. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)
[ https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281017#comment-17281017 ] Benjamin Teke commented on YARN-10609: -- Hi [~zhuqi], Thanks for the patch! Generally the content seems to be OK to me, however I would rephrase it a little to be more akin to the rest of the documentation: _The multiple of the queue capacity which can be configured to allow a single user to acquire more resources. By default this is set to 1 which ensures that a single user can never take more than the queue's configured capacity irrespective of how idle the cluster is._ Setting this to -1 will disable the feature. _Value is specified as a float._ Note: using the flexible auto queue creation (yarn.scheduler.capacity..auto-queue-creation-v2) with weights will automatically set this property to -1, as the dynamic queues will be created with the hardcoded weight of 1 and in idle cluster scenarios they should be able to use more resources than calculated. What do you think about is? cc: [~gandras], [~snemeth], [~pbacsko], [~shuzirra] > Update the document for YARN-10531(Be able to disable user limit factor for > CapacityScheduler Leaf Queue) > - > > Key: YARN-10609 > URL: https://issues.apache.org/jira/browse/YARN-10609 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Attachments: YARN-10609.001.patch > > > Since we have finished YARN-10531. > We should update the corresponding document. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10614) Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion
[ https://issues.apache.org/jira/browse/YARN-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke resolved YARN-10614. -- Resolution: Invalid Marking it as invalid, it was already implemented at the time of the creation. > Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion > > > Key: YARN-10614 > URL: https://issues.apache.org/jira/browse/YARN-10614 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > > Weight mode conversion was added to fs2cs in YARN-10525 and in YARN-10600 > changes were made to convert the root queue to weight mode. However root > queue doesn't get the auto-create-v2 enabled flag, which again can be > misleading to former FS users. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10614) Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion
[ https://issues.apache.org/jira/browse/YARN-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10614: - Description: Weight mode conversion was added to fs2cs in YARN-10525 and in YARN-10600 changes were made to convert the root queue to weight mode. However root queue doesn't get the auto-create-v2 enabled flag, which again can be misleading to former FS users. (was: Weight mode conversion was added to fs2cs in YARN-10525. However root queue doesn't get converted to weight mode, which can be misleading to former FS users, so it should be added to fs2cs.) > Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion > > > Key: YARN-10614 > URL: https://issues.apache.org/jira/browse/YARN-10614 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > > Weight mode conversion was added to fs2cs in YARN-10525 and in YARN-10600 > changes were made to convert the root queue to weight mode. However root > queue doesn't get the auto-create-v2 enabled flag, which again can be > misleading to former FS users. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10614) Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion
Benjamin Teke created YARN-10614: Summary: Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion Key: YARN-10614 URL: https://issues.apache.org/jira/browse/YARN-10614 Project: Hadoop YARN Issue Type: Sub-task Reporter: Benjamin Teke Assignee: Benjamin Teke Fix For: 3.4.0 Weight mode conversion was added to fs2cs in YARN-10525. However root queue doesn't get converted to weight mode, which can be misleading to former FS users, so it should be added to fs2cs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10605) Add queue-mappings-override.enable property in FS2CS conversions
[
https://issues.apache.org/jira/browse/YARN-10605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276356#comment-17276356
]
Benjamin Teke commented on YARN-10605:
--
[~gandras] thanks for the patch. I have one minor nit about it: in
TestFSConfigToCSConfigConverter you introduced an unused import
{{org.junit.Assert.assertFalse}}. Otherwise the patch LGTM, +1 (non-binding).
> Add queue-mappings-override.enable property in FS2CS conversions
>
>
> Key: YARN-10605
> URL: https://issues.apache.org/jira/browse/YARN-10605
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10605.001.patch
>
>
> In Capacity Scheduler the
> {noformat}
> queue-mappings-override.enable
> {noformat}
> property is false by default. As this is not set during an FS2CS conversion,
> the converted placement rules (aka. mapping rules in CS) are ignored during
> application submission. We should enable this property in the conversion
> logic if there are placement rules to be converted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10604) Support auto queue creation without mapping rules
[
https://issues.apache.org/jira/browse/YARN-10604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276344#comment-17276344
]
Benjamin Teke commented on YARN-10604:
--
[~gandras] thanks for the patch. Looks good to me, +1 (non-binding).
> Support auto queue creation without mapping rules
> -
>
> Key: YARN-10604
> URL: https://issues.apache.org/jira/browse/YARN-10604
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10604.001.patch
>
>
> Currently, the Capacity Scheduler skips auto queue creation entirely, if the
> ApplicationPlacementContext is null, which happens, when the mapping rules
> are turned off by:
> {noformat}
>
> yarn.scheduler.capacity.queue-mappings-override.enable
> false
> {noformat}
> We should allow the auto queue creation to be taken into consideration
> without disrupting the application submission flow.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10178) Global Scheduler async thread crash caused by 'Comparison method violates its general contract'
[
https://issues.apache.org/jira/browse/YARN-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273883#comment-17273883
]
Benjamin Teke commented on YARN-10178:
--
[~zhuqi] thanks for the patch. The logic looks good to me, +1 (non-binding)
from my side.
> Global Scheduler async thread crash caused by 'Comparison method violates its
> general contract'
> ---
>
> Key: YARN-10178
> URL: https://issues.apache.org/jira/browse/YARN-10178
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 3.2.1
>Reporter: tuyu
>Assignee: zhuqi
>Priority: Major
> Attachments: YARN-10178.001.patch, YARN-10178.002.patch
>
>
> Global Scheduler Async Thread crash stack
> {code:java}
> ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type CRITICAL_THREAD_CRASH, caused by a critical thread,
> Thread-6066574, that exited unexpectedly: java.lang.IllegalArgumentException:
> Comparison method violates its general contract!
>at
> java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:177)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.policy.PriorityUtilizationQueueOrderingPolicy.getAssignmentIterator(PriorityUtilizationQueueOrderingPolicy.java:221)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.sortAndGetChildrenAllocationIterator(ParentQueue.java:777)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:791)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:623)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1635)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1629)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1732)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1481)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.schedule(CapacityScheduler.java:569)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler$AsyncScheduleThread.run(CapacityScheduler.java:616)
> {code}
> JAVA 8 Arrays.sort default use timsort algo, and timsort has few require
> {code:java}
> 1.x.compareTo(y) != y.compareTo(x)
> 2.x>y,y>z --> x > z
> 3.x=y, x.compareTo(z) == y.compareTo(z)
> {code}
> if not Arrays paramters not satify this require,TimSort will throw
> 'java.lang.IllegalArgumentException'
> look at PriorityUtilizationQueueOrderingPolicy.compare function,we will know
> Capacity Scheduler use this these queue resource usage to compare
> {code:java}
> AbsoluteUsedCapacity
> UsedCapacity
> ConfiguredMinResource
> AbsoluteCapacity
> {code}
> In Capacity Scheduler Global Scheduler AsyncThread use
> PriorityUtilizationQueueOrderingPolicy function to choose queue to assign
> container,and construct a CSAssignment struct, and use
> submitResourceCommitRequest function add CSAssignment to backlogs
> ResourceCommitterService will tryCommit this CSAssignment,look tryCommit
> function,there will update queue resource usage
> {code:java}
> public boolean tryCommit(Resource cluster, ResourceCommitRequest r,
> boolean updatePending) {
> long commitStart = System.nanoTime();
> ResourceCommitRequest request =
> (ResourceCommitRequest) r;
>
> ...
> boolean isSuccess = false;
> if (attemptId != null) {
> FiCaSchedulerApp app = getApplicationAttempt(attemptId);
> // Required sanity check for attemptId - when async-scheduling enabled,
> // proposal might be outdated if AM failover just finished
> // and proposal queue was not be consumed in time
> if (app != null && attemptId.equals(app.getApplicationAttemptId())) {
> if (app.accept(cluster, request, updatePending)
> && app.apply(cluster, request, updatePending)) {
[jira] [Commented] (YARN-10590) Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss.
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273860#comment-17273860 ] Benjamin Teke commented on YARN-10590: -- Thanks [~zhuqi], for working on this. Can you please elaborate on the part where you talk about the validity of the current logic? Because as we discussed in YARN-10504 the initialization of auto created queues from template was changed (see [comment|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549] and [comment|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17262562=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17262562]). Also once we decide that we follow with the test modifications we probably should modify the jiras description to contain the original logic change introduced in YARN-10504. > Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute > caculation loss. > - > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: zhuqi >Priority: Major > Attachments: YARN-10590.001.patch, YARN-10590.002.patch > > > Because we have introduced YARN-10504. > In auto created leaf queue with absolute mode, the caculation of effective > min resource will different from other mode. > We should fix the related test in TestCapacitySchedulerAutoCreatedQueueBase. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10600) Convert root queue in fs2cs weight mode conversion
[ https://issues.apache.org/jira/browse/YARN-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10600: - Attachment: YARN-10600.001.patch > Convert root queue in fs2cs weight mode conversion > -- > > Key: YARN-10600 > URL: https://issues.apache.org/jira/browse/YARN-10600 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10600.001.patch > > > Weight mode conversion was added to fs2cs in YARN-10525. However root queue > doesn't get converted to weight mode, which can be misleading to former FS > users, so it should be added to fs2cs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10600) Convert root queue in fs2cs weight mode conversion
[ https://issues.apache.org/jira/browse/YARN-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10600: - Description: Weight mode conversion was added to fs2cs in YARN-10525. However root queue doesn't get converted to weight mode, which can be misleading to former FS users, so it should be added to fs2cs. (was: Weight mode conversion was added to fs2cs in YARN-10525. However root queue doesn't get converted to weight mode, which can be misleading to former FS users. Currently, we convert FS weights to percentages, however, it will be more useful to keep those values and use them in CS as well.) > Convert root queue in fs2cs weight mode conversion > -- > > Key: YARN-10600 > URL: https://issues.apache.org/jira/browse/YARN-10600 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > > Weight mode conversion was added to fs2cs in YARN-10525. However root queue > doesn't get converted to weight mode, which can be misleading to former FS > users, so it should be added to fs2cs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10600) Convert root queue in fs2cs weight mode conversion
[ https://issues.apache.org/jira/browse/YARN-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10600: - Description: Weight mode conversion was added to fs2cs in YARN-10525. However root queue doesn't get converted to weight mode, which can be misleading to former FS users. Currently, we convert FS weights to percentages, however, it will be more useful to keep those values and use them in CS as well. was: Weight mode will be added to Capacity Scheduler. Currently, we convert FS weights to percentages, however, it will be more useful to keep those values and use them in CS as well. > Convert root queue in fs2cs weight mode conversion > -- > > Key: YARN-10600 > URL: https://issues.apache.org/jira/browse/YARN-10600 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0 > > > Weight mode conversion was added to fs2cs in YARN-10525. However root queue > doesn't get converted to weight mode, which can be misleading to former FS > users. > Currently, we convert FS weights to percentages, however, it will be more > useful to keep those values and use them in CS as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10600) Convert root queue in fs2cs weight mode conversion
[ https://issues.apache.org/jira/browse/YARN-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-10600: Assignee: Benjamin Teke (was: Peter Bacsko) > Convert root queue in fs2cs weight mode conversion > -- > > Key: YARN-10600 > URL: https://issues.apache.org/jira/browse/YARN-10600 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > > Weight mode conversion was added to fs2cs in YARN-10525. However root queue > doesn't get converted to weight mode, which can be misleading to former FS > users. > Currently, we convert FS weights to percentages, however, it will be more > useful to keep those values and use them in CS as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10600) Convert root queue in fs2cs weight mode conversion
Benjamin Teke created YARN-10600: Summary: Convert root queue in fs2cs weight mode conversion Key: YARN-10600 URL: https://issues.apache.org/jira/browse/YARN-10600 Project: Hadoop YARN Issue Type: Sub-task Reporter: Benjamin Teke Assignee: Peter Bacsko Fix For: 3.4.0 Weight mode will be added to Capacity Scheduler. Currently, we convert FS weights to percentages, however, it will be more useful to keep those values and use them in CS as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272997#comment-17272997 ] Benjamin Teke commented on YARN-10598: -- [~snemeth] Thanks for the review, both of them are valid points. Uploaded a new patch. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch, YARN-10598.002.patch, > YARN-10598.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Attachment: YARN-10598.003.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch, YARN-10598.002.patch, > YARN-10598.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272881#comment-17272881 ] Benjamin Teke commented on YARN-10598: -- Thank [~gandras] for the review. Fixed the suggestions and the checkstyle issues that doesn't need a larger refactor. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch, YARN-10598.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Attachment: YARN-10598.002.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch, YARN-10598.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Attachment: YARN-10598.001.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail:
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * **Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also imlemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. The queue type should hold these values: * Auto-created parent queue: *autoCreatedParent* * Auto-created leaf queue: *autoCreatedLeaf* * Static parent: *staticParent* * Static leaf: *staticLeaf* > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * **Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Description: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* was: Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also implemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. To extend/modify the values added in YARN-10581 these 3 fields will describe a queue: * queueType : *parent/leaf* * creationMethod : *static/dynamicLegacy/dynamicFlexible* * autoCreationEligibility : *off/legacy/flexible* After this change here are some example cases: * Static parent queue which has the auto-creation-enabled-v2 false: ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *off* * **Static managed parent (can have dynamic children): ** queueType : *parent* ** creationMethod : *static* ** autoCreationEligibility : *legacy* * Legacy auto-created leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicLegacy* ** autoCreationEligibility : *off* * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: ** queueType : *parent* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *flexible* * Auto-created (v2) leaf queue (cannot have children): ** queueType : *leaf* ** creationMethod : *dynamicFlexible* ** autoCreationEligibility : *off* > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, auto-creation-enabled-v2 true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail:
[jira] [Assigned] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-10598: Assignee: Benjamin Teke (was: Szilard Nemeth) > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also imlemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > The queue type should hold these values: > * Auto-created parent queue: *autoCreatedParent* > * Auto-created leaf queue: *autoCreatedLeaf* > * Static parent: *staticParent* > * Static leaf: *staticLeaf* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
Benjamin Teke created YARN-10598: Summary: CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information Key: YARN-10598 URL: https://issues.apache.org/jira/browse/YARN-10598 Project: Hadoop YARN Issue Type: Sub-task Reporter: Benjamin Teke Assignee: Szilard Nemeth Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504. Auto-queue creation has been also imlemented with YARN-10506. Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint. The queue type should hold these values: * Auto-created parent queue: *autoCreatedParent* * Auto-created leaf queue: *autoCreatedLeaf* * Static parent: *staticParent* * Static leaf: *staticLeaf* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271419#comment-17271419
]
Benjamin Teke commented on YARN-10596:
--
[~gandras] thanks for the patch. I have a suggestion for making the condition a
bit more readable (for me at least). Consider the following part:
{code:java}
boolean isEligibleForAutoQueueCreation = false;
// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isEligibleForAutoQueueCreation = ((ParentQueue)
oldQueue).isDynamicQueue();
}
// if a queue is eligible for auto queue creation v2
// it must be a ParentQueue
if (!isEligibleForAutoQueueCreation) {
isEligibleForAutoQueueCreation =
conf.isAutoQueueCreationV2Enabled(fullQueueName);
}
{code}
Basically the condition is: if the queue is a dynamic parent, or it has the the
AutoQueueCreationV2 enabled then it can be an empty parent. As the old
autoQueueEnabled flag is parsed and stored as a boolean I think it would be
nicer if you got the conf.isAutoQueueCreationV2Enabled there and kept the
isDynamicParent intact (as isDynamicParent induces the eligibility for auto
creation, however it is not the same). You could skip the (bit) misleading if
(!isEligibleForAutoQueueCreation) part, and instead of the if
(childQueueNames.size() == 0 && !isEligibleForAutoQueueCreation) condition you
could say that the queue should not have 0 childQueues, should not be a dynamic
parent and should not be eligible for auto creation based on the config.
> Allow static definition of childless ParentQueues with auto-queue-creation-v2
> enabled
> -
>
> Key: YARN-10596
> URL: https://issues.apache.org/jira/browse/YARN-10596
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10596.001.patch
>
>
> The old auto queue creation/managed queue logic allowed the definition of
> childless parents to be created statically, if the auto-create-child-queue
> flag was turned on the parent (thus making it a ManagedParentQueue).
> Since it is not an edge case, we also need to support the creation of a
> ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled,
> even when no child queue is defined under the parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271371#comment-17271371 ] Benjamin Teke edited comment on YARN-10585 at 1/25/21, 3:34 PM: Thank [~shuzirra] for working on this. Based on your answers to [~gandras]'s comments the patch looks good to me, so generally +1 (non-binding) from my side. Another patch might be needed because the 002 generates license warnings and has a checkstyle issue. was (Author: bteke): Thank [~shuzirra] for working on this. Based on your answers to [~gandras]'s comments the patch looks good to me, so +1 (non-binding) from my side. > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10585.001.patch, YARN-10585.002.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format
[ https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271371#comment-17271371 ] Benjamin Teke commented on YARN-10585: -- Thank [~shuzirra] for working on this. Based on your answers to [~gandras]'s comments the patch looks good to me, so +1 (non-binding) from my side. > Create a class which can convert from legacy mapping rule format to the new > JSON format > --- > > Key: YARN-10585 > URL: https://issues.apache.org/jira/browse/YARN-10585 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10585.001.patch, YARN-10585.002.patch > > > To make transition easier we need to create tooling to support the migration > effort. The first step is to create a class which can migrate from legacy to > the new JSON format. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10565:
-
Description:
In YARN-10504 weight mode support was introduced to CS. This jira is a followup
to simplify and restructure the initialization, so that the weight
calculation/absolute/percentage mode is easier to understand and modify.
To be refactored:
* In ParentQueue.java#1099 the error message should be more specific, instead
of the {{LOG.error("Fatal issue found: e", e);}}
* AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
just to be on the safe side
* Uncomment the commented assertions in
TestCapacitySchedulerAutoCreatedQueueBase.validateEffectiveMinResource
* Check whether the assertion modification in TestRMWebServices is absolutely
necessary or could be hiding a bug.
* Same for TestRMWebServicesForCSWithPartitions.java
was:
In YARN-10504 weight mode support was introduced to CS. This jira is a followup
to simplify and restructure the initialization, so that the weight
calculation/absolute/percentage mode is easier to understand and modify.
To be refactored:
* In ParentQueue.java#1099 the error message should be more specific, instead
of the {{LOG.error("Fatal issue found: e", e);}}
* AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
just to be on the safe side
*
> Refactor CS queue initialization to simplify weight mode calculation
>
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch, YARN-10565.002.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a
> followup to simplify and restructure the initialization, so that the weight
> calculation/absolute/percentage mode is easier to understand and modify.
> To be refactored:
> * In ParentQueue.java#1099 the error message should be more specific, instead
> of the {{LOG.error("Fatal issue found: e", e);}}
> * AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
> just to be on the safe side
> * Uncomment the commented assertions in
> TestCapacitySchedulerAutoCreatedQueueBase.validateEffectiveMinResource
> * Check whether the assertion modification in TestRMWebServices is absolutely
> necessary or could be hiding a bug.
> * Same for TestRMWebServicesForCSWithPartitions.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[
https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10565:
-
Description:
In YARN-10504 weight mode support was introduced to CS. This jira is a followup
to simplify and restructure the initialization, so that the weight
calculation/absolute/percentage mode is easier to understand and modify.
To be refactored:
* In ParentQueue.java#1099 the error message should be more specific, instead
of the {{LOG.error("Fatal issue found: e", e);}}
* AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
just to be on the safe side
*
was:In YARN-10504 weight mode support was introduced to CS. This jira is a
followup to simplify and restructure the initialization, so that the weight
calculation/absolute/percentage mode is easier to understand and modify.
> Refactor CS queue initialization to simplify weight mode calculation
>
>
> Key: YARN-10565
> URL: https://issues.apache.org/jira/browse/YARN-10565
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10565.001.patch, YARN-10565.002.patch
>
>
> In YARN-10504 weight mode support was introduced to CS. This jira is a
> followup to simplify and restructure the initialization, so that the weight
> calculation/absolute/percentage mode is easier to understand and modify.
> To be refactored:
> * In ParentQueue.java#1099 the error message should be more specific, instead
> of the {{LOG.error("Fatal issue found: e", e);}}
> * AutoCreatedLeafQueue.clearConfigurableFields should clear NORMALIZED_WEIGHT
> just to be on the safe side
> *
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode and enable it by default
[
https://issues.apache.org/jira/browse/YARN-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267417#comment-17267417
]
Benjamin Teke commented on YARN-10573:
--
The findbugs issue should be solved by now (see YARN-10574) so retriggering the
CI job should get rid of it.
> Enhance placement rule conversion in fs2cs in weight mode and enable it by
> default
> --
>
> Key: YARN-10573
> URL: https://issues.apache.org/jira/browse/YARN-10573
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10573-001.patch, YARN-10573-002.patch,
> YARN-10573-003.patch
>
>
> If we're using weight mode, we have much more freedom when it comes to
> placement rules.
> We need to do three things in this ticket:
> #1
> In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
> means that we can support nested rules properly and also queues can be
> created under {{root}}.
> Therefore, a lot of warnings and validations inside
> {{QueuePlacementConverter}} are not necessary and only relevant if the user
> chose percentage-based conversion in the command line.
> #2
> Remove unnecessary stuff in the RuleHandler which are already supported (plus
> the test code which affected by this):
> * SPECIFIED_NOT_FIRST
> * USER_MAX_APPS_DEFAULT
> * USER_MAX_RUNNING_APPS
> #3
> Currently, users have to use "\-m" or "\-\-convert\-placement\-rules" switch
> to convert the placement rules from FS.
> Initially, we converted to the old mapping rule format, which has serious
> limitations, so we disabled the automatic conversion.
> With the new JSON-based format and placement engine, this conversion should
> happen automatically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode and enable it by default
[
https://issues.apache.org/jira/browse/YARN-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267388#comment-17267388
]
Benjamin Teke edited comment on YARN-10573 at 1/18/21, 4:11 PM:
[~pbacsko] thanks for the patch. Went through it, it looks good to me. +1
(non-binding). Let's see the CI results.
was (Author: bteke):
[~pbacsko] thanks for the patch. Went through it, it looks good to me. +1
(non-binding).
> Enhance placement rule conversion in fs2cs in weight mode and enable it by
> default
> --
>
> Key: YARN-10573
> URL: https://issues.apache.org/jira/browse/YARN-10573
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10573-001.patch, YARN-10573-002.patch,
> YARN-10573-003.patch
>
>
> If we're using weight mode, we have much more freedom when it comes to
> placement rules.
> We need to do three things in this ticket:
> #1
> In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
> means that we can support nested rules properly and also queues can be
> created under {{root}}.
> Therefore, a lot of warnings and validations inside
> {{QueuePlacementConverter}} are not necessary and only relevant if the user
> chose percentage-based conversion in the command line.
> #2
> Remove unnecessary stuff in the RuleHandler which are already supported (plus
> the test code which affected by this):
> * SPECIFIED_NOT_FIRST
> * USER_MAX_APPS_DEFAULT
> * USER_MAX_RUNNING_APPS
> #3
> Currently, users have to use "\-m" or "\-\-convert\-placement\-rules" switch
> to convert the placement rules from FS.
> Initially, we converted to the old mapping rule format, which has serious
> limitations, so we disabled the automatic conversion.
> With the new JSON-based format and placement engine, this conversion should
> happen automatically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10573) Enhance placement rule conversion in fs2cs in weight mode and enable it by default
[
https://issues.apache.org/jira/browse/YARN-10573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267388#comment-17267388
]
Benjamin Teke commented on YARN-10573:
--
[~pbacsko] thanks for the patch. Went through it, it looks good to me. +1
(non-binding).
> Enhance placement rule conversion in fs2cs in weight mode and enable it by
> default
> --
>
> Key: YARN-10573
> URL: https://issues.apache.org/jira/browse/YARN-10573
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10573-001.patch, YARN-10573-002.patch,
> YARN-10573-003.patch
>
>
> If we're using weight mode, we have much more freedom when it comes to
> placement rules.
> We need to do three things in this ticket:
> #1
> In YARN-10525, weight conversion became the default in {{fs2cs}}. This also
> means that we can support nested rules properly and also queues can be
> created under {{root}}.
> Therefore, a lot of warnings and validations inside
> {{QueuePlacementConverter}} are not necessary and only relevant if the user
> chose percentage-based conversion in the command line.
> #2
> Remove unnecessary stuff in the RuleHandler which are already supported (plus
> the test code which affected by this):
> * SPECIFIED_NOT_FIRST
> * USER_MAX_APPS_DEFAULT
> * USER_MAX_RUNNING_APPS
> #3
> Currently, users have to use "\-m" or "\-\-convert\-placement\-rules" switch
> to convert the placement rules from FS.
> Initially, we converted to the old mapping rule format, which has serious
> limitations, so we disabled the automatic conversion.
> With the new JSON-based format and placement engine, this conversion should
> happen automatically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266036#comment-17266036
]
Benjamin Teke commented on YARN-10506:
--
Thanks for working on this! Checked the latest patch, generally seems good to
me. I have two minor nits about it:
* In AbstractCSQueue the field {{lastSubmittedTimestamp}} is unused. It has a
"setter" and a getter, but neither of them are called. Shouldn't this be
removed to avoid confusion?
* The same goes for CapacitySchedulerAutoQueueHandler's conf field.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch,
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch,
> YARN-10506.014.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[ https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10565: - Attachment: YARN-10565.002.patch > Refactor CS queue initialization to simplify weight mode calculation > > > Key: YARN-10565 > URL: https://issues.apache.org/jira/browse/YARN-10565 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10565.001.patch, YARN-10565.002.patch > > > In YARN-10504 weight mode support was introduced to CS. This jira is a > followup to simplify and restructure the initialization, so that the weight > calculation/absolute/percentage mode is easier to understand and modify. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[ https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-10565: Assignee: Benjamin Teke > Refactor CS queue initialization to simplify weight mode calculation > > > Key: YARN-10565 > URL: https://issues.apache.org/jira/browse/YARN-10565 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10565.001.patch > > > In YARN-10504 weight mode support was introduced to CS. This jira is a > followup to simplify and restructure the initialization, so that the weight > calculation/absolute/percentage mode is easier to understand and modify. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS
[
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266010#comment-17266010
]
Benjamin Teke commented on YARN-10512:
--
Thank you [~snemeth] for working on this.
Regarding the 2nd point in [~gandras]'s comment: as I see it
{{getCapacityConfigurationTypeForQueues}} basically uses the same approach
(check whether the QueueCapacities' weight or capacity is set), except it
doesn't consider the queue's {{getCapacityConfigType()}} method. That method
should be sufficient for differentiating ABSOLUTE and PERCENTAGE+WEIGHT mode,
as {{AbstractCSQueue.updateEffectiveResources}} uses it as well. And the
dynamic queues will have their weight set in their respective QueueCapacities
object, as it holds the calculated values, not the ones read from the
configuration. All in all I suggest to keep this patch as simple as possible
and consider refactoring this after the YARN-10504 refactor is finished
(YARN-10565).
> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include
> mode of operation for CS
> --
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10512.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS
> with YARN-10504.
> We would like to expose the mode of operation with the RM's /scheduler REST
> endpoint.
> The field name will be 'mode'.
> All queue representations in the response will be uniformly hold any of the
> mode values of: "percentage", "absolute", "weight".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10496) [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264106#comment-17264106 ] Benjamin Teke commented on YARN-10496: -- [~pbacsko], regarding the max capacity: as of now YARN-10504 disabled the validation for the absolute and absolute max capacity of a queue. I think we should allow some flexibility by either introducing a flag or a special format like you mentioned. Couple of concerns/questions: * Should we allow the max capacity to be lower than the capacity? ** In "relative to the cluster" mode this can be straightforward, especially with weight mode, I can setup a quite large queue hierarchy with weights and not worry about any queue eating up large part of the cluster resources. ** In "relative to the parent" mode this can allow an option where the weights are basically disabled, and the queues are configured with the max capacity. Not necessarily a problem, but this can lead to hard to read configurations. * If we keep/reintroduce the capacity < max capacity constraint in weight mode the user might have to calculate the percentages from weight manually. For example child1 and child2 are the only child queues under a parent with weights 3 and 1. In this setup child1 has to have the configured max capacity as 75% while child2 can have anything above 25%. This is ok for a static parent, but if/when auto-create templates/wildcard configs will be supported the capacity can greatly change based on the number of dynamic queues. If I want to express the max capacity of any child as 33% of the parent's resources I will need to define at least 3 static queues with the same weight, I can't allow these to be auto created (because 1 queue with weight 1 will have the capacity 100%, 2 queues with weight 1 will have 50%). This is another reason to let this constraint go. > [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler > - > > Key: YARN-10496 > URL: https://issues.apache.org/jira/browse/YARN-10496 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacity scheduler >Reporter: Wangda Tan >Priority: Major > > CapacityScheduler today doesn’t support an auto queue creation which is > flexible enough. The current constraints: > * Only leaf queues can be auto-created > * A parent can only have either static queues or dynamic ones. This causes > multiple constraints. For example: > * It isn’t possible to have a VIP user like Alice with a static queue > root.user.alice with 50% capacity while the other user queues (under > root.user) are created dynamically and they share the remaining 50% of > resources. > > * In comparison, FairScheduler allows the following scenarios, Capacity > Scheduler doesn’t: > ** This implies that there is no possibility to have both dynamically > created and static queues at the same time under root > * A new queue needs to be created under an existing parent, while the parent > already has static queues > * Nested queue mapping policy, like in the following example: > | > > | > * Here two levels of queues may need to be created > If an application belongs to user _alice_ (who has the primary_group of > _engineering_), the scheduler checks whether _root.engineering_ exists, if it > doesn’t, it’ll be created. Then scheduler checks whether > _root.engineering.alice_ exists, and creates it if it doesn't. > > When we try to move users from FairScheduler to CapacityScheduler, we face > feature gaps which blocks users migrate from FS to CS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[ https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264003#comment-17264003 ] Benjamin Teke commented on YARN-10565: -- Added most of the changes from YARN-10506.011 patch, as it was abandoned there. > Refactor CS queue initialization to simplify weight mode calculation > > > Key: YARN-10565 > URL: https://issues.apache.org/jira/browse/YARN-10565 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > Attachments: YARN-10565.001.patch > > > In YARN-10504 weight mode support was introduced to CS. This jira is a > followup to simplify and restructure the initialization, so that the weight > calculation/absolute/percentage mode is easier to understand and modify. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[ https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10565: - Attachment: YARN-10565.001.patch > Refactor CS queue initialization to simplify weight mode calculation > > > Key: YARN-10565 > URL: https://issues.apache.org/jira/browse/YARN-10565 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > Attachments: YARN-10565.001.patch > > > In YARN-10504 weight mode support was introduced to CS. This jira is a > followup to simplify and restructure the initialization, so that the weight > calculation/absolute/percentage mode is easier to understand and modify. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10512) CS Flexible Auto Queue Creation Check RM REST API impact
[ https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-10512: Assignee: Benjamin Teke > CS Flexible Auto Queue Creation Check RM REST API impact > > > Key: YARN-10512 > URL: https://issues.apache.org/jira/browse/YARN-10512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10525) Add weight mode conversion to fs2cs
[ https://issues.apache.org/jira/browse/YARN-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263566#comment-17263566 ] Benjamin Teke commented on YARN-10525: -- Thanks [~pbacsko] for working on this. The patch LGTM (non-binding), and the tests are passing locally. [~snemeth] can you please retrigger the CI? > Add weight mode conversion to fs2cs > --- > > Key: YARN-10525 > URL: https://issues.apache.org/jira/browse/YARN-10525 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10525-001.patch, YARN-10525-002.patch, > YARN-10525-003.patch, YARN-10525-004.patch, YARN-10525-005.patch > > > Weight mode will be added to Capacity Scheduler. > Currently, we convert FS weights to percentages, however, it will be more > useful to keep those values and use them in CS as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262884#comment-17262884
]
Benjamin Teke commented on YARN-10504:
--
I agree with your comment, so if it got an LGTM lets fix the remaining issues
in a later patch.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.011.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262720#comment-17262720
]
Benjamin Teke commented on YARN-10504:
--
Uploaded patch 11 containing the said modification, however it causes an issue
in
TestAbsoluteResourceConfiguration.testSimpleMinMaxResourceConfigurartionPerQueue.
I'll look into it, in the meantime patch 11 can be considered obsolete.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.011.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10504:
-
Attachment: YARN-10504.011.patch
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.011.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
[ https://issues.apache.org/jira/browse/YARN-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262597#comment-17262597 ] Benjamin Teke commented on YARN-10565: -- [~sunilg] created this item based on your [comment|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17262580=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17262580] on YARN-10504 > Refactor CS queue initialization to simplify weight mode calculation > > > Key: YARN-10565 > URL: https://issues.apache.org/jira/browse/YARN-10565 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Priority: Major > > In YARN-10504 weight mode support was introduced to CS. This jira is a > followup to simplify and restructure the initialization, so that the weight > calculation/absolute/percentage mode is easier to understand and modify. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10565) Refactor CS queue initialization to simplify weight mode calculation
Benjamin Teke created YARN-10565: Summary: Refactor CS queue initialization to simplify weight mode calculation Key: YARN-10565 URL: https://issues.apache.org/jira/browse/YARN-10565 Project: Hadoop YARN Issue Type: Sub-task Reporter: Benjamin Teke In YARN-10504 weight mode support was introduced to CS. This jira is a followup to simplify and restructure the initialization, so that the weight calculation/absolute/percentage mode is easier to understand and modify. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
]
Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:33 AM:
-
[~zhuqi],
Thanks for the detailed explanation on the root cause of the Absolute mode -
auto creation test issue. I came to the same conclusion in my comment
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
so I agree with your suggestion. Will upload a patch containing it, if no
objections.
Thanks [~sunilg] for the review. I'll create the followup jira.
was (Author: bteke):
[~zhuqi],
As for the root cause of the Absolute mode - auto creation test issue I came to
the same conclusion in my comment
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
so I agree with your suggestion. Will upload a patch containing it, if no
objections.
Thanks [~sunilg] for the review. I'll create the followup jira.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261549#comment-17261549
]
Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:32 AM:
-
Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure:
{{AutoCreatedLeafQueue#reinitializeFromTemplate}} was refactored, now the
getting and merging the QueueCapacities happens *before* calling the
{{ParentQueue#updateClusterResource}} (and
{{LeafQueue#updateClusterResource}}). In {{LeafQueue#updateClusterResource}}
the {{AbstractCSQueue#updateEffectiveResources}} is called where the
effectiveMinResource of the created queue is overridden with the template's
effectiveMinResources which is exactly the same the test is getting in the
asserts.
{code:java}
void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
Resource resourceByLabel = labelManager.getResourceByLabel(label,
clusterResource);
Resource minResource = queueResourceQuotas.getConfiguredMinResource(
label); // Update effective resource (min/max) to each child
queue.
if (getCapacityConfigType().equals(
CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));
...{code}
was (Author: bteke):
Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure:
{{AutoCreatedLeafQueue#reinitializeFromTemplate }}was refactored, now the
getting and merging the QueueCapacities happens{{ *before* }}calling the{{
ParentQueue#updateClusterResource}} (and {{LeafQueue#updateClusterResource}}).
In {{LeafQueue#updateClusterResource}} the
AbstractCSQueue#updateEffectiveResources is called where the
effectiveMinResource of the created queue is overridden with the template's
effectiveMinResources which is exactly the same the test is getting in the
asserts.
{code:java}
void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
Resource resourceByLabel = labelManager.getResourceByLabel(label,
clusterResource); Resource minResource =
queueResourceQuotas.getConfiguredMinResource(
label); // Update effective resource (min/max) to each child
queue.
if (getCapacityConfigType().equals(
CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));
...{code}
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261549#comment-17261549
]
Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:31 AM:
-
Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure:
{{AutoCreatedLeafQueue#reinitializeFromTemplate }}was refactored, now the
getting and merging the QueueCapacities happens{{ *before* }}calling the{{
ParentQueue#updateClusterResource}} (and {{LeafQueue#updateClusterResource}}).
In {{LeafQueue#updateClusterResource}} the
AbstractCSQueue#updateEffectiveResources is called where the
effectiveMinResource of the created queue is overridden with the template's
effectiveMinResources which is exactly the same the test is getting in the
asserts.
{code:java}
void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
Resource resourceByLabel = labelManager.getResourceByLabel(label,
clusterResource); Resource minResource =
queueResourceQuotas.getConfiguredMinResource(
label); // Update effective resource (min/max) to each child
queue.
if (getCapacityConfigType().equals(
CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));
...{code}
was (Author: bteke):
Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure:
{{AutoCreatedLeafQueue#reinitializeFromTemplate }}was refactored, now the
getting and merging the QueueCapacities happens *before* calling the
{{ParentQueue#updateClusterResource}} (and
{{LeafQueue#updateClusterResource}}). In {{LeafQueue#updateClusterResource
}}the {{AbstractCSQueue#updateEffectiveResources }}is called where the
effectiveMinResource of the created queue is overridden with the template's
effectiveMinResources which is exactly the same the test is getting in the
asserts.
{code:java}
void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
Resource resourceByLabel = labelManager.getResourceByLabel(label,
clusterResource); Resource minResource =
queueResourceQuotas.getConfiguredMinResource(
label); // Update effective resource (min/max) to each child
queue.
if (getCapacityConfigType().equals(
CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));
...{code}
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
]
Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:29 AM:
-
[~zhuqi],
As for the root cause of the Absolute mode - auto creation test issue I came to
the same conclusion in my comment
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
so I agree with your suggestion. Will upload a patch containing it, if no
objections.
Thanks [~sunilg] for the review. I'll create the followup jira.
was (Author: bteke):
[~zhuqi],
As for the root cause of the Absolute mode - auto creation test issue I came to
the same conclusion in my comment above, so I agree with your suggestion. Will
upload a patch containing it, if no objections.
Thanks [~sunilg] for the review. I'll create the followup jira.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
]
Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:29 AM:
-
[~zhuqi],
As for the root cause of the Absolute mode - auto creation test issue I came to
the same conclusion in my comment above, so I agree with your suggestion. Will
upload a patch containing it, if no objections.
Thanks [~sunilg] for the review. I'll create the followup jira.
was (Author: bteke):
[~zhuqi],
I came to the same conclusion in my comment
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
so I agree with your suggestion. Will upload a patch containing it, if no
objections.
Thanks [~sunilg] for the review. I'll create the followup jira.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
]
Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:28 AM:
-
[~zhuqi],
I came to the same conclusion in my comment
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
so I agree with your suggestion. Will upload a patch containing it, if no
objections.
Thanks [~sunilg] for the review. I'll create the followup jira.
was (Author: bteke):
[~zhuqi],
I came to the same conclusion in my comment above, so I agree with your
suggestion. Will upload a patch containing it, if no objections.
Thanks [~sunilg] for the review. I'll create the followup jira.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
]
Benjamin Teke edited comment on YARN-10504 at 1/11/21, 11:27 AM:
-
[~zhuqi],
I came to the same conclusion in my comment above, so I agree with your
suggestion. Will upload a patch containing it, if no objections.
Thanks [~sunilg] for the review. I'll create the followup jira.
was (Author: bteke):
[~zhuqi],
I came to the same conclusion in my comment
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
so I agree with your suggestion. Will upload a patch containing it, if no
objections.
Thanks [~sunilg] for the review! I'll create the followup jira.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262590#comment-17262590
]
Benjamin Teke commented on YARN-10504:
--
[~zhuqi],
I came to the same conclusion in my comment
[above|https://issues.apache.org/jira/browse/YARN-10504?focusedCommentId=17261549=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17261549],
so I agree with your suggestion. Will upload a patch containing it, if no
objections.
Thanks [~sunilg] for the review! I'll create the followup jira.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.008.patch,
> YARN-10504.009.patch, YARN-10504.010.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261961#comment-17261961
]
Benjamin Teke commented on YARN-10504:
--
[~wangda], [~zhuqi]
Added a small new patch. ver.6 had a small issue in
ParentQueue.getCapacityConfigurationTypeForQueues, where if the passed queues
collection was empty the iterator where the mixed mode check was implemented
threw a NoSuchElementException, causing all of the AutoCreatedTests to fail.
Also setChildQueues has a similar mixed mode check, where the root queue is
included in the check, causing an unnecessary exception. So I extended that if
as well. Additionally I added [~zhuqi]'s suggestion to
LeafQueue.updateClusterResource.
There is one more issue: now 2 of the TestAbsoluteResourceWithAutoQueue tests
are failing, because the apps are stuck in SUBMITTED state. [~wangda] do you
have an idea on this one?
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10504:
-
Attachment: YARN-10504.007.patch
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.007.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261954#comment-17261954
]
Benjamin Teke commented on YARN-10504:
--
[~wangda]/[~zhuqi]/[~gandras],
Let me take care of the suggestions in [~zhuqi]'s comment above. I'll base it
on the ver.6 patch.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.006.patch, YARN-10504.ver-1.patch, YARN-10504.ver-2.patch,
> YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261549#comment-17261549
]
Benjamin Teke commented on YARN-10504:
--
Sharing my latest findings on TestAbsoluteResourceWithAutoQueue failure:
{{AutoCreatedLeafQueue#reinitializeFromTemplate }}was refactored, now the
getting and merging the QueueCapacities happens *before* calling the
{{ParentQueue#updateClusterResource}} (and
{{LeafQueue#updateClusterResource}}). In {{LeafQueue#updateClusterResource
}}the {{AbstractCSQueue#updateEffectiveResources }}is called where the
effectiveMinResource of the created queue is overridden with the template's
effectiveMinResources which is exactly the same the test is getting in the
asserts.
{code:java}
void updateEffectiveResources(Resource clusterResource) {
Set configuredNodelabels =
csContext.getConfiguration().getConfiguredNodeLabels(getQueuePath());
for (String label : configuredNodelabels) {
Resource resourceByLabel = labelManager.getResourceByLabel(label,
clusterResource); Resource minResource =
queueResourceQuotas.getConfiguredMinResource(
label); // Update effective resource (min/max) to each child
queue.
if (getCapacityConfigType().equals(
CapacityConfigType.ABSOLUTE_RESOURCE)) {
queueResourceQuotas.setEffectiveMinResource(label,
getMinResourceNormalized(queuePath,
((ParentQueue) parent).getEffectiveMinRatioPerResource(),
minResource));
...{code}
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17261381#comment-17261381
]
Benjamin Teke commented on YARN-10504:
--
Hey [~zhuqi],
Thanks for your updates. Took the liberty to refactor/include your changes in
my latest patch. Also based on our offline discussion with [~gandras] I removed
some parts of his 002 patch, as it will be handled differently. As you have
mentioned the AutoQueueCreation should now work, because the leafQueueTemplates
should now have the correct absolute capacities. I will now turn to the
AbsoluteResourceWithAutoQueue issue and handle the maxApplications in auto
created cases.
If you upload a new patch please use the 005 patch as base, because that
contains the latest from our side and also includes your changes from 003/004
patch.
Thanks!
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-10504:
-
Attachment: YARN-10504.005.patch
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.003.patch, YARN-10504.004.patch, YARN-10504.005.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260292#comment-17260292
]
Benjamin Teke edited comment on YARN-10504 at 1/7/21, 7:53 AM:
---
[~wangda],
Regarding the AutoCreatedLeafQueue failures: we are looking into the issue.
Currently it seems like the way the mock setup is structured the
updateClusterResource is not called at the correct time. We'll update the patch
once the issue is verified and fixed.
was (Author: bteke):
[~wangda],
Regarding the AutoCreatedLeafQueue: we are looking into the issue. Cuurently it
seems like the way the mock setup is structured the updateClusterResource is
not called at the correct time. We'll update the patch once the issue is fixed.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260292#comment-17260292
]
Benjamin Teke commented on YARN-10504:
--
[~wangda],
Regarding the AutoCreatedLeafQueue: we are looking into the issue. Cuurently it
seems like the way the mock setup is structured the updateClusterResource is
not called at the correct time. We'll update the patch once the issue is fixed.
> Implement weight mode in Capacity Scheduler
> ---
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.002.patch,
> YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
