[jira] [Commented] (YARN-10380) Import logic of multi-node allocation in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239076#comment-17239076
]
zhuqi commented on YARN-10380:
--
[~wangda] [~sunilg] [~BilwaST]
I have attached the draft patch, if you any advice.
Thanks.
> Import logic of multi-node allocation in CapacityScheduler
> --
>
> Key: YARN-10380
> URL: https://issues.apache.org/jira/browse/YARN-10380
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Critical
> Attachments: YARN-10380.001.patch
>
>
> *1) Entry point:*
> When we do multi-node allocation, we're using the same logic of async
> scheduling:
> {code:java}
> // Allocate containers of node [start, end)
> for (FiCaSchedulerNode node : nodes) {
> if (current++ >= start) {
> if (shouldSkipNodeSchedule(node, cs, printSkipedNodeLogging)) {
> continue;
> }
> cs.allocateContainersToNode(node.getNodeID(), false);
> }
> } {code}
> Is it the most effective way to do multi-node scheduling? Should we allocate
> based on partitions? In above logic, if we have thousands of node in one
> partition, we will repeatly access all nodes of the partition thousands of
> times.
> I would suggest looking at making entry-point for node-heartbeat,
> async-scheduling (single node), and async-scheduling (multi-node) to be
> different.
> Node-heartbeat and async-scheduling (single node) can be still similar and
> share most of the code.
> async-scheduling (multi-node): should iterate partition first, using pseudo
> code like:
> {code:java}
> for (partition : all partitions) {
> allocateContainersOnMultiNodes(getCandidate(partition))
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10380) Import logic of multi-node allocation in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhuqi reassigned YARN-10380:
Assignee: zhuqi
> Import logic of multi-node allocation in CapacityScheduler
> --
>
> Key: YARN-10380
> URL: https://issues.apache.org/jira/browse/YARN-10380
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: zhuqi
>Priority: Critical
> Attachments: YARN-10380.001.patch
>
>
> *1) Entry point:*
> When we do multi-node allocation, we're using the same logic of async
> scheduling:
> {code:java}
> // Allocate containers of node [start, end)
> for (FiCaSchedulerNode node : nodes) {
> if (current++ >= start) {
> if (shouldSkipNodeSchedule(node, cs, printSkipedNodeLogging)) {
> continue;
> }
> cs.allocateContainersToNode(node.getNodeID(), false);
> }
> } {code}
> Is it the most effective way to do multi-node scheduling? Should we allocate
> based on partitions? In above logic, if we have thousands of node in one
> partition, we will repeatly access all nodes of the partition thousands of
> times.
> I would suggest looking at making entry-point for node-heartbeat,
> async-scheduling (single node), and async-scheduling (multi-node) to be
> different.
> Node-heartbeat and async-scheduling (single node) can be still similar and
> share most of the code.
> async-scheduling (multi-node): should iterate partition first, using pseudo
> code like:
> {code:java}
> for (partition : all partitions) {
> allocateContainersOnMultiNodes(getCandidate(partition))
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10497) Fix an issue in CapacityScheduler which fails to delete queues
[
https://issues.apache.org/jira/browse/YARN-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238837#comment-17238837
]
Wangda Tan commented on YARN-10497:
---
[~shuzirra], I would still prefer to use the
CapacitySchedulerConfiguration#getQueues, it is the same method we used in
other places, and we should use the same util function to avoid future problems
like this. Plz let me know what your thoughts are.
Attached ver.4 to fix the remaining checkstyle issue.
> Fix an issue in CapacityScheduler which fails to delete queues
> --
>
> Key: YARN-10497
> URL: https://issues.apache.org/jira/browse/YARN-10497
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
> Attachments: YARN-10497.001.patch, YARN-10497.002.patch,
> YARN-10497.003.patch, YARN-10497.004.patch
>
>
> We saw an exception when using queue mutation APIs:
> {code:java}
> 2020-11-13 16:47:46,327 WARN
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices:
> CapacityScheduler configuration validation failed:java.io.IOException: Queue
> root.am2cmQueueSecond not found
> {code}
> Which comes from this code:
> {code:java}
> List siblingQueues = getSiblingQueues(queueToRemove,
> proposedConf);
> if (!siblingQueues.contains(queueName)) {
> throw new IOException("Queue " + queueToRemove + " not found");
> }
> {code}
> (Inside MutableCSConfigurationProvider)
> If you look at the method:
> {code:java}
>
> private List getSiblingQueues(String queuePath, Configuration conf)
> {
> String parentQueue = queuePath.substring(0, queuePath.lastIndexOf('.'));
> String childQueuesKey = CapacitySchedulerConfiguration.PREFIX +
> parentQueue + CapacitySchedulerConfiguration.DOT +
> CapacitySchedulerConfiguration.QUEUES;
> return new ArrayList<>(conf.getStringCollection(childQueuesKey));
> }
> {code}
> And here's capacity-scheduler.xml I got
> {code:java}
> yarn.scheduler.capacity.root.queuesdefault, q1,
> q2
> {code}
> You can notice there're spaces between default, q1, a2
> So conf.getStringCollection returns:
> {code:java}
> default
> q1
> ...
> {code}
> Which causes match issue when we try to delete the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10497) Fix an issue in CapacityScheduler which fails to delete queues
[
https://issues.apache.org/jira/browse/YARN-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-10497:
--
Attachment: YARN-10497.004.patch
> Fix an issue in CapacityScheduler which fails to delete queues
> --
>
> Key: YARN-10497
> URL: https://issues.apache.org/jira/browse/YARN-10497
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
> Attachments: YARN-10497.001.patch, YARN-10497.002.patch,
> YARN-10497.003.patch, YARN-10497.004.patch
>
>
> We saw an exception when using queue mutation APIs:
> {code:java}
> 2020-11-13 16:47:46,327 WARN
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices:
> CapacityScheduler configuration validation failed:java.io.IOException: Queue
> root.am2cmQueueSecond not found
> {code}
> Which comes from this code:
> {code:java}
> List siblingQueues = getSiblingQueues(queueToRemove,
> proposedConf);
> if (!siblingQueues.contains(queueName)) {
> throw new IOException("Queue " + queueToRemove + " not found");
> }
> {code}
> (Inside MutableCSConfigurationProvider)
> If you look at the method:
> {code:java}
>
> private List getSiblingQueues(String queuePath, Configuration conf)
> {
> String parentQueue = queuePath.substring(0, queuePath.lastIndexOf('.'));
> String childQueuesKey = CapacitySchedulerConfiguration.PREFIX +
> parentQueue + CapacitySchedulerConfiguration.DOT +
> CapacitySchedulerConfiguration.QUEUES;
> return new ArrayList<>(conf.getStringCollection(childQueuesKey));
> }
> {code}
> And here's capacity-scheduler.xml I got
> {code:java}
> yarn.scheduler.capacity.root.queuesdefault, q1,
> q2
> {code}
> You can notice there're spaces between default, q1, a2
> So conf.getStringCollection returns:
> {code:java}
> default
> q1
> ...
> {code}
> Which causes match issue when we try to delete the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9449) Non-exclusive labels can create reservation loop on cluster without unlabeled node
[
https://issues.apache.org/jira/browse/YARN-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238718#comment-17238718
]
Aadithya commented on YARN-9449:
Hi,
Can any one provide me the solution or work around for this issue.This is
frequently occurring in EMR clusters where node labels are enabled.
I appreciate any help provided.
> Non-exclusive labels can create reservation loop on cluster without unlabeled
> node
> --
>
> Key: YARN-9449
> URL: https://issues.apache.org/jira/browse/YARN-9449
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.5
>Reporter: Brandon Scheller
>Priority: Major
>
> https://issues.apache.org/jira/browse/YARN-5342 Added a counter to Yarn so
> that unscheduled resource requests were attempted to be scheduled on
> unlabeled nodes first.
> This counter is reset only when an attempt to schedule happens on an
> unlabeled node.
> On hadoop clusters with only labeled nodes, this counter can never be reset
> and therefore it will block skipping that node.
> Because the node will not be skipped, it creates the loop shown below in the
> Yarn RM logs.
> This can block scheduling of a spark executor for example and cause the spark
> application to get stuck.
>
> {{_2019-02-18 23:54:22,591 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl
> (ResourceManager Event Processor): container_1550533628872_0003_01_23
> Container Transitioned from NEW to RESERVED 2019-02-18 23:54:22,591 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator
> (ResourceManager Event Processor): Reserved container
> application=application_1550533628872_0003 resource=
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@6ffe0dc3
> cluster= 2019-02-18 23:54:22,592 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue
> (ResourceManager Event Processor): assignedContainer queue=root
> usedCapacity=0.0 absoluteUsedCapacity=0.0 used=
> cluster= 2019-02-18 23:54:23,592 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
> (ResourceManager Event Processor): Trying to fulfill reservation for
> application application_1550533628872_0003 on node:
> ip-10-0-0-122.ec2.internal:8041 2019-02-18 23:54:23,592 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp
> (ResourceManager Event Processor): Application
> application_1550533628872_0003 unreserved on node host:
> ip-10-0-0-122.ec2.internal:8041 #containers=1 available= vCores:7> used=, currently has 0 at priority 1;
> currentReservation on node-label=LABELED 2019-02-18
> 23:54:23,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl
> (ResourceManager Event Processor): container_1550533628872_0003_01_24
> Container Transitioned from NEW to RESERVED 2019-02-18 23:54:23,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator
> (ResourceManager Event Processor): Reserved container
> application=application_1550533628872_0003 resource=
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@6ffe0dc3
> cluster= 2019-02-18 23:54:23,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue
> (ResourceManager Event Processor): assignedContainer queue=root
> usedCapacity=0.0 absoluteUsedCapacity=0.0 used=
> cluster= 2019-02-18 23:54:24,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
> (ResourceManager Event Processor): Trying to fulfill reservation for
> application application_1550533628872_0003 on node:
> ip-10-0-0-122.ec2.internal:8041 2019-02-18 23:54:24,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp
> (ResourceManager Event Processor): Application
> application_1550533628872_0003 unreserved on node host:
> ip-10-0-0-122.ec2.internal:8041 #containers=1 available= vCores:7> used=, currently has 0 at priority 1;
> currentReservation on node-label=LABELED 2019-02-18
> 23:54:24,594 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl
> (ResourceManager Event Processor): container_1550533628872_0003_01_25
> Container Transitioned from NEW to RESERVED 2019-02-18 23:54:24,594 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator
> (ResourceManager Event Processor): Reserved container
> application=application_1550533628872_0003 resource=
>
[jira] [Comment Edited] (YARN-9449) Non-exclusive labels can create reservation loop on cluster without unlabeled node
[
https://issues.apache.org/jira/browse/YARN-9449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238718#comment-17238718
]
Aadithya edited comment on YARN-9449 at 11/25/20, 12:22 PM:
Hi,
Can any one provide me the solution or work around for this issue.This is
frequently occurring in EMR clusters where node labels are enabled by default
I appreciate any help provided.
was (Author: aditya9277):
Hi,
Can any one provide me the solution or work around for this issue.This is
frequently occurring in EMR clusters where node labels are enabled.
I appreciate any help provided.
> Non-exclusive labels can create reservation loop on cluster without unlabeled
> node
> --
>
> Key: YARN-9449
> URL: https://issues.apache.org/jira/browse/YARN-9449
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.5
>Reporter: Brandon Scheller
>Priority: Major
>
> https://issues.apache.org/jira/browse/YARN-5342 Added a counter to Yarn so
> that unscheduled resource requests were attempted to be scheduled on
> unlabeled nodes first.
> This counter is reset only when an attempt to schedule happens on an
> unlabeled node.
> On hadoop clusters with only labeled nodes, this counter can never be reset
> and therefore it will block skipping that node.
> Because the node will not be skipped, it creates the loop shown below in the
> Yarn RM logs.
> This can block scheduling of a spark executor for example and cause the spark
> application to get stuck.
>
> {{_2019-02-18 23:54:22,591 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl
> (ResourceManager Event Processor): container_1550533628872_0003_01_23
> Container Transitioned from NEW to RESERVED 2019-02-18 23:54:22,591 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator
> (ResourceManager Event Processor): Reserved container
> application=application_1550533628872_0003 resource=
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@6ffe0dc3
> cluster= 2019-02-18 23:54:22,592 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue
> (ResourceManager Event Processor): assignedContainer queue=root
> usedCapacity=0.0 absoluteUsedCapacity=0.0 used=
> cluster= 2019-02-18 23:54:23,592 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
> (ResourceManager Event Processor): Trying to fulfill reservation for
> application application_1550533628872_0003 on node:
> ip-10-0-0-122.ec2.internal:8041 2019-02-18 23:54:23,592 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp
> (ResourceManager Event Processor): Application
> application_1550533628872_0003 unreserved on node host:
> ip-10-0-0-122.ec2.internal:8041 #containers=1 available= vCores:7> used=, currently has 0 at priority 1;
> currentReservation on node-label=LABELED 2019-02-18
> 23:54:23,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl
> (ResourceManager Event Processor): container_1550533628872_0003_01_24
> Container Transitioned from NEW to RESERVED 2019-02-18 23:54:23,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.AbstractContainerAllocator
> (ResourceManager Event Processor): Reserved container
> application=application_1550533628872_0003 resource=
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@6ffe0dc3
> cluster= 2019-02-18 23:54:23,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue
> (ResourceManager Event Processor): assignedContainer queue=root
> usedCapacity=0.0 absoluteUsedCapacity=0.0 used=
> cluster= 2019-02-18 23:54:24,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
> (ResourceManager Event Processor): Trying to fulfill reservation for
> application application_1550533628872_0003 on node:
> ip-10-0-0-122.ec2.internal:8041 2019-02-18 23:54:24,593 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp
> (ResourceManager Event Processor): Application
> application_1550533628872_0003 unreserved on node host:
> ip-10-0-0-122.ec2.internal:8041 #containers=1 available= vCores:7> used=, currently has 0 at priority 1;
> currentReservation on node-label=LABELED 2019-02-18
> 23:54:24,594 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl
> (ResourceManager Event Processor): container_1550533628872_0003_01_25
> Container Transitioned from NEW to RESERVED 2019-02-18
[jira] [Commented] (YARN-10497) Fix an issue in CapacityScheduler which fails to delete queues
[
https://issues.apache.org/jira/browse/YARN-10497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238683#comment-17238683
]
Gergely Pollak commented on YARN-10497:
---
[~wangda] you didn't use getStringCollection, but the method which is
responsible for the root cause did. The original implementation of
getSiblingQueues used getStringCollection instead of
getTrimmedStringCollection, and this caused the trimming issue. So if we only
update that method to use getTrimmedStringCollection, then the issue would be
fixed I think.
> Fix an issue in CapacityScheduler which fails to delete queues
> --
>
> Key: YARN-10497
> URL: https://issues.apache.org/jira/browse/YARN-10497
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
> Attachments: YARN-10497.001.patch, YARN-10497.002.patch,
> YARN-10497.003.patch
>
>
> We saw an exception when using queue mutation APIs:
> {code:java}
> 2020-11-13 16:47:46,327 WARN
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices:
> CapacityScheduler configuration validation failed:java.io.IOException: Queue
> root.am2cmQueueSecond not found
> {code}
> Which comes from this code:
> {code:java}
> List siblingQueues = getSiblingQueues(queueToRemove,
> proposedConf);
> if (!siblingQueues.contains(queueName)) {
> throw new IOException("Queue " + queueToRemove + " not found");
> }
> {code}
> (Inside MutableCSConfigurationProvider)
> If you look at the method:
> {code:java}
>
> private List getSiblingQueues(String queuePath, Configuration conf)
> {
> String parentQueue = queuePath.substring(0, queuePath.lastIndexOf('.'));
> String childQueuesKey = CapacitySchedulerConfiguration.PREFIX +
> parentQueue + CapacitySchedulerConfiguration.DOT +
> CapacitySchedulerConfiguration.QUEUES;
> return new ArrayList<>(conf.getStringCollection(childQueuesKey));
> }
> {code}
> And here's capacity-scheduler.xml I got
> {code:java}
> yarn.scheduler.capacity.root.queuesdefault, q1,
> q2
> {code}
> You can notice there're spaces between default, q1, a2
> So conf.getStringCollection returns:
> {code:java}
> default
> q1
> ...
> {code}
> Which causes match issue when we try to delete the queue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
