[jira] [Updated] (YARN-10705) Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10705:
---
Affects Version/s: 3.4.0
> Misleading DEBUG log for container assignment needs to be removed when the
> container is actually reserved, not assigned in FairScheduler
>
>
> Key: YARN-10705
> URL: https://issues.apache.org/jira/browse/YARN-10705
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
>
> Following DEBUG logs are logged if a container reservation is made when a
> node has been offered to the queue in FairScheduler:
> {code}
> 2021-02-10 07:33:55,049 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> application_1610442362681_2607's resource request is reserved.
> 2021-02-10 07:33:55,049 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> Assigned container in queue:root.pj_dc_pe container:
> {code}
> The latter log from above seems to indicate a bad container assignment with
> resource allocation, whereas, in actual, it is a bad
> log which shouldn't have been logged in the first place.
> This log comes from [1] after an application attempt with an unmet demand is
> checked for container assignment/reservation.
> If the container for this app attempt is reserved on the node, then, it
> returns from [2].
> From [3]:
> {quote}
>* If an assignment was made, returns the resources allocated to the
>* container. If a reservation was made, returns
>* FairScheduler.CONTAINER_RESERVED. If no assignment or reservation
> was
>* made, returns an empty resource.
> {quote}
> We are checking for the empty resource at [4], but not
> FairScheduler.CONTAINER_RESERVED before logging out a message for container
> assignment specifically which is incorrect.
> Instead of:
> {code}
> if (!assigned.equals(none())) {
> LOG.debug("Assigned container in queue:{} container:{}",
> getName(), assigned);
> break;
> }
> {code}
> it should be:
> {code}
> // check if an assignment or a reservation was made.
> if (!assigned.equals(none())) {
> // only log container assignment if there is
> // an actual assignment, not a reservation.
> if (!assigned.equals(FairScheduler.CONTAINER_RESERVED)
> && LOG.isDebugEnabled()) {
> LOG.debug("Assigned container in queue:" + getName() + " " +
> "container:" + assigned);
> }
> break;
> }
> {code}
> [1]
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java#L356
> [2]
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L911
> [3]
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L842
> [4]
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java#L355
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10705) Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Ahuja updated YARN-10705:
---
Component/s: yarn
> Misleading DEBUG log for container assignment needs to be removed when the
> container is actually reserved, not assigned in FairScheduler
>
>
> Key: YARN-10705
> URL: https://issues.apache.org/jira/browse/YARN-10705
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Minor
>
> Following DEBUG logs are logged if a container reservation is made when a
> node has been offered to the queue in FairScheduler:
> {code}
> 2021-02-10 07:33:55,049 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt:
> application_1610442362681_2607's resource request is reserved.
> 2021-02-10 07:33:55,049 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue:
> Assigned container in queue:root.pj_dc_pe container:
> {code}
> The latter log from above seems to indicate a bad container assignment with
> resource allocation, whereas, in actual, it is a bad
> log which shouldn't have been logged in the first place.
> This log comes from [1] after an application attempt with an unmet demand is
> checked for container assignment/reservation.
> If the container for this app attempt is reserved on the node, then, it
> returns from [2].
> From [3]:
> {quote}
>* If an assignment was made, returns the resources allocated to the
>* container. If a reservation was made, returns
>* FairScheduler.CONTAINER_RESERVED. If no assignment or reservation
> was
>* made, returns an empty resource.
> {quote}
> We are checking for the empty resource at [4], but not
> FairScheduler.CONTAINER_RESERVED before logging out a message for container
> assignment specifically which is incorrect.
> Instead of:
> {code}
> if (!assigned.equals(none())) {
> LOG.debug("Assigned container in queue:{} container:{}",
> getName(), assigned);
> break;
> }
> {code}
> it should be:
> {code}
> // check if an assignment or a reservation was made.
> if (!assigned.equals(none())) {
> // only log container assignment if there is
> // an actual assignment, not a reservation.
> if (!assigned.equals(FairScheduler.CONTAINER_RESERVED)
> && LOG.isDebugEnabled()) {
> LOG.debug("Assigned container in queue:" + getName() + " " +
> "container:" + assigned);
> }
> break;
> }
> {code}
> [1]
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java#L356
> [2]
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L911
> [3]
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java#L842
> [4]
> https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java#L355
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
