[
https://issues.apache.org/jira/browse/YARN-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tarun Parimi resolved YARN-10378.
---------------------------------
Resolution: Duplicate
Looks like YARN-10034 fixes this issue for NM going down scenario also. Closing
as duplicate.
> When NM goes down and comes back up, PC allocation tags are not removed for
> completed containers
> ------------------------------------------------------------------------------------------------
>
> Key: YARN-10378
> URL: https://issues.apache.org/jira/browse/YARN-10378
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Affects Versions: 3.2.0, 3.1.1
> Reporter: Tarun Parimi
> Assignee: Tarun Parimi
> Priority: Major
>
> We are using placement constaints anti-affinity in an application along with
> node label. The application requests two containers with anti affinity on the
> node label containing only two nodes.
> So two containers will be allocated in the two nodes, one on each node
> satisfying anti-affinity.
> When one nodemanager goes down for some time, the node is marked as lost by
> RM and then it will kill all containers in that node.
> The AM will now have one pending container request, since the previous
> container got killed.
> When the Nodemanager becomes up after some time, the pending container is not
> getting allocated in that node again and the application has to wait forever
> for that container.
> If the ResourceManager is restarted, this issue disappears and the container
> gets allocated on the NodeManager which came back up recently.
> This seems to be an issue with the allocation tags not removed.
> The allocation tag is added for the container
> container_e68_1595886973474_0005_01_000003 .
> {code:java}
> 2020-07-28 17:02:04,091 DEBUG constraint.AllocationTagsManager
> (AllocationTagsManager.java:addContainer(355)) - Added
> container=container_e68_1595886973474_0005_01_000003 with tags=[hbase]\
> {code}
> However, the allocation tag is not removed when the container
> container_e68_1595886973474_0005_01_000003 is released. There is no
> equivalent DEBUG message seen for removing tags. This means that the tags are
> not getting removed. If the tag is not removed, then scheduler will not
> allocate in the same node due to anti-affinity resulting in the issue
> observed.
> {code:java}
> 2020-07-28 17:19:34,353 DEBUG scheduler.AbstractYarnScheduler
> (AbstractYarnScheduler.java:updateCompletedContainers(1038)) - Container
> FINISHED: container_e68_1595886973474_0005_01_000003
> 2020-07-28 17:19:34,353 INFO scheduler.AbstractYarnScheduler
> (AbstractYarnScheduler.java:completedContainer(669)) - Container
> container_e68_1595886973474_0005_01_000003 completed with event FINISHED, but
> corresponding RMContainer doesn't exist.
> {code}
> This seems to be due to changes done in YARN-8511 . Change here was made to
> remove the tags only after NM confirms container is released. However, in our
> scenario this is not happening. So the tag will never get removed until RM
> restart.
> Reverting YARN-8511 fixes this particular issue and tags are getting removed.
> But this is not a valid solution since the problem that YARN-8511 solves is
> also valid. We need to find a solution which does not break YARN-8511 and
> also fixes this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
