[ https://issues.apache.org/jira/browse/YARN-10378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tarun Parimi resolved YARN-10378. --------------------------------- Resolution: Duplicate Looks like YARN-10034 fixes this issue for NM going down scenario also. Closing as duplicate. > When NM goes down and comes back up, PC allocation tags are not removed for > completed containers > ------------------------------------------------------------------------------------------------ > > Key: YARN-10378 > URL: https://issues.apache.org/jira/browse/YARN-10378 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler > Affects Versions: 3.2.0, 3.1.1 > Reporter: Tarun Parimi > Assignee: Tarun Parimi > Priority: Major > > We are using placement constaints anti-affinity in an application along with > node label. The application requests two containers with anti affinity on the > node label containing only two nodes. > So two containers will be allocated in the two nodes, one on each node > satisfying anti-affinity. > When one nodemanager goes down for some time, the node is marked as lost by > RM and then it will kill all containers in that node. > The AM will now have one pending container request, since the previous > container got killed. > When the Nodemanager becomes up after some time, the pending container is not > getting allocated in that node again and the application has to wait forever > for that container. > If the ResourceManager is restarted, this issue disappears and the container > gets allocated on the NodeManager which came back up recently. > This seems to be an issue with the allocation tags not removed. > The allocation tag is added for the container > container_e68_1595886973474_0005_01_000003 . > {code:java} > 2020-07-28 17:02:04,091 DEBUG constraint.AllocationTagsManager > (AllocationTagsManager.java:addContainer(355)) - Added > container=container_e68_1595886973474_0005_01_000003 with tags=[hbase]\ > {code} > However, the allocation tag is not removed when the container > container_e68_1595886973474_0005_01_000003 is released. There is no > equivalent DEBUG message seen for removing tags. This means that the tags are > not getting removed. If the tag is not removed, then scheduler will not > allocate in the same node due to anti-affinity resulting in the issue > observed. > {code:java} > 2020-07-28 17:19:34,353 DEBUG scheduler.AbstractYarnScheduler > (AbstractYarnScheduler.java:updateCompletedContainers(1038)) - Container > FINISHED: container_e68_1595886973474_0005_01_000003 > 2020-07-28 17:19:34,353 INFO scheduler.AbstractYarnScheduler > (AbstractYarnScheduler.java:completedContainer(669)) - Container > container_e68_1595886973474_0005_01_000003 completed with event FINISHED, but > corresponding RMContainer doesn't exist. > {code} > This seems to be due to changes done in YARN-8511 . Change here was made to > remove the tags only after NM confirms container is released. However, in our > scenario this is not happening. So the tag will never get removed until RM > restart. > Reverting YARN-8511 fixes this particular issue and tags are getting removed. > But this is not a valid solution since the problem that YARN-8511 solves is > also valid. We need to find a solution which does not break YARN-8511 and > also fixes this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org