[ https://issues.apache.org/jira/browse/YARN-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055852#comment-14055852 ]
Sunil G commented on YARN-1408: ------------------------------- Thanks [~jianhe] for the comment. {code} // Remove from the list of newly allocated containers if found newlyAllocatedContainers.remove(rmContainer); {code} In *FicaSchedulerApp#containerCompleted*, the preempted container is removed from newlyAllocatedContainers (as in patch). So pullNewlyAllocatedContainersAndNMTokens will not get this container from newlyAllocatedContainers. So duplicate containers will not reach AM. I missed to add this to FSSchedulerApp. Please check this part and let me know if I am inline with your scenario. > Preemption caused Invalid State Event: ACQUIRED at KILLED and caused a task > timeout for 30mins > ---------------------------------------------------------------------------------------------- > > Key: YARN-1408 > URL: https://issues.apache.org/jira/browse/YARN-1408 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager > Affects Versions: 2.2.0 > Reporter: Sunil G > Assignee: Sunil G > Attachments: Yarn-1408.1.patch, Yarn-1408.2.patch, Yarn-1408.3.patch, > Yarn-1408.4.patch, Yarn-1408.5.patch, Yarn-1408.6.patch, Yarn-1408.7.patch, > Yarn-1408.patch > > > Capacity preemption is enabled as follows. > * yarn.resourcemanager.scheduler.monitor.enable= true , > * > yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy > Queue = a,b > Capacity of Queue A = 80% > Capacity of Queue B = 20% > Step 1: Assign a big jobA on queue a which uses full cluster capacity > Step 2: Submitted a jobB to queue b which would use less than 20% of cluster > capacity > JobA task which uses queue b capcity is been preempted and killed. > This caused below problem: > 1. New Container has got allocated for jobA in Queue A as per node update > from an NM. > 2. This container has been preempted immediately as per preemption. > Here ACQUIRED at KILLED Invalid State exception came when the next AM > heartbeat reached RM. > ERROR > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > Can't handle this event at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > ACQUIRED at KILLED > This also caused the Task to go for a timeout for 30minutes as this Container > was already killed by preemption. > attempt_1380289782418_0003_m_000000_0 Timed out after 1800 secs -- This message was sent by Atlassian JIRA (v6.2#6252)