[ https://issues.apache.org/jira/browse/FLINK-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896835#comment-16896835 ]
Zhu Zhu commented on FLINK-13421: --------------------------------- Hi [~till.rohrmann], the PR is opened. The solution is a bit different from what was proposed. Only the SlotSharingManager#listResolvedRootSlotInfo is changed. SlotSharingManager#getUnresolvedRootSlot and MultiTaskSlot#release are not changed as they are not necessary or may cause some other inconsistency issues. > Unexpected ConcurrentModificationException when RM notifies JM about > allocation failure > --------------------------------------------------------------------------------------- > > Key: FLINK-13421 > URL: https://issues.apache.org/jira/browse/FLINK-13421 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination > Affects Versions: 1.9.0 > Reporter: Zhu Zhu > Assignee: Zhu Zhu > Priority: Blocker > Labels: pull-request-available > Fix For: 1.9.0 > > Attachments: jm_concurrentmodification.log > > Time Spent: 10m > Remaining Estimate: 0h > > When a TM lost and RM identified it first, it will notify JM about it through > JobMaster#notifyAllocationFailure. > We observed unexpected ConcurrentModificationException in this process, stack > as below: > > Caused by: java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > at java.util.HashMap$ValueIterator.next(HashMap.java:1466) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:477) > at > org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:712) > at > org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:692) > at > org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:538) > at > org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:664) > ... 26 more > > This can cause an allocated slot to be removed from SlotPool#allocatedSlots > but not all of its payload tasks get failed. Tasks may hang in scheduled > forever, not able to fail or timeout, as in the attached log. > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)