Zhu Zhu created FLINK-13421: ------------------------------- Summary: Unexpected ConcurrentModificationException when RM notify JM about allocation failure Key: FLINK-13421 URL: https://issues.apache.org/jira/browse/FLINK-13421 Project: Flink Issue Type: Bug Reporter: Zhu Zhu
When a TM lost and RM identified it first, it will notify JM about it through JobMaster#notifyAllocationFailure. We observed unexpected ConcurrentModificationException in this process, stack as below: Caused by: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) at java.util.HashMap$ValueIterator.next(HashMap.java:1466) at org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:477) at org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149) at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:712) at org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:692) at org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:538) at org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:664) ... 26 more This can cause a allocated slot removed from SlotPool#allocatedSlots but not all of its payload tasks get failed. Tasks may hang in scheduled forever, as in the attached log. It is not figured out yet that how a concurrent modification can happen. We do not have a debug log for it and is not able to re-pro it with debug log enabled yet. However, we can let SlotSharingManager$MultiTaskSlot do not iterate on its children directly to avoid ConcurrentModificationException to occur in any case. -- This message was sent by Atlassian JIRA (v7.6.14#76016)