[
https://issues.apache.org/jira/browse/FLINK-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892776#comment-16892776
]
Zhu Zhu commented on FLINK-13421:
---------------------------------
I think I find the cause.
ConcurrentModificationException happened in the children release loop in
_*MultiTaskSlot#release*_.
It happened because new slot allocations happened, adding a new child to the
MultiTaskSlot, thus modified the *_children_* field.
New slot allocations happened because a task failover is caused by a child
slot's payload releasing. The failover canceled tasks and returned allocated
slots to SlotPool. The returned allocated slots were assigned to some tasks
that were not canceled yet. The slot assignment satisfies the _*input location
preference constraints*_ of some downstream tasks, and triggered the slot
allocation of these tasks.
To fix this, I think we may change _*MultiTaskSlot*_ to be not available for
slot allocation(in SlotSharingManager#listResolvedRootSlotInfo and
SlotSharingManager#getUnresolvedRootSlot) when it is in _*releasingChildren*_
state. Besides, changing _*MultiTaskSlot#release*_ to not iterate on its
_*children*_ field directly would be better to avoid
ConcurrentModificationException to happen in any case.
Hi [~till.rohrmann], do you have any suggestion for it?
> Unexpected ConcurrentModificationException when RM notifies JM about
> allocation failure
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-13421
> URL: https://issues.apache.org/jira/browse/FLINK-13421
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.9.0
> Reporter: Zhu Zhu
> Priority: Blocker
> Attachments: jm_concurrentmodification.log
>
>
> When a TM lost and RM identified it first, it will notify JM about it through
> JobMaster#notifyAllocationFailure.
> We observed unexpected ConcurrentModificationException in this process, stack
> as below:
>
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> at java.util.HashMap$ValueIterator.next(HashMap.java:1466)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:477)
> at
> org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:712)
> at
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:692)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:538)
> at
> org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:664)
> ... 26 more
>
> This can cause an allocated slot to be removed from SlotPool#allocatedSlots
> but not all of its payload tasks get failed. Tasks may hang in scheduled
> forever, not able to fail or timeout, as in the attached log.
> It is not figured out yet that how a concurrent modification can happen. We
> do not have a debug log for it and is not able to re-pro it with debug log
> enabled yet.
> However, we can let SlotSharingManager$MultiTaskSlot do not iterate on its
> children directly to avoid ConcurrentModificationException to occur in any
> case.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
