Rui Fan created FLINK-40525:
-------------------------------
Summary: RecoveredInputChannel.finishReadRecoveredState holds
receivedBuffers across notifyChannelNonEmpty, risking an ABBA deadlock
Key: FLINK-40525
URL: https://issues.apache.org/jira/browse/FLINK-40525
Project: Flink
Issue Type: Sub-task
Components: Runtime / Checkpointing
Reporter: Rui Fan
Assignee: Rui Fan
`finishReadRecoveredState` wraps its `onRecoveredStateBuffer` call in an extra
`synchronized (receivedBuffers)` [1]. On the `wasEmpty` branch that call
reaches `notifyChannelNonEmpty()` → `SingleInputGate.queueChannel`, which takes
`inputChannelsWithData` — so the IO thread holds `receivedBuffers` and then
acquires `inputChannelsWithData`. The task thread does the reverse:
`waitAndGetNextData` holds `inputChannelsWithData` and then acquires
`receivedBuffers` via `isReleased()`. When a channel is enqueued while its
`receivedBuffers` is empty (e.g. `setError` from
`BufferManager.notifyBufferAvailable`'s catch enqueues without adding a
buffer), the ABBA cycle closes: task holds `inputChannelsWithData` waiting on
`receivedBuffers`, IO holds `receivedBuffers` waiting on
`inputChannelsWithData`.
Fix: remove the outer `synchronized`, which is safe because
`onRecoveredStateBuffer` already locks `receivedBuffers` internally around the
actual add — the collection stays protected, and the outer lock only served to
hold it across `notifyChannelNonEmpty`, which is exactly what inverts the lock
order.
[1]
[https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RecoveredInputChannel.java#L163-L170]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
