Ryu Kobayashi created TEZ-4732:
----------------------------------
Summary: MergeManager: deadlock when usedMemory exceeds limit but
commitMemory is below merge threshold
Key: TEZ-4732
URL: https://issues.apache.org/jira/browse/TEZ-4732
Project: Apache Tez
Issue Type: Bug
Reporter: Ryu Kobayashi
Problem
In the ordered grouped shuffle, the shuffle scheduler can hang indefinitely due
to a deadlock in MergeManager. Once the deadlock occurs, the task makes no
progress and eventually times out.
Root Cause
MergeManager.waitForShuffleToMergeMemory() waits with no timeout until
usedMemory <= memoryLimit. The only thread that schedules fetchers
(ShuffleSchedulerCallable) calls this method before scheduling the next fetch.
This creates a circular deadlock under the following condition:
- usedMemory > memoryLimit: memory is full, no new data can be accepted
- commitMemory < mergeThreshold: not enough committed data to trigger an
in-memory merge
In this state:
# The fetcher scheduler is blocked waiting for memory to be freed
# Memory is freed only after a merge completes
# A merge is triggered only when commitMemory >= mergeThreshold
# commitMemory increases only when fetchers commit new data
# Fetchers are not scheduled because the scheduler is blocked
There is no code path to break out of this cycle.
Conditions That Make This Likely
The deadlock window is narrow under normal memory configurations, but becomes
much more likely when per-input memory is small. This happens when a Reducer
reads from a large number of source vertices (e.g., via many UNION inputs),
causing Tez to allocate an independent MergeManager per source with a
proportionally small memoryLimit. If maxSingleShuffleLimit exceeds the gap
between mergeThreshold and memoryLimit, a single fetch response can push
usedMemory over the limit before commitMemory reaches the threshold,
immediately entering the deadlock state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
