[
https://issues.apache.org/jira/browse/TEZ-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092516#comment-18092516
]
Ryu Kobayashi commented on TEZ-4732:
------------------------------------
PR: https://github.com/apache/tez/pull/515
> MergeManager: deadlock when usedMemory exceeds limit but commitMemory is
> below merge threshold
> ----------------------------------------------------------------------------------------------
>
> Key: TEZ-4732
> URL: https://issues.apache.org/jira/browse/TEZ-4732
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Ryu Kobayashi
> Priority: Major
>
> Problem
> In the ordered grouped shuffle, the shuffle scheduler can hang indefinitely
> due to a deadlock in MergeManager. Once the deadlock occurs, the task makes
> no progress and eventually times out.
> Root Cause
> MergeManager.waitForShuffleToMergeMemory() waits with no timeout until
> usedMemory <= memoryLimit. The only thread that schedules fetchers
> (ShuffleSchedulerCallable) calls this method before scheduling the next
> fetch. This creates a circular deadlock under the following condition:
> - usedMemory > memoryLimit: memory is full, no new data can be accepted
> - commitMemory < mergeThreshold: not enough committed data to trigger an
> in-memory merge
> In this state:
> # The fetcher scheduler is blocked waiting for memory to be freed
> # Memory is freed only after a merge completes
> # A merge is triggered only when commitMemory >= mergeThreshold
> # commitMemory increases only when fetchers commit new data
> # Fetchers are not scheduled because the scheduler is blocked
> There is no code path to break out of this cycle.
> Conditions That Make This Likely
> The deadlock window is narrow under normal memory configurations, but becomes
> much more likely when per-input memory is small. This happens when a Reducer
> reads from a large number of source vertices (e.g., via many UNION inputs),
> causing Tez to allocate an independent MergeManager per source with a
> proportionally small memoryLimit. If maxSingleShuffleLimit exceeds the gap
> between mergeThreshold and memoryLimit, a single fetch response can push
> usedMemory over the limit before commitMemory reaches the threshold,
> immediately entering the deadlock state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
