[
https://issues.apache.org/jira/browse/HIVE-30058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-30058:
--------------------------------
Description:
LLAP task deadlock when hive.llap.daemon.task.scheduler.enable.preemption=false
and a downstream vertex holds executors blocked on a broadcast/runtime-value
edge from another vertex sharing the pool.
TPCDS Q1:
{code}
| Map 1 <- Map 11 (BROADCAST_EDGE), Reducer 10 (BROADCAST_EDGE),
Reducer 4 (BROADCAST_EDGE) |
| Map 5 <- Reducer 2 (BROADCAST_EDGE), Reducer 9 (BROADCAST_EDGE) |
| Map 7 <- Map 11 (BROADCAST_EDGE), Reducer 4 (BROADCAST_EDGE) |
| Reducer 10 <- Reducer 9 (CUSTOM_SIMPLE_EDGE) |
| Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (BROADCAST_EDGE) |
| Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE) |
| Reducer 6 <- Map 5 (SIMPLE_EDGE) |
| Reducer 8 <- Map 7 (SIMPLE_EDGE) |
| Reducer 9 <- Reducer 8 (SIMPLE_EDGE) |
{code}
{code}
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
----------------------------------------------------------------------------------------------
Map 11 ......... llap SUCCEEDED 1 1 0 0
0 0
Map 3 .......... llap SUCCEEDED 9 9 0 0
0 0
Reducer 4 ...... llap SUCCEEDED 1 1 0 0
0 0
Map 7 .......... llap SUCCEEDED 218 218 0 0
0 0
Reducer 8 ...... llap SUCCEEDED 62 62 0 0
0 0
Reducer 9 .... llap RUNNING 40 32 8 0
0 0
Reducer 10 llap RUNNING 1 0 1 0
1 0
Map 1 llap RUNNING 218 0 218 0
129 88
Reducer 2 llap INITED 486 0 0 486
0 0
Map 5 llap INITED 202 0 0 202
0 0
Reducer 6 llap INITED 1 0 0 1
0 0
----------------------------------------------------------------------------------------------
VERTICES: 05/11 [======>>--------------------] 26% ELAPSED TIME: 603.95 s
----------------------------------------------------------------------------------------------
{code}
{code}
Map 1 (218 running, holds every executor)
└─ blocked on ShuffleManager: Reducer_10 -> Map_1 (broadcast /
DynamicValueRegistry)
│
▼
Reducer 10 (1/1 running, needs to finish shuffling from Reducer 9)
└─ blocked on Reducer 9's remaining output
│
▼
Reducer 9 (32/40 done, 8 "RUNNING" but actually unschedulable)
└─ blocked waiting for an executor slot
│
▼
… which Map 1 is holding. Cycle closed.
{code}
- Map 1 is the jailer: 218 attempts occupy all 120 executors. Each one is
sitting in ShuffleManager for the Reducer_10 → Map_1 broadcast edge (the
mapjoin/DPP runtime value). It cannot make forward progress and — with
preemption off — cannot be evicted.
- Reducer 10 has been admitted (1 running) but can't complete because Reducer
9's tail hasn't produced its final events yet.
- Reducer 9 shows 8 as "RUNNING" in the UI, but that just means the AM has
issued the attempts; the daemons keep rejecting/queueing them because every
executor is held by Map 1. Effectively pending.
- The 129 failed + 88 killed on Map 1 are the debris from before you flipped
preemption off — earlier attempts that were preempted or timed out. The current
218 "RUNNING" are the retries, and now nothing can dislodge them.
- Reducer 2 / Map 5 / Reducer 6 are INITED — they're downstream of the mess and
will never see slots.
The chain reads cleanly left-to-right: Map 1 → (broadcast dep) → Reducer 10 →
(shuffle dep) → Reducer 9 → (resource dep) → Map 1. Classic resource-vs-data
circular wait, and it went from "preemption papers over it" to "preemption was
the only thing papering over it" the moment we set the flag to false.
was:
LLAP task deadlock when hive.llap.daemon.task.scheduler.enable.preemption=false
and a downstream vertex holds executors blocked on a broadcast/runtime-value
edge from another vertex sharing the pool.
TPCDS Q1:
{code}
| Map 1 <- Map 11 (BROADCAST_EDGE), Reducer 10 (BROADCAST_EDGE),
Reducer 4 (BROADCAST_EDGE) |
| Map 5 <- Reducer 2 (BROADCAST_EDGE), Reducer 9 (BROADCAST_EDGE) |
| Map 7 <- Map 11 (BROADCAST_EDGE), Reducer 4 (BROADCAST_EDGE) |
| Reducer 10 <- Reducer 9 (CUSTOM_SIMPLE_EDGE) |
| Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (BROADCAST_EDGE) |
| Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE) |
| Reducer 6 <- Map 5 (SIMPLE_EDGE) |
| Reducer 8 <- Map 7 (SIMPLE_EDGE) |
| Reducer 9 <- Reducer 8 (SIMPLE_EDGE) |
{code}
{code}
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
----------------------------------------------------------------------------------------------
Map 11 ......... llap SUCCEEDED 1 1 0 0
0 0
Map 3 .......... llap SUCCEEDED 9 9 0 0
0 0
Reducer 4 ...... llap SUCCEEDED 1 1 0 0
0 0
Map 7 .......... llap SUCCEEDED 218 218 0 0
0 0
Reducer 8 ...... llap SUCCEEDED 62 62 0 0
0 0
Reducer 9 .... llap RUNNING 40 32 8 0
0 0
Reducer 10 llap RUNNING 1 0 1 0
1 0
Map 1 llap RUNNING 218 0 218 0
129 88
Reducer 2 llap INITED 486 0 0 486
0 0
Map 5 llap INITED 202 0 0 202
0 0
Reducer 6 llap INITED 1 0 0 1
0 0
----------------------------------------------------------------------------------------------
VERTICES: 05/11 [======>>--------------------] 26% ELAPSED TIME: 603.95 s
----------------------------------------------------------------------------------------------
{code}
{code}
Map 1 (218 running, holds every executor)
└─ blocked on ShuffleManager: Reducer_10 -> Map_1 (broadcast /
DynamicValueRegistry)
│
▼
Reducer 10 (1/1 running, needs to finish shuffling from Reducer 9)
└─ blocked on Reducer 9's remaining output
│
▼
Reducer 9 (32/40 done, 8 "RUNNING" but actually unschedulable)
└─ blocked waiting for an executor slot
│
▼
… which Map 1 is holding. Cycle closed.
- Map 1 is the jailer: 218 attempts occupy all 120 executors. Each one is
sitting in ShuffleManager for the Reducer_10 → Map_1 broadcast edge (the
mapjoin/DPP runtime value). It cannot make forward progress and — with
preemption off — cannot be evicted.
- Reducer 10 has been admitted (1 running) but can't complete because Reducer
9's tail hasn't produced its final events yet.
- Reducer 9 shows 8 as "RUNNING" in the UI, but that just means the AM has
issued the attempts; the daemons keep rejecting/queueing them because every
executor is held by Map 1. Effectively pending.
- The 129 failed + 88 killed on Map 1 are the debris from before you flipped
preemption off — earlier attempts that were preempted or timed out. The current
218 "RUNNING" are the retries, and now nothing can dislodge them.
- Reducer 2 / Map 5 / Reducer 6 are INITED — they're downstream of the mess and
will never see slots.
The chain reads cleanly left-to-right: Map 1 → (broadcast dep) → Reducer 10 →
(shuffle dep) → Reducer 9 → (resource dep) → Map 1. Classic resource-vs-data
circular wait, and it went from "preemption papers over it" to "preemption was
the only thing papering over it" the moment we set the flag to false.
{code}
> LLAP task deadlock when preemption=false and a downstream vertex holds
> executors blocked on a broadcast/runtime-value edge from another vertex
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30058
> URL: https://issues.apache.org/jira/browse/HIVE-30058
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Priority: Major
>
> LLAP task deadlock when
> hive.llap.daemon.task.scheduler.enable.preemption=false and a downstream
> vertex holds executors blocked on a broadcast/runtime-value edge from another
> vertex sharing the pool.
> TPCDS Q1:
> {code}
> | Map 1 <- Map 11 (BROADCAST_EDGE), Reducer 10 (BROADCAST_EDGE),
> Reducer 4 (BROADCAST_EDGE) |
> | Map 5 <- Reducer 2 (BROADCAST_EDGE), Reducer 9 (BROADCAST_EDGE) |
> | Map 7 <- Map 11 (BROADCAST_EDGE), Reducer 4 (BROADCAST_EDGE) |
> | Reducer 10 <- Reducer 9 (CUSTOM_SIMPLE_EDGE) |
> | Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (BROADCAST_EDGE) |
> | Reducer 4 <- Map 3 (CUSTOM_SIMPLE_EDGE) |
> | Reducer 6 <- Map 5 (SIMPLE_EDGE) |
> | Reducer 8 <- Map 7 (SIMPLE_EDGE) |
> | Reducer 9 <- Reducer 8 (SIMPLE_EDGE) |
> {code}
> {code}
> ----------------------------------------------------------------------------------------------
> VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING
> FAILED KILLED
> ----------------------------------------------------------------------------------------------
> Map 11 ......... llap SUCCEEDED 1 1 0 0
> 0 0
> Map 3 .......... llap SUCCEEDED 9 9 0 0
> 0 0
> Reducer 4 ...... llap SUCCEEDED 1 1 0 0
> 0 0
> Map 7 .......... llap SUCCEEDED 218 218 0 0
> 0 0
> Reducer 8 ...... llap SUCCEEDED 62 62 0 0
> 0 0
> Reducer 9 .... llap RUNNING 40 32 8 0
> 0 0
> Reducer 10 llap RUNNING 1 0 1 0
> 1 0
> Map 1 llap RUNNING 218 0 218 0
> 129 88
> Reducer 2 llap INITED 486 0 0 486
> 0 0
> Map 5 llap INITED 202 0 0 202
> 0 0
> Reducer 6 llap INITED 1 0 0 1
> 0 0
> ----------------------------------------------------------------------------------------------
> VERTICES: 05/11 [======>>--------------------] 26% ELAPSED TIME: 603.95 s
> ----------------------------------------------------------------------------------------------
> {code}
> {code}
> Map 1 (218 running, holds every executor)
> └─ blocked on ShuffleManager: Reducer_10 -> Map_1 (broadcast /
> DynamicValueRegistry)
> │
> ▼
> Reducer 10 (1/1 running, needs to finish shuffling from Reducer 9)
> └─ blocked on Reducer 9's remaining output
> │
> ▼
> Reducer 9 (32/40 done, 8 "RUNNING" but actually unschedulable)
> └─ blocked waiting for an executor slot
> │
> ▼
> … which Map 1 is holding. Cycle closed.
> {code}
> - Map 1 is the jailer: 218 attempts occupy all 120 executors. Each one is
> sitting in ShuffleManager for the Reducer_10 → Map_1 broadcast edge (the
> mapjoin/DPP runtime value). It cannot make forward progress and — with
> preemption off — cannot be evicted.
> - Reducer 10 has been admitted (1 running) but can't complete because Reducer
> 9's tail hasn't produced its final events yet.
> - Reducer 9 shows 8 as "RUNNING" in the UI, but that just means the AM has
> issued the attempts; the daemons keep rejecting/queueing them because every
> executor is held by Map 1. Effectively pending.
> - The 129 failed + 88 killed on Map 1 are the debris from before you flipped
> preemption off — earlier attempts that were preempted or timed out. The
> current 218 "RUNNING" are the retries, and now nothing can dislodge them.
> - Reducer 2 / Map 5 / Reducer 6 are INITED — they're downstream of the mess
> and will never see slots.
> The chain reads cleanly left-to-right: Map 1 → (broadcast dep) → Reducer 10 →
> (shuffle dep) → Reducer 9 → (resource dep) → Map 1. Classic resource-vs-data
> circular wait, and it went from "preemption papers over it" to "preemption
> was the only thing papering over it" the moment we set the flag to false.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
