[
https://issues.apache.org/jira/browse/HIVE-29780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29780:
--------------------------------
Description:
{{LlapTaskCommunicator}} and {{LlapTaskSchedulerService}} pair up at
construction using a single class-static slot ({{{}pluginInitLock{}}} +
{{{}static … instance{}}}). When two DAGs are constructed concurrently in the
same JVM, the second-to-arrive plugin from DAG-A can accidentally pair with the
plugin from DAG-B. DAG-A's real plugin is then left with {{scheduler == null}}
and the first task submission fails with:
{code:java}
6-07-29T06:48:51,723 INFO [pool-163-thread-1] tezplugins.LlapTaskCommunicator:
Failed to run task: attempt_1785332929111_0001_1_00_000000_0 on containerId:
container_111101111_0001_00_000003
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
because "this.this$0.scheduler" is null
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:540)
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:517)
at
org.apache.hadoop.hive.llap.AsyncResponseHandler$AsyncResponseHandlerRunnable.run(AsyncResponseHandler.java:90)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
{code}
A production Tez {{DAGAppMaster}} runs one DAG at a time per AM, so the two
LLAP plugins for that DAG are never in a race with plugins for a different DAG
in the same JVM. The issue only surfaces in JVMs that host many concurrent DAGs
— MiniHS2 / MiniLlapCluster / anything that shares an AM classloader across
DAGs. Repro seen locally with a 30-way concurrent {{INSERT}} against a MiniHS2
LLAP cluster: 5 of 30 sessions failed with the NPE.
The above failure eventually led to a query failure:
{code}
2026-07-29T07:37:10,980 ERROR [HiveServer2-Background-Pool: Thread-1165]
ql.Driver: FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
vertexId=vertex_1785335826346_0001_1_00, diagnostics=[Task failed,
taskId=task_1785335826346_0001_1_00_000000, diagnostics=[TaskAttempt 0 failed,
info=[Cannot invoke
"org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
because "this.this$0.scheduler" is null], TaskAttempt 1 failed, info=[Cannot
invoke
"org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
because "this.this$0.scheduler" is null], TaskAttempt 2 failed, info=[Cannot
invoke
"org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
because "this.this$0.scheduler" is null], TaskAttempt 3 failed, info=[Cannot
invoke
"org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
because "this.this$0.scheduler" is null]], Vertex did not succeed due to
OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex
vertex_1785335826346_0001_1_00 [Map 1] killed/failed due
to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2,
vertexId=vertex_1785335826346_0001_1_01, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE,
failedTasks:0 killedTasks:2, Vertex vertex_1785335826346_0001_1_01 [Reducer 2]
killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed, vertexName=Reducer 3,
vertexId=vertex_1785335826346_0001_1_02, diagnostics=[Vertex received Kill
while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE,
failedTasks:0 killedTasks:2, Vertex vertex_1785335826346_0001_1_02 [Reducer 3]
killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to
VERTEX_FAILURE. failedVertices:1 killedVertices:2
{code}
*Production impact:*
None — the map contains at most *one* entry at any time in a production AM (one
DAG per AM, and the entry is removed the moment the second plugin arrives, or
on {{{}shutdown(){}}}). A {{{}ConcurrentHashMap.put{}}}/{{{}remove{}}} pair per
DAG init is nanoseconds; the existing {{synchronized (pluginInitLock)}} section
was already there. No new allocations on the hot path (task submission,
scheduling), only at plugin construction / shutdown.
was:
{{LlapTaskCommunicator}} and {{LlapTaskSchedulerService}} pair up at
construction using a single class-static slot ({{pluginInitLock}} + {{static …
instance}}). When two DAGs are constructed concurrently in the same JVM, the
second-to-arrive plugin from DAG-A can accidentally pair with the plugin from
DAG-B. DAG-A's real plugin is then left with {{scheduler == null}} and the
first task submission fails with:
{code}
6-07-29T06:48:51,723 INFO [pool-163-thread-1] tezplugins.LlapTaskCommunicator:
Failed to run task: attempt_1785332929111_0001_1_00_000000_0 on containerId:
container_111101111_0001_00_000003
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
because "this.this$0.scheduler" is null
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:540)
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:517)
at
org.apache.hadoop.hive.llap.AsyncResponseHandler$AsyncResponseHandlerRunnable.run(AsyncResponseHandler.java:90)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
{code}
A production Tez {{DAGAppMaster}} runs one DAG at a time per AM, so the two
LLAP plugins for that DAG are never in a race with plugins for a different DAG
in the same JVM. The issue only surfaces in JVMs that host many concurrent DAGs
— MiniHS2 / MiniLlapCluster / anything that shares an AM classloader across
DAGs. Repro seen locally with a 30-way concurrent {{INSERT}} against a MiniHS2
LLAP cluster: 5 of 30 sessions failed with the NPE.
please find: [^hive_LLAP_NPE.log.zip]
*Production impact:*
None — the map contains at most *one* entry at any time in a production AM (one
DAG per AM, and the entry is removed the moment the second plugin arrives, or
on {{shutdown()}}). A {{ConcurrentHashMap.put}}/{{remove}} pair per DAG init is
nanoseconds; the existing {{synchronized (pluginInitLock)}} section was already
there. No new allocations on the hot path (task submission, scheduling), only
at plugin construction / shutdown.
> Fix LLAP plugin rendezvous race under concurrent DAGs (NullPointerException
> in LlapTaskCommunicator.setResponse)
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29780
> URL: https://issues.apache.org/jira/browse/HIVE-29780
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
>
> {{LlapTaskCommunicator}} and {{LlapTaskSchedulerService}} pair up at
> construction using a single class-static slot ({{{}pluginInitLock{}}} +
> {{{}static … instance{}}}). When two DAGs are constructed concurrently in the
> same JVM, the second-to-arrive plugin from DAG-A can accidentally pair with
> the plugin from DAG-B. DAG-A's real plugin is then left with {{scheduler ==
> null}} and the first task submission fails with:
> {code:java}
> 6-07-29T06:48:51,723 INFO [pool-163-thread-1]
> tezplugins.LlapTaskCommunicator: Failed to run task:
> attempt_1785332929111_0001_1_00_000000_0 on containerId:
> container_111101111_0001_00_000003
> java.lang.NullPointerException: Cannot invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:540)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:517)
> at
> org.apache.hadoop.hive.llap.AsyncResponseHandler$AsyncResponseHandlerRunnable.run(AsyncResponseHandler.java:90)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> {code}
> A production Tez {{DAGAppMaster}} runs one DAG at a time per AM, so the two
> LLAP plugins for that DAG are never in a race with plugins for a different
> DAG in the same JVM. The issue only surfaces in JVMs that host many
> concurrent DAGs — MiniHS2 / MiniLlapCluster / anything that shares an AM
> classloader across DAGs. Repro seen locally with a 30-way concurrent
> {{INSERT}} against a MiniHS2 LLAP cluster: 5 of 30 sessions failed with the
> NPE.
> The above failure eventually led to a query failure:
> {code}
> 2026-07-29T07:37:10,980 ERROR [HiveServer2-Background-Pool: Thread-1165]
> ql.Driver: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1785335826346_0001_1_00, diagnostics=[Task failed,
> taskId=task_1785335826346_0001_1_00_000000, diagnostics=[TaskAttempt 0
> failed, info=[Cannot invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null], TaskAttempt 1 failed, info=[Cannot
> invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null], TaskAttempt 2 failed, info=[Cannot
> invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null], TaskAttempt 3 failed, info=[Cannot
> invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null]], Vertex did not succeed due to
> OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex
> vertex_1785335826346_0001_1_00 [Map 1] killed/failed due
> to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2,
> vertexId=vertex_1785335826346_0001_1_01, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE,
> failedTasks:0 killedTasks:2, Vertex vertex_1785335826346_0001_1_01 [Reducer
> 2] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed,
> vertexName=Reducer 3, vertexId=vertex_1785335826346_0001_1_02,
> diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not
> succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:2, Vertex
> vertex_1785335826346_0001_1_02 [Reducer 3] killed/failed due
> to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE.
> failedVertices:1 killedVertices:2
> {code}
>
> *Production impact:*
> None — the map contains at most *one* entry at any time in a production AM
> (one DAG per AM, and the entry is removed the moment the second plugin
> arrives, or on {{{}shutdown(){}}}). A
> {{{}ConcurrentHashMap.put{}}}/{{{}remove{}}} pair per DAG init is
> nanoseconds; the existing {{synchronized (pluginInitLock)}} section was
> already there. No new allocations on the hot path (task submission,
> scheduling), only at plugin construction / shutdown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
