[
https://issues.apache.org/jira/browse/HIVE-29780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29780:
--------------------------------
Attachment: hive_LLAP_NPE.log.zip
> Fix LLAP plugin rendezvous race under concurrent DAGs (NullPointerException
> in LlapTaskCommunicator.setResponse)
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29780
> URL: https://issues.apache.org/jira/browse/HIVE-29780
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: hive_LLAP_NPE.log.zip
>
>
> {{LlapTaskCommunicator}} and {{LlapTaskSchedulerService}} pair up at
> construction using a single class-static slot ({{{}pluginInitLock{}}} +
> {{{}static … instance{}}}). When two DAGs are constructed concurrently in the
> same JVM, the second-to-arrive plugin from DAG-A can accidentally pair with
> the plugin from DAG-B. DAG-A's real plugin is then left with {{scheduler ==
> null}} and the first task submission fails with:
> {code:java}
> 6-07-29T06:48:51,723 INFO [pool-163-thread-1]
> tezplugins.LlapTaskCommunicator: Failed to run task:
> attempt_1785332929111_0001_1_00_000000_0 on containerId:
> container_111101111_0001_00_000003
> java.lang.NullPointerException: Cannot invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:540)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:517)
> at
> org.apache.hadoop.hive.llap.AsyncResponseHandler$AsyncResponseHandlerRunnable.run(AsyncResponseHandler.java:90)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> {code}
> A production Tez {{DAGAppMaster}} runs one DAG at a time per AM, so the two
> LLAP plugins for that DAG are never in a race with plugins for a different
> DAG in the same JVM. The issue only surfaces in JVMs that host many
> concurrent DAGs — MiniHS2 / MiniLlapCluster / anything that shares an AM
> classloader across DAGs. Repro seen locally with a 30-way concurrent
> {{INSERT}} against a MiniHS2 LLAP cluster: 5 of 30 sessions failed with the
> NPE.
> The above failure eventually led to a query failure:
> {code}
> 2026-07-29T07:37:10,980 ERROR [HiveServer2-Background-Pool: Thread-1165]
> ql.Driver: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1,
> vertexId=vertex_1785335826346_0001_1_00, diagnostics=[Task failed,
> taskId=task_1785335826346_0001_1_00_000000, diagnostics=[TaskAttempt 0
> failed, info=[Cannot invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null], TaskAttempt 1 failed, info=[Cannot
> invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null], TaskAttempt 2 failed, info=[Cannot
> invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null], TaskAttempt 3 failed, info=[Cannot
> invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null]], Vertex did not succeed due to
> OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex
> vertex_1785335826346_0001_1_00 [Map 1] killed/failed due
> to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2,
> vertexId=vertex_1785335826346_0001_1_01, diagnostics=[Vertex received Kill
> while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE,
> failedTasks:0 killedTasks:2, Vertex vertex_1785335826346_0001_1_01 [Reducer
> 2] killed/failed due to:OTHER_VERTEX_FAILURE]Vertex killed,
> vertexName=Reducer 3, vertexId=vertex_1785335826346_0001_1_02,
> diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not
> succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:2, Vertex
> vertex_1785335826346_0001_1_02 [Reducer 3] killed/failed due
> to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE.
> failedVertices:1 killedVertices:2
> {code}
>
> *Production impact:*
> None — the map contains at most *one* entry at any time in a production AM
> (one DAG per AM, and the entry is removed the moment the second plugin
> arrives, or on {{{}shutdown(){}}}). A
> {{{}ConcurrentHashMap.put{}}}/{{{}remove{}}} pair per DAG init is
> nanoseconds; the existing {{synchronized (pluginInitLock)}} section was
> already there. No new allocations on the hot path (task submission,
> scheduling), only at plugin construction / shutdown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
