[
https://issues.apache.org/jira/browse/HIVE-29780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-29780:
--------------------------------
Description:
{{LlapTaskCommunicator}} and {{LlapTaskSchedulerService}} pair up at
construction using a single class-static slot ({{pluginInitLock}} + {{static …
instance}}). When two DAGs are constructed concurrently in the same JVM, the
second-to-arrive plugin from DAG-A can accidentally pair with the plugin from
DAG-B. DAG-A's real plugin is then left with {{scheduler == null}} and the
first task submission fails with:
{code}
6-07-29T06:48:51,723 INFO [pool-163-thread-1] tezplugins.LlapTaskCommunicator:
Failed to run task: attempt_1785332929111_0001_1_00_000000_0 on containerId:
container_111101111_0001_00_000003
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
because "this.this$0.scheduler" is null
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:540)
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:517)
at
org.apache.hadoop.hive.llap.AsyncResponseHandler$AsyncResponseHandlerRunnable.run(AsyncResponseHandler.java:90)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
{code}
A production Tez {{DAGAppMaster}} runs one DAG at a time per AM, so the two
LLAP plugins for that DAG are never in a race with plugins for a different DAG
in the same JVM. The issue only surfaces in JVMs that host many concurrent DAGs
— MiniHS2 / MiniLlapCluster / anything that shares an AM classloader across
DAGs. Repro seen locally with a 30-way concurrent {{INSERT}} against a MiniHS2
LLAP cluster: 5 of 30 sessions failed with the NPE.
please find: [^hive_LLAP_NPE.log.zip]
*Production impact:*
None — the map contains at most *one* entry at any time in a production AM (one
DAG per AM, and the entry is removed the moment the second plugin arrives, or
on {{shutdown()}}). A {{ConcurrentHashMap.put}}/{{remove}} pair per DAG init is
nanoseconds; the existing {{synchronized (pluginInitLock)}} section was already
there. No new allocations on the hot path (task submission, scheduling), only
at plugin construction / shutdown.
was:
{{LlapTaskCommunicator}} and {{LlapTaskSchedulerService}} pair up at
construction using a single class-static slot ({{pluginInitLock}} + {{static …
instance}}). When two DAGs are constructed concurrently in the same JVM, the
second-to-arrive plugin from DAG-A can accidentally pair with the plugin from
DAG-B. DAG-A's real plugin is then left with {{scheduler == null}} and the
first task submission fails with:
{code}
6-07-29T06:48:51,723 INFO [pool-163-thread-1] tezplugins.LlapTaskCommunicator:
Failed to run task: attempt_1785332929111_0001_1_00_000000_0 on containerId:
container_111101111_0001_00_000003
java.lang.NullPointerException: Cannot invoke
"org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
because "this.this$0.scheduler" is null
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:540)
at
org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:517)
at
org.apache.hadoop.hive.llap.AsyncResponseHandler$AsyncResponseHandlerRunnable.run(AsyncResponseHandler.java:90)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
{code}
A production Tez {{DAGAppMaster}} runs one DAG at a time per AM, so the two
LLAP plugins for that DAG are never in a race with plugins for a different DAG
in the same JVM. The issue only surfaces in JVMs that host many concurrent DAGs
— MiniHS2 / MiniLlapCluster / anything that shares an AM classloader across
DAGs. Repro seen locally with a 30-way concurrent {{INSERT}} against a MiniHS2
LLAP cluster: 5 of 30 sessions failed with the NPE.
please find: [^hive_LLAP_NPE.log.zip]
None — the map contains at most *one* entry at any time in a production AM (one
DAG per AM, and the entry is removed the moment the second plugin arrives, or
on {{shutdown()}}). A {{ConcurrentHashMap.put}}/{{remove}} pair per DAG init is
nanoseconds; the existing {{synchronized (pluginInitLock)}} section was already
there. No new allocations on the hot path (task submission, scheduling), only
at plugin construction / shutdown.
> Fix LLAP plugin rendezvous race under concurrent DAGs (NullPointerException
> in LlapTaskCommunicator.setResponse)
> ----------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29780
> URL: https://issues.apache.org/jira/browse/HIVE-29780
> Project: Hive
> Issue Type: Bug
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Attachments: hive_LLAP_NPE.log.zip
>
>
> {{LlapTaskCommunicator}} and {{LlapTaskSchedulerService}} pair up at
> construction using a single class-static slot ({{pluginInitLock}} + {{static
> … instance}}). When two DAGs are constructed concurrently in the same JVM,
> the second-to-arrive plugin from DAG-A can accidentally pair with the plugin
> from DAG-B. DAG-A's real plugin is then left with {{scheduler == null}} and
> the first task submission fails with:
> {code}
> 6-07-29T06:48:51,723 INFO [pool-163-thread-1]
> tezplugins.LlapTaskCommunicator: Failed to run task:
> attempt_1785332929111_0001_1_00_000000_0 on containerId:
> container_111101111_0001_00_000003
> java.lang.NullPointerException: Cannot invoke
> "org.apache.hadoop.hive.llap.tezplugins.LlapTaskSchedulerService.notifyStarted(org.apache.tez.dag.records.TezTaskAttemptID)"
> because "this.this$0.scheduler" is null
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:540)
> at
> org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:517)
> at
> org.apache.hadoop.hive.llap.AsyncResponseHandler$AsyncResponseHandlerRunnable.run(AsyncResponseHandler.java:90)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
> at java.base/java.lang.Thread.run(Thread.java:1583)
> {code}
> A production Tez {{DAGAppMaster}} runs one DAG at a time per AM, so the two
> LLAP plugins for that DAG are never in a race with plugins for a different
> DAG in the same JVM. The issue only surfaces in JVMs that host many
> concurrent DAGs — MiniHS2 / MiniLlapCluster / anything that shares an AM
> classloader across DAGs. Repro seen locally with a 30-way concurrent
> {{INSERT}} against a MiniHS2 LLAP cluster: 5 of 30 sessions failed with the
> NPE.
> please find: [^hive_LLAP_NPE.log.zip]
> *Production impact:*
> None — the map contains at most *one* entry at any time in a production AM
> (one DAG per AM, and the entry is removed the moment the second plugin
> arrives, or on {{shutdown()}}). A {{ConcurrentHashMap.put}}/{{remove}} pair
> per DAG init is nanoseconds; the existing {{synchronized (pluginInitLock)}}
> section was already there. No new allocations on the hot path (task
> submission, scheduling), only at plugin construction / shutdown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
