abstractdog opened a new pull request, #6659:
URL: https://github.com/apache/hive/pull/6659
### What changes were proposed in this pull request?
Replace the class-static single-slot rendezvous between
`LlapTaskCommunicator` and `LlapTaskSchedulerService` with a
per-`ApplicationAttemptId` handshake. Each side now parks itself in a
`ConcurrentMap<ApplicationAttemptId, ThisSide>` keyed by the appAttemptId both
plugins for the same DAG share (via
`TaskCommunicatorContext.getApplicationAttemptId()` /
`TaskSchedulerContext.getApplicationAttemptId()`). The existing
`pluginInitLock` stays so the pair-or-park section is atomic per DAG.
`shutdown()` on each side reaps its own leftover entry with
`ConcurrentMap.remove(key, this)` so a failed DAG init doesn't leak.
### Why are the changes needed?
The single-slot handshake pairs plugins across DAGs when multiple DAGs come
up concurrently in the same JVM: the communicator from DAG-A ends up bound to
the scheduler from DAG-B, DAG-A's real communicator is left with `scheduler ==
null`, and the first task submission hits
```
java.lang.NullPointerException: Cannot invoke
"LlapTaskSchedulerService.notifyStarted(TezTaskAttemptID)"
because "this.this$0.scheduler" is null
at LlapTaskCommunicator$3.setResponse(LlapTaskCommunicator.java:540)
```
which surfaces as `Vertex Map 1 killed/failed due to:OWN_TASK_FAILURE` →
`MoveTask return code 2`. In a 30-way concurrent-INSERT reproduction on MiniHS2
(LLAP) 5 of 30 sessions failed with this NPE.
### Does this PR introduce _any_ user-facing change?
No. A production Tez `DAGAppMaster` runs one DAG at a time so the race is
not reachable there, and the map holds at most one entry at any moment; the hot
paths (task submission, scheduling) are unchanged.
### How was this patch tested?
30-way concurrent `INSERT` burst on MiniHS2 (`clusterType=llap`,
`-DminiHS2.isMetastoreRemote=true`) against an insert-only managed partitioned
table. Before: 25/30 sessions succeeded, 5 hit the NPE and `hive.log` had 120
stack traces from Tez task-attempt retries. After: 30/30 sessions succeeded,
zero `notifyStarted` NPEs in `hive.log`, all 30 writers landed in their own
`delta_<writeId>_<writeId>_0000` subdirectory as expected for insert-only ACID.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
