L. C. Hsieh created SPARK-60222:
-----------------------------------
Summary: Keep task completion listeners on the completing thread
when another thread invokes listeners
Key: SPARK-60222
URL: https://issues.apache.org/jira/browse/SPARK-60222
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
TaskContextImpl runs task listeners on one thread at a time, tracked by
listenerInvocationThread. In two cases, the completion listeners do not run on
the thread that completes the task:
1. When another thread is invoking listeners as the task completes,
markTaskCompleted returns without running any completion listener. For example,
Task.kill calls markInterrupted on the kill thread, and if an interrupt
listener throws, markTaskFailed runs the failure listeners on that thread. The
completion listeners are then left until some other thread adds a listener, or
skipped entirely. The task can then reach cleanUpAllAllocatedMemory while
resources that the completion listeners would release are still in use. The
ErrorNotifier doc and a StreamingShuffleSuite test describe this race.
2. When a listener throws, invokeListeners clears listenerInvocationThread
while markTaskFailed runs the failure listeners, and reclaims it afterwards.
Another thread that adds a listener in between can take over running the
pending listeners, possibly concurrently with this thread.
Failure listeners that another thread queues while a thread runs the completion
listeners, and completion listeners queued while a thread runs failure
listeners after completion, are also dropped.
This matters for helper threads that the completion listeners stop and wait
for, such as the pipelined Python UDF writer (SPARK-60113): if the helper
thread adds a listener lazily at such a moment, it ends up running the listener
that waits for itself.
The proposed fix keeps the completion listeners on the completing thread:
markTaskCompleted waits for another thread invoking listeners and takes the
invocation over directly, failure listeners run inside the current invocation
when a listener throws, and a thread runs the listeners left to it before it
releases the invocation. This was first done in apache/spark#59332 and split
out of it during review.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
