Ivan Sadikov created SPARK-60950:
------------------------------------
Summary: MapOutputTrackerMaster never replies to a map output
status request whose handler throws
Key: SPARK-60950
URL: https://issues.apache.org/jira/browse/SPARK-60950
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Ivan Sadikov
MapOutputTrackerMasterEndpoint doesn't answer GetMapOutputStatuses,
GetMapAndMergeResultStatuses or GetShufflePushMergerLocations itself. It posts
each request, with its RpcCallContext, to a queue served by
MapOutputTrackerMaster.MessageLoop threads. The RPC Inbox treats the request as
handled once it's queued, so from then on only MessageLoop can reply.
If handling the message throws a non-fatal exception, MessageLoop logs it and
moves on:
{code}
try {
val data = mapOutputTrackerMasterMessages.take()
...
data match \{ ... }
} catch {
case NonFatal(e) => logError(log"${MDC(ERROR, e.getMessage)}", e)
}
{code}
Nothing replies to or fails the RpcCallContext, and data isn't in scope in the
catch, so it couldn't. The caller, normally MapOutputTrackerWorker on an
executor, blocks in askSync until spark.rpc.askTimeout runs out (120s by
default, from spark.network.timeout). Then it fails with a generic
RpcTimeoutException instead of the real error. If the task is retried and hits
the same exception, every attempt waits out the full timeout, and the job can
stall for a long time with no useful error on the executor side.
One easy trigger is a request for a shuffle that was unregistered after the
request was posted but before MessageLoop handled it. handleStatusMessage calls
shuffleStatuses.get(shuffleId).head, which throws NoSuchElementException for a
missing shuffle, and that request never gets a reply. Errors while serializing
or sending the reply hit the same path.
How to reproduce / test:
- Post a GetMapOutputMessage for an unregistered shuffle with a mocked
RpcCallContext. Today neither reply nor sendFailure is called; with the fix,
sendFailure gets the NoSuchElementException.
- Make reply throw. sendFailure should then be called with that exception.
- Make sendFailure throw too, with
spark.shuffle.mapOutput.dispatcher.numThreads=1. A later request should still
get a normal reply.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
