[
https://issues.apache.org/jira/browse/SPARK-60950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60950:
-----------------------------------
Labels: pull-request-available (was: )
> MapOutputTrackerMaster never replies to a map output status request whose
> handler throws
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-60950
> URL: https://issues.apache.org/jira/browse/SPARK-60950
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Ivan Sadikov
> Priority: Major
> Labels: pull-request-available
>
> MapOutputTrackerMasterEndpoint doesn't answer GetMapOutputStatuses,
> GetMapAndMergeResultStatuses or GetShufflePushMergerLocations itself. It
> posts each request, with its RpcCallContext, to a queue served by
> MapOutputTrackerMaster.MessageLoop threads. The RPC Inbox treats the request
> as handled once it's queued, so from then on only MessageLoop can reply.
> If handling the message throws a non-fatal exception, MessageLoop logs it and
> moves on:
> {code}
> try {
> val data = mapOutputTrackerMasterMessages.take()
> ...
> data match \{ ... }
> } catch {
> case NonFatal(e) => logError(log"${MDC(ERROR, e.getMessage)}", e)
> }
> {code}
> Nothing replies to or fails the RpcCallContext, and data isn't in scope in
> the catch, so it couldn't. The caller, normally MapOutputTrackerWorker on an
> executor, blocks in askSync until spark.rpc.askTimeout runs out (120s by
> default, from spark.network.timeout). Then it fails with a generic
> RpcTimeoutException instead of the real error. If the task is retried and
> hits the same exception, every attempt waits out the full timeout, and the
> job can stall for a long time with no useful error on the executor side.
> One easy trigger is a request for a shuffle that was unregistered after the
> request was posted but before MessageLoop handled it. handleStatusMessage
> calls shuffleStatuses.get(shuffleId).head, which throws
> NoSuchElementException for a missing shuffle, and that request never gets a
> reply. Errors while serializing or sending the reply hit the same path.
> How to reproduce / test:
> - Post a GetMapOutputMessage for an unregistered shuffle with a mocked
> RpcCallContext. Today neither reply nor sendFailure is called; with the fix,
> sendFailure gets the NoSuchElementException.
> - Make reply throw. sendFailure should then be called with that exception.
> - Make sendFailure throw too, with
> spark.shuffle.mapOutput.dispatcher.numThreads=1. A later request should still
> get a normal reply.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
