Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/17543#discussion_r110011422
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1281,10 +1281,24 @@ class DAGScheduler(
val failedStage = stageIdToStage(task.stageId)
val mapStage = shuffleIdToMapStage(shuffleId)
+ def invalidateLostFilesAndExecutor(): Unit = {
+ // Mark the map whose fetch failed as broken in the map stage
+ if (mapId != -1) {
+ mapStage.removeOutputLoc(mapId, bmAddress)
+ mapOutputTracker.unregisterMapOutput(shuffleId, mapId,
bmAddress)
+ }
+
+ // TODO: mark the executor as failed only if there were lots of
fetch failures on it
+ if (bmAddress != null) {
+ handleExecutorLost(bmAddress.executorId, filesLost = true,
Some(task.epoch))
--- End diff --
Good question. Let me explain my setup and what happened in more detail:
Environment:
Each instance (machine) has 1 worker, and 4 executors.
Events:
1. A ShuffleMapStage was performed.
2. The instance with the shuffle file output was removed
3. First the Master said that the executor was lost, and not the worker.
So, the shuffle output was not removed from the mapOutputTracker
(`handleExecutorLost` was called with `filesLost = false`)
4. Later, the master tried launching more executors on the worker
5. All of them eventually failed, and the master said that the worker is
now dead.
6. Once the worker died, the StandaloneSchedulerBackend said that the
worker is dead, but used the latest attempted executorId.
7. This executor never had any shuffle outputs, therefore the metadata for
the old executor never got cleaned up, and was used in further retries of the
stage.
8. Job eventually died due to FetchFailedExceptions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
