colinmjj commented on a change in pull request #25759: [SPARK-19147][CORE]
Gracefully handle error in task after executor is stopped
URL: https://github.com/apache/spark/pull/25759#discussion_r325687758
##########
File path: core/src/main/scala/org/apache/spark/executor/Executor.scala
##########
@@ -604,6 +604,21 @@ private[spark] class Executor(
val serializedTK = ser.serialize(TaskKilled(killReason, accUpdates,
accums, metricPeaks))
execBackend.statusUpdate(taskId, TaskState.KILLED, serializedTK)
+ // When put the task in the pool, executor.stop may be called before
task.run.
+ // The exception will be thrown from the task becauseof the unexpected
status,
+ // see: SPARK-19147, here is to process the exception after
executor.stop
+ // as the excepted exception.
+ case t: Throwable if !isLocal && env.isStopped =>
Review comment:
@srowen @squito , thanks for the comments, I check the code again and make
clearly how metrics & heartbeat work. You're right, report metrics is
meaningless after executor.close(), because heartbeat won't work.
Update the pr and the exception will be processed in "case t: Throwable =>"
part with log only.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
