Ngone51 commented on a change in pull request #26980: [SPARK-27348][Core]
HeartbeatReceiver should remove lost executors from
CoarseGrainedSchedulerBackend
URL: https://github.com/apache/spark/pull/26980#discussion_r360796114
##########
File path: core/src/main/scala/org/apache/spark/HeartbeatReceiver.scala
##########
@@ -199,14 +201,20 @@ private[spark] class HeartbeatReceiver(sc: SparkContext,
clock: Clock)
if (now - lastSeenMs > executorTimeoutMs) {
logWarning(s"Removing executor $executorId with no recent heartbeats:
" +
s"${now - lastSeenMs} ms exceeds timeout $executorTimeoutMs ms")
- scheduler.executorLost(executorId, SlaveLost("Executor heartbeat " +
- s"timed out after ${now - lastSeenMs} ms"))
- // Asynchronously kill the executor to avoid blocking the current
thread
+ // Asynchronously kill the executor to avoid blocking the current
thread
killExecutorThread.submit(new Runnable {
override def run(): Unit = Utils.tryLogNonFatalError {
// Note: we want to get an executor back after expiring this one,
// so do not simply call `sc.killExecutor` here (SPARK-8119)
sc.killAndReplaceExecutor(executorId)
+ // In case of the executors which are not gracefully shut down, we
should remove
Review comment:
No, no.
Please pay attention to the comment above and note that, here, we're not
saying that this issue happens due to `CoarseGrainedSchedulerBackend` may shut
down executors non-gracefully. But we say that it may happens due to the
executor itself is not gracefully shut down.
It's possible that `CoarseGrainedSchedulerBackend` asks someone executor to
shutdown and the executor may shutdown non-gracefully. And it's also possible
that the host where the executor located suddenly crushed that makes the
executor shutdown non-gracefully.
Shutdown non-gracefully here only means that an executor shutdown but
`CoarseGrainedSchedulerBackend` doesn't receive disconnect event from it. It
really doesn't matter what causes the executor shutdown non-gracefully.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
