holdenk commented on a change in pull request #29817:
URL: https://github.com/apache/spark/pull/29817#discussion_r494434701
##########
File path:
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
##########
@@ -166,17 +171,6 @@ private[spark] class CoarseGrainedExecutorBackend(
if (executor == null) {
exitExecutor(1, "Received LaunchTask command but executor was null")
} else {
- if (decommissioned) {
- val msg = "Asked to launch a task while decommissioned."
- logError(msg)
- driver match {
- case Some(endpoint) =>
- logInfo("Sending DecommissionExecutor to driver.")
- endpoint.send(DecommissionExecutor(executorId,
ExecutorDecommissionInfo(msg)))
- case _ =>
- logError("No registered driver to send Decommission to.")
- }
- }
Review comment:
Right, so we should resend the notice then right?
##########
File path:
core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
##########
@@ -213,9 +207,17 @@ private[spark] class CoarseGrainedExecutorBackend(
logInfo(s"Received tokens of ${tokenBytes.length} bytes")
SparkHadoopUtil.get.addDelegationTokens(tokenBytes, env.conf)
- case DecommissionSelf =>
- logInfo("Received decommission self")
+ case DecommissionExecutor =>
decommissionSelf()
+
+ case ExecutorSigPWRReceived =>
+ decommissionSelf()
+ if (driver.nonEmpty) {
Review comment:
So we don’t ask the driver to stop scheduling jobs on us first, and the
driver could ask us to run a job while we are part way through decommissioning.
This won’t result in a failure because well accept the job but it will slow
down the decommissioning. So swap the order of these two.
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
##########
@@ -1809,7 +1809,9 @@ private[spark] class BlockManager(
blocksToRemove.size
}
- def decommissionBlockManager(): Unit = synchronized {
+ def decommissionBlockManager(): Unit =
storageEndpoint.ask(DecommissionBlockManager)
Review comment:
Why did you make this change?
##########
File path: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala
##########
@@ -70,7 +70,10 @@ private[deploy] class Worker(
if (conf.get(config.DECOMMISSION_ENABLED)) {
logInfo("Registering SIGPWR handler to trigger decommissioning.")
SignalUtils.register("PWR", "Failed to register SIGPWR handler - " +
- "disabling worker decommission feature.")(decommissionSelf)
+ "disabling worker decommission feature.") {
+ self.send(WorkerSigPWRReceived)
Review comment:
Can you look into what the difference of this behavior might cause at
the system level and then tell me if that’s a desired change? I’m ok with us
making changes here, I just want us to be intentional and know if we need to
test the change and it seems like this change was incidental.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
