Ngone51 commented on a change in pull request #29579:
URL: https://github.com/apache/spark/pull/29579#discussion_r479876219
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -922,28 +910,8 @@ private[spark] class TaskSchedulerImpl(
synchronized {
// Don't bother noting decommissioning for executors that we don't know
about
if (executorIdToHost.contains(executorId)) {
- val oldDecomStateOpt = executorsPendingDecommission.get(executorId)
- val newDecomState = if (oldDecomStateOpt.isEmpty) {
- // This is the first time we are hearing of decommissioning this
executor,
- // so create a brand new state.
- ExecutorDecommissionState(
- clock.getTimeMillis(),
- decommissionInfo.isHostDecommissioned)
- } else {
- val oldDecomState = oldDecomStateOpt.get
- if (!oldDecomState.isHostDecommissioned &&
decommissionInfo.isHostDecommissioned) {
- // Only the cluster manager is allowed to send decommission
messages with
- // isHostDecommissioned set. So the new decommissionInfo is from
the cluster
- // manager and is thus authoritative. Flip isHostDecommissioned to
true but keep the old
- // decommission start time.
- ExecutorDecommissionState(
- oldDecomState.startTime,
- isHostDecommissioned = true)
- } else {
- oldDecomState
- }
- }
- executorsPendingDecommission(executorId) = newDecomState
+ executorsPendingDecommission(executorId) =
+ ExecutorDecommissionState(clock.getTimeMillis(),
decommissionInfo.hostOpt)
Review comment:
The multiple decommission messages for the same executor shouldn't
happen as it has been guaranteed by
`CoarseGrainedSchedulerBackend.decommissionExecutors`:
https://github.com/apache/spark/blob/a1e459ed9f6777fb8d5a2d09fda666402f9230b9/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L465-L475
we won't decommission an executor which is already decommissioned(in-active).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
