Ngone51 commented on a change in pull request #29468:
URL: https://github.com/apache/spark/pull/29468#discussion_r472651414
##########
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##########
@@ -1062,25 +1062,36 @@ private[spark] class TaskSchedulerImpl(
}
def getExecutorsAliveOnHost(host: String): Option[Set[String]] =
synchronized {
- hostToExecutors.get(host).map(_.toSet)
+
hostToExecutors.get(host).map(_.filterNot(isExecutorDecommissioned)).map(_.toSet)
Review comment:
hmm..I think this more like a problem of definition or something like
that. In this PR, what I'm trying to do is to strengthen the definition of the
active status for the executor or host. That is, for an active executor or
host, it should not be decommissioned at the same time. This function is
getting the active host, so we should filter out decommissioned ones. I think
this's quite obvious. And this function is used by multiple tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
