AngersZhuuuu commented on code in PR #52793:
URL: https://github.com/apache/spark/pull/52793#discussion_r2567127456
##########
core/src/main/scala/org/apache/spark/scheduler/TaskScheduler.scala:
##########
@@ -121,4 +121,8 @@ private[spark] trait TaskScheduler {
*/
def applicationAttemptId(): Option[String]
+ /**
+ * Mark executor is pending to remove.
Review Comment:
> ah now I recall. I think we already have this feature: don't launch tasks
on a to-be-decommissioned executor. @AngersZhuuuu can we leverage the existing
framework?
When `CoarseGrainedSchedulerBackend` call `makeOffers` will check executor
active
```
override def isExecutorActive(id: String): Boolean = synchronized {
executorDataMap.contains(id) &&
!executorsPendingToRemove.contains(id) &&
!executorsPendingLossReason.contains(id) &&
!executorsPendingDecommission.contains(id)
}
```
after `makeOffer` call `TaskSchedulerImpl` `resourceOffers`,
when `launchedAnyTask` is false, we will check whether the given task set
has been excluded to the point that it can't run anywhere. then call
`getCompletelyExcludedTaskIfAny`
Here have 2 questions:
1. We can't use `hostToExecutors` when call
`taskSet.getCompletelyExcludedTaskIfAny()`, since if use `hostToExecutors`,
will won't find out excluded tasks since `pendingToRemoveExecutors` not
excluded in `hostToExecutors`. Then can't update TaskSet to
`UnschedulableTaskSet`
2. If we filter executors on `hostToExecutors` when call
`taskSet.getCompletelyExcludedTaskIfAny()` , it's cost too heavy.
So we need to maintain `availableHostsToExecutors`.
I don't quite understand what you mean by "leveraging the existing
framework."
Are you referring to reusing data structures? It's not convenient to
directly call the contents of CoarseGrainedSchedulerBackend in
TaskSchedulerImpl.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
