mridulm commented on a change in pull request #28287:
URL: https://github.com/apache/spark/pull/28287#discussion_r451694092
##########
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##########
@@ -789,6 +811,23 @@ private[spark] class ExecutorAllocationManager(
}
}
+ override def onUnschedulableBlacklistTaskSubmitted
+ (blacklistedTask: SparkListenerUnschedulableBlacklistTaskSubmitted):
Unit = {
+ val stageId = blacklistedTask.stageId
+ val stageAttemptId = blacklistedTask.stageAttemptId
+ allocationManager.synchronized {
+ (stageId, stageAttemptId) match {
+ case (Some(stageId), Some(stageAttemptId)) =>
+ val stageAttempt = StageAttempt(stageId, stageAttemptId)
+ unschedulableTaskSets.add(stageAttempt)
+ case (None, _) =>
+ // Clear unschedulableTaskSets since atleast one task becomes
schedulable now
+ unschedulableTaskSets.clear()
Review comment:
There is a difference in both cases.
If all tasksets are removed from
`TaskSchedulerImpl.unschedulableTaskSetToExpiryTime`, it would defer aborting
unschedulable tasksets in the assumption that some progress is being made and
eventually tasksets might become scheduleable (or gets readded subsequently).
On other hand, if an unschedulable taskset remains in
`unschedulableTaskSets`, this can trigger a successful executor allocation
sooner - which could make it (and other taskset's) schedulable.
If we clear `unschedulableTaskSets` here, an eventual resourceOffer will
result in re-addition to `unschedulableTaskSets` after a delay. If we want to
defer resource acquisition, that sounds fine - even with this, current PR is
still a strict improvement over what we have.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
