mridulm commented on a change in pull request #28287:
URL: https://github.com/apache/spark/pull/28287#discussion_r451701402
##########
File path: core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
##########
@@ -829,12 +868,25 @@ private[spark] class ExecutorAllocationManager(
numTotalTasks - numRunning
}
+ def pendingUnschedulableTasksPerResourceProfile(rp: Int): Int = {
Review comment:
We should either base additional allocations on:
* Number of unschedulable tasks; we need to make sure it is not aggressive
overallocation.
* Do something similar to speculative tasks - so that there is atleast one
executor allocation to make progress; this could result in taking longer to
make progress.
Currently, we are using unschedulable tasksets as a proxy for number of
unschedulable tasks - which is a hybrid of both of these approaches.
Also note that given the discussion on clearing `unschedulableTaskSets`, we
could toggle between asking for more resources and asking for less (as soon as
a taskset becomes schedulable).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
