attilapiros commented on a change in pull request #33492:
URL: https://github.com/apache/spark/pull/33492#discussion_r675425508
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
##########
@@ -217,9 +219,15 @@ private[spark] class ExecutorPodsAllocator(
}
}
+ // sum of all the pending pods unknown by the scheduler (total for all the
resources)
var totalPendingCount = 0
- // The order we request executors for each ResourceProfile is not
guaranteed.
- totalExpectedExecutorsPerResourceProfileId.asScala.foreach { case (rpId,
targetNum) =>
+ // total not running pods (including scheduler known & unknown, pending &
newly requested ones)
+ var totalNotRunningPodCount = 0
+ val podsToAllocateWithRpId = totalExpectedExecutorsPerResourceProfileId
+ .asScala
+ .toSeq
+ .sortBy(_._1)
+ .flatMap { case (rpId, targetNum) =>
Review comment:
Instead of `foreach` here is `flatMap` as we need to do the process in
two steps for counting all the not running PODs for all the resource profiles
before we decide how to split the remaining pending PODs slot between the
resource profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
