attilapiros commented on a change in pull request #31445:
URL: https://github.com/apache/spark/pull/31445#discussion_r570299313
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
##########
@@ -223,14 +223,15 @@ private[spark] class ExecutorPodsAllocator(
if (knownPodCount > targetNum) {
val excess = knownPodCount - targetNum
+ val newlyCreatedToDelete = newlyCreatedExecutorsForRpId
+ .filter { case (_, (_, createTime)) =>
+ currentTime - createTime > executorIdleTimeout
+ }.keys.take(excess).toList
val knownPendingToDelete = currentPendingExecutors
.filter(x => isExecutorIdleTimedOut(x._2, currentTime))
+ .take(excess - newlyCreatedToDelete.size)
.map { case (id, _) => id }
- .take(excess - newlyCreatedExecutorsForRpId.size)
Review comment:
Before this PR this line caused the error in the new unit test.
As some `newlyCreatedExecutorsForRpId` might be not timed out and in place
of those we should choose timed out pending executor pod requests.
##########
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
##########
@@ -223,14 +223,15 @@ private[spark] class ExecutorPodsAllocator(
if (knownPodCount > targetNum) {
val excess = knownPodCount - targetNum
+ val newlyCreatedToDelete = newlyCreatedExecutorsForRpId
+ .filter { case (_, (_, createTime)) =>
+ currentTime - createTime > executorIdleTimeout
+ }.keys.take(excess).toList
val knownPendingToDelete = currentPendingExecutors
.filter(x => isExecutorIdleTimedOut(x._2, currentTime))
+ .take(excess - newlyCreatedToDelete.size)
.map { case (id, _) => id }
- .take(excess - newlyCreatedExecutorsForRpId.size)
Review comment:
Before this PR this line caused the error in the new unit test.
As some `newlyCreatedExecutorsForRpId` might be not timed out and in place
of those we should choose timed out pending executor pod requests.
So the `.size` must be called on the filtered `newlyCreatedExecutors` which
only contains the timed out elements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
