sunchao commented on code in PR #59756:
URL: https://github.com/apache/spark/pull/59756#discussion_r4402826461
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala:
##########
@@ -654,7 +810,31 @@ class ExecutorPodsAllocator(
}
}
+ override def stopAllocation(): Unit = synchronized {
+ if (!podAllocationParallelEnabled || stopped) return
+ stopped = true
+ podCreationThreadPool.foreach { pool =>
+ // Do not interrupt an in-flight create: Fabric8 stops waiting for the
response on interrupt,
+ // but the API server may still create the resource. Let workers observe
the result and clean
+ // up before the backend deletes PVCs and closes the client. Queued
tasks need not run.
+ pool.shutdown()
+ val queuedTasks = new ArrayList[Runnable]()
+ pool.getQueue.drainTo(queuedTasks)
Review Comment:
Confirmed fixed in a449f16fe9c3617f4176bdc14d60be09bf73f768. The second
shutdown signal wakes idle workers without interrupting active requests. I
rechecked the forced interleaving on JDK 17 and 21: the old sequence fails,
while the fixed sequence terminates in 500/500 iterations on each JDK. The new
Scala regression test exercises this interleaving and passes in the
current-head CI report.
##########
docs/running-on-kubernetes.md:
##########
@@ -1006,6 +1006,47 @@ See the [configuration page](configuration.html) for
information on Spark config
</td>
<td>4.2.0</td>
</tr>
+<tr>
+ <td><code>spark.kubernetes.allocation.batch.parallel.enabled</code></td>
+ <td><code>false</code></td>
+ <td>
+ Create executor pods concurrently within each allocation batch. Pod
specifications and PVC reuse
Review Comment:
Confirmed addressed in a449f16fe9c3617f4176bdc14d60be09bf73f768: all three
documentation rows and their Config.scala descriptions now state the
direct-allocator scope. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
