Wenjun Ruan created SPARK-55257:
-----------------------------------
Summary: Is it possible to allocate executor pod in parallel
Key: SPARK-55257
URL: https://issues.apache.org/jira/browse/SPARK-55257
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 4.1.1
Reporter: Wenjun Ruan
Currently, the driver allocates executor pods one by one.
When a large number of executors need to be allocated, this step can become a
bottleneck.
Since the allocation has already been split at this stage, would it be possible
to perform the pod creation in parallel?
```
ExecutorPodsAllocator.splitSlots(podsToAllocateWithRpId,
remainingSlotFromPendingPods)
.foreach { case ((rpId, podCountForRpId, targetNum, pendingPodCountForRpId),
sharedSlotFromPendingPods) =>
val remainingSlotsForRpId = maxPendingPodsPerRpid - pendingPodCountForRpId
val numMissingPodsForRpId = targetNum - podCountForRpId
val numExecutorsToAllocate = Seq(numMissingPodsForRpId, podAllocationSize,
sharedSlotFromPendingPods, remainingSlotsForRpId).min
logInfo(log"Going to request ${MDC(LogKeys.COUNT, numExecutorsToAllocate)}
executors from" +
log" Kubernetes for ResourceProfile Id: ${MDC(LogKeys.RESOURCE_PROFILE_ID,
rpId)}, " +
log"target: ${MDC(LogKeys.NUM_POD_TARGET, targetNum)}, " +
log"known: ${MDC(LogKeys.NUM_POD, podCountForRpId)}, sharedSlotFromPendingPods:
" +
log"${MDC(LogKeys.NUM_POD_SHARED_SLOT, sharedSlotFromPendingPods)}.")
requestNewExecutors(numExecutorsToAllocate, applicationId, rpId,
k8sKnownPVCNames)
}
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
