[
https://issues.apache.org/jira/browse/SPARK-55257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18054978#comment-18054978
]
Dongjoon Hyun edited comment on SPARK-55257 at 1/28/26 11:26 PM:
-----------------------------------------------------------------
Apache Spark supports `spark.kubernetes.allocation.pods.allocator`
configuration to support direct executor pod allocation and K8s-native
`StatefulSet` and `Deployment` for advanced executor pod allocation. Please
refer the K8s documentation and the following JIRA issues for the technical
details.
- SPARK-36058
- SPARK-54173
https://spark.apache.org/docs/4.2.0-preview1/running-on-kubernetes.html
{quote}Allocator to use for pods. Possible values are direct (the default),
statefulset, deployment, or a full class name of a class implementing
`AbstractPodsAllocator`. Future version may add Job or replicaset. This is a
developer API and may change or be removed at anytime.{quote}
was (Author: dongjoon):
Apache Spark supports `spark.kubernetes.allocation.pods.allocator`
configuration to support direct executor pod allocation and K8s-native
`StatefulSet` and `Deployment` for advanced executor pod allocation. Please
refer the K8s documentation and the following JIRA issues for the technical
details.
- SPARK-36058
- SPARK-54173
> Is it possible to allocate executor pod in parallel
> ---------------------------------------------------
>
> Key: SPARK-55257
> URL: https://issues.apache.org/jira/browse/SPARK-55257
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 4.1.1
> Reporter: Wenjun Ruan
> Priority: Major
>
> Currently, the driver allocates executor pods one by one.
> When a large number of executors need to be allocated, this step can become a
> bottleneck.
> Since the allocation has already been split at this stage, would it be
> possible to perform the pod creation in parallel?
>
> ```
> ExecutorPodsAllocator.splitSlots(podsToAllocateWithRpId,
> remainingSlotFromPendingPods)
> .foreach { case ((rpId, podCountForRpId, targetNum, pendingPodCountForRpId),
> sharedSlotFromPendingPods) =>
> val remainingSlotsForRpId = maxPendingPodsPerRpid - pendingPodCountForRpId
> val numMissingPodsForRpId = targetNum - podCountForRpId
> val numExecutorsToAllocate = Seq(numMissingPodsForRpId, podAllocationSize,
> sharedSlotFromPendingPods, remainingSlotsForRpId).min
> logInfo(log"Going to request ${MDC(LogKeys.COUNT, numExecutorsToAllocate)}
> executors from" +
> log" Kubernetes for ResourceProfile Id: ${MDC(LogKeys.RESOURCE_PROFILE_ID,
> rpId)}, " +
> log"target: ${MDC(LogKeys.NUM_POD_TARGET, targetNum)}, " +
> log"known: ${MDC(LogKeys.NUM_POD, podCountForRpId)},
> sharedSlotFromPendingPods: " +
> log"${MDC(LogKeys.NUM_POD_SHARED_SLOT, sharedSlotFromPendingPods)}.")
> requestNewExecutors(numExecutorsToAllocate, applicationId, rpId,
> k8sKnownPVCNames)
> }
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
