[
https://issues.apache.org/jira/browse/SPARK-60077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chenyu Zheng updated SPARK-60077:
---------------------------------
Description:
Spark Dynamic Allocation couples two responsibilities inside the driver process:
* ExecutorAllocationManager computes a desired executor request count based on
backlog and idle state
* ExecutorPodsAllocator creates executor pods directly against the Kubernetes
API.
This does not align with the Kubernetes declarative mode, where intent is
expressed in spec and outcomes are reflected in status. And it is impossible
for external admission systems (e.g. Kueue) to participate in executor scaling.
I think we need to introduce an ExecutorScaling CRD to decouple executor demand
computation from pod creation.
For more see: https://github.com/kubernetes-sigs/kueue/issues/12944
was:
Spark Dynamic Allocation on Kubernetes tightly couples executor demand
computation with pod creation inside the driver, making it impossible for
external admission systems (e.g. Kueue) to participate in executor scaling.
This issue proposes introducing an ExecutorScaling CRD as a generic, pluggable
admission interface.
For more see: https://github.com/kubernetes-sigs/kueue/issues/12944
> [K8S] Decouple executor demand computation from pod creation to enable
> external admission for Dynamic Allocation
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60077
> URL: https://issues.apache.org/jira/browse/SPARK-60077
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 5.0.0
> Reporter: Chenyu Zheng
> Priority: Major
>
> Spark Dynamic Allocation couples two responsibilities inside the driver
> process:
> * ExecutorAllocationManager computes a desired executor request count based
> on backlog and idle state
> * ExecutorPodsAllocator creates executor pods directly against the Kubernetes
> API.
> This does not align with the Kubernetes declarative mode, where intent is
> expressed in spec and outcomes are reflected in status. And it is impossible
> for external admission systems (e.g. Kueue) to participate in executor
> scaling.
> I think we need to introduce an ExecutorScaling CRD to decouple executor
> demand computation from pod creation.
> For more see: https://github.com/kubernetes-sigs/kueue/issues/12944
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
