[
https://issues.apache.org/jira/browse/SPARK-60077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124884#comment-18124884
]
Chenyu Zheng commented on SPARK-60077:
--------------------------------------
[~dongjoon] I noticed that https://github.com/apache/spark-kubernetes-operator
also doesn't directly manage executor pods. Do you think we should expose an
`ExecutorScaling` CRD to achieve decoupling? Could you give me some suggesstion?
> [K8S] Decouple executor demand computation from pod creation to enable
> external admission for Dynamic Allocation
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60077
> URL: https://issues.apache.org/jira/browse/SPARK-60077
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 5.0.0
> Reporter: Chenyu Zheng
> Priority: Major
>
> Spark Dynamic Allocation couples two responsibilities inside the driver
> process:
> * ExecutorAllocationManager computes a desired executor request count based
> on backlog and idle state
> * ExecutorPodsAllocator creates executor pods directly against the Kubernetes
> API.
> This does not align with the Kubernetes declarative mode, where intent is
> expressed in spec and outcomes are reflected in status. And it is impossible
> for external admission systems (e.g. Kueue) to participate in executor
> scaling.
> I think we need to introduce an ExecutorScaling CRD to decouple executor
> demand computation from pod creation.
> For more see: https://github.com/kubernetes-sigs/kueue/issues/12944
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
