[
https://issues.apache.org/jira/browse/SPARK-55555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-55555:
----------------------------------
Description:
Historically, the Apache Spark manages executors in the well-known ways via (1)
Dynamic Allocation, (2) Resource profiles, (3) Executor Rolling. However, many
code patch still assumes homogeneous executors.
This issue aims to cover more advanced resource managements by taking advantage
of new K8s features. For example,
- We can resize CPU and Memory requests and limits of executors via In-Place
Pod Update.
- We can scale-up the PVC volumes via `AllowVolumeExpansion`.
- We can introduce a new kind of executor type, **Recovery-mode Executor**, to
allow only a single task per executor JVM.
was:
Historically, the Apache Spark manages executors in the well-known ways via (1)
Dynamic Allocation, (2) Resource profiles, (3) Executor Rolling. However, many
code patch still assumes homogeneous executors.
This issue aims to cover more advanced resource managements by taking advantage
of new K8s features. For example,
- We can resize CPU and Memory requests and limits of executors via In-Place
Pod Update.
- We can scale-up the PVC volumes via `AllowVolumeExpansion`.
> Support heterogeneous K8s executor management
> ---------------------------------------------
>
> Key: SPARK-55555
> URL: https://issues.apache.org/jira/browse/SPARK-55555
> Project: Spark
> Issue Type: Umbrella
> Components: Kubernetes
> Affects Versions: 4.2.0
> Reporter: Dongjoon Hyun
> Assignee: Dongjoon Hyun
> Priority: Critical
>
> Historically, the Apache Spark manages executors in the well-known ways via
> (1) Dynamic Allocation, (2) Resource profiles, (3) Executor Rolling. However,
> many code patch still assumes homogeneous executors.
> This issue aims to cover more advanced resource managements by taking
> advantage of new K8s features. For example,
> - We can resize CPU and Memory requests and limits of executors via In-Place
> Pod Update.
> - We can scale-up the PVC volumes via `AllowVolumeExpansion`.
> - We can introduce a new kind of executor type, **Recovery-mode Executor**,
> to allow only a single task per executor JVM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
