[
https://issues.apache.org/jira/browse/SPARK-54596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18042895#comment-18042895
]
Nan Zhu commented on SPARK-54596:
---------------------------------
SPIP coming soon
> Burst-aware memoryOverhead Allocation Algorithm for Spark@K8S
> -------------------------------------------------------------
>
> Key: SPARK-54596
> URL: https://issues.apache.org/jira/browse/SPARK-54596
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes, Spark Core
> Affects Versions: 4.2
> Reporter: Nan Zhu
> Priority: Major
>
> memoryOverhead is one of the most significant memory consumers for Spark
> workloads. Users tend to book a big chunk of such memory space to avoid
> triggering OOMKiller in the cluster environment.
> However, the usage of pattern of memoryOverhead space, such as se/deser,
> compress/decompress, direct memory, etc. , is usually bursty. As a result,
> users have to book the peak usage of memoryOverhead to ensure the reliability
> of their jobs while leaving a big chunk of this space unused in the majority
> of the job lifecycle.
> To resolve this problem in the context of Spark@K8S, we implemented the
> algorithm presented in the paper "Towards Resource Efficiency: Practical
> Insights into Large-Scale Spark Workloads at ByteDance" from ByteDance team
> ([https://www.vldb.org/pvldb/vol17/p3759-shi.pdf)] . We introduce a
> configurable parameter to split the user requested memoryOverhead space into
> two parts, guaranteed/shared. The guaranteed part is dedicated to a pod
> started by a spark job and shared part is used by multiple pods in a
> time-sharing fashion. Being specific to the K8S environment, the memory
> request of a pod will be user-requested on-heap space + guaranteed
> memoryOverhead and the memory limit will be user-requested on-heap space +
> guaranteed memoryOverhead + shared memoryOverhead .
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
