Dongjoon Hyun created SPARK-28042:
-------------------------------------
Summary: Support mapping spark.local.dir to hostPath volume
Key: SPARK-28042
URL: https://issues.apache.org/jira/browse/SPARK-28042
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 3.0.0
Reporter: Junjie Chen
Currently, the k8s executor builder mount spark.local.dir as emptyDir or
memory, it should satisfy some small workload, while in some heavily workload
like TPCDS, both of them can have some problem, such as pods are evicted due to
disk pressure when using emptyDir, and OOM when using tmpfs.
In particular on cloud environment, users may allocate cluster with minimum
configuration and add cloud storage when running workload. In this case, we can
specify multiple elastic storage as spark.local.dir to accelerate the spilling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
