Suman Somasundar commented on SPARK-23529:

I have a couple of use-cases for this as explained below:
 # There are multiple Spark jobs running in the cluster and I would want to be 
able to extract the log files from these jobs. In this case, I can mount a 
hostpath volume in all the driver and executor pods and have them write logs to 
that path. I can login to the nodes and copy all the log files from all jobs.
 # My jobs have a dependency on large data files and fat jars. I dont want to 
copy them over (from HDFS) to the driver and executor pods every time I submit 
the job. Packaging these data files and jars in the image is not an option as 
the image size will be large and I dont want to have images specific to a job. 
In this case, I can download the dependencies to a local volume and mount it as 
hostpath on the driver and executor pods and use the dependencies as local 

> Specify hostpath volume and mount the volume in Spark driver and executor 
> pods in Kubernetes
> --------------------------------------------------------------------------------------------
>                 Key: SPARK-23529
>                 URL: https://issues.apache.org/jira/browse/SPARK-23529
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 2.3.0
>            Reporter: Suman Somasundar
>            Assignee: Anirudh Ramanathan
>            Priority: Minor

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to