[
https://issues.apache.org/jira/browse/SPARK-20741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-20741:
------------------------------------
Assignee: (was: Apache Spark)
> SparkSubmit does not clean up after uploading spark_libs to the distributed
> cache
> ---------------------------------------------------------------------------------
>
> Key: SPARK-20741
> URL: https://issues.apache.org/jira/browse/SPARK-20741
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 2.1.1
> Reporter: Lior Regev
> Priority: Minor
>
> Running spark submit has to distribute Spark's JARs to a distributed cache in
> order for the executors to access it.
> When neither spark.yarn.jars or spark.yarn.archive is provided, SparkSubmit
> creates a ZIP of all the JARs in $SPARK_HOME/jars and uploads it to the
> distributed cache.
> After uploading the ZIP file, SparkSubmit does not delete the local copy of
> it. This, in turn can cause the disk on the local machine to fill up (200MB
> at a time) until no more submissions are possible.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
