GitHub user devaraj-kavali opened a pull request:
https://github.com/apache/spark/pull/19141
[SPARK-21384] [YARN] Spark 2.2 + YARN without spark.yarn.jars /
spark.yarn.archive fails
## What changes were proposed in this pull request?
When the libraries temp directory(i.e. __spark_libs__*.zip dir) file system
and staging dir(destination) file systems are the same then the
__spark_libs__*.zip is not copying to the staging directory. But after making
this decision the libraries zip file is getting deleted immediately and
becoming unavailable for the Node Manager's localization.
This change removes the deletion of the libraries zip file immediately and
allowing it to delete as part of the ShutdownHookManager deletion of paths.
## How was this patch tested?
I have verified it manually in yarn/cluster and yarn/client modes with hdfs
and local file systems.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/devaraj-kavali/spark SPARK-21384
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19141.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19141
----
commit 208bb685cc899b705aadb7c5aba51334f2d340f0
Author: Devaraj K <deva...@apache.org>
Date: 2017-09-06T00:22:54Z
[SPARK-21384] [YARN] Spark 2.2 + YARN without spark.yarn.jars /
spark.yarn.archive fails
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
