GitHub user kiszk opened a pull request:
https://github.com/apache/spark/pull/21251
[SPARK-10878][core] Fix race condition when multiple clients resolves
artifacts at the same time
## What changes were proposed in this pull request?
When multiple clients attempt to resolve artifacts via the `--packages`
parameter, they could run into race condition when they each attempt to modify
the dummy `org.apache.spark-spark-submit-parent-default.xml` file created in
the default ivy cache dir.
This PR changes the behavior to encode UUID in the dummy module descriptor
so each client will operate on a different resolution file in the ivy cache
dir. In addition, this patch changes the behavior of when and which resolution
files are cleaned to prevent accumulation of resolution files in the default
ivy cache dir.
Since this PR is a successor of #18801, close #18801. Many codes were
ported from #11494. **Many efforts were put here. I think this PR should credit
to @Victsm .**
## How was this patch tested?
added UT into `SparkSubmitUtilsSuite`
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kiszk/spark SPARK-10878
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21251.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21251
----
commit 949ec1db20a99443dcc596f6e348170cb96f0124
Author: Kazuaki Ishizaki <ishizaki@...>
Date: 2018-05-06T10:22:06Z
initial commit
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
