GitHub user Victsm opened a pull request:
https://github.com/apache/spark/pull/18801
SPARK-10878 Fix race condition when multiple clients resolves artifacts at
the same time
## What changes were proposed in this pull request?
When multiple clients attempt to resolve artifacts via the "--packages"
parameter, they could run into race condition when they each attempt to modify
the dummy "org.apache.spark-spark-submit-parent-default.xml" file created in
the default ivy cache dir.
This patch changes the behavior to encode timestamp in the dummy module
descriptor so each client will operate on a different resolution file in the
ivy cache dir. In addition, this patch changes the behavior of when and which
resolution files are cleaned to prevent accumulation of resolution files in the
default ivy cache dir.
## How was this patch tested?
Unit test added in `SparkSubmitUtilsSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Victsm/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18801.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18801
----
commit 1ace5cc8232536bcc336042aec686fed1204f799
Author: Min Shen <[email protected]>
Date: 2017-07-31T23:57:12Z
SPARK-10878 Fix race condidtion when multiple clients resolves artifacts at
the same time.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
