GitHub user li-zhihui opened a pull request:
https://github.com/apache/spark/pull/1616
[SPARK-2713] Executors of same application in same host should only
download files & jars once
If Spark lunched multiple executors in one host for one application, every
executor would download it dependent files and jars (if not using local: url)
independently. It maybe result in huge latency. In my case, it result in 20
seconds latency to download dependent jars(about 17M) when I lunched 32
executors in every host(total 4 hosts).
This patch will cache downloaded files and jars for executors to reduce
network throughput and download latency. I my case, the latency was reduced
from 20 seconds to less than 1 second.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/li-zhihui/spark cachefiles
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1616.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1616
----
commit 967817e082d72fbd7c7150780ceeba6f69588c34
Author: Li Zhihui <[email protected]>
Date: 2014-07-28T01:41:11Z
Executors of same application in same host should only download files &
jars once
commit 0bb6224ee33ef2639e75fdfc1396f84f1821ec77
Author: Li Zhihui <[email protected]>
Date: 2014-07-28T01:54:20Z
Release lock before copy files
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
