Zhihui created SPARK-2713:
-----------------------------
Summary: Executors of same application in same host should only
download files & jars once
Key: SPARK-2713
URL: https://issues.apache.org/jira/browse/SPARK-2713
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.0.0
Reporter: Zhihui
If spark lunched multiple executors in one host for one application, every
executor will download it dependent files and jars (if not using local: url)
independently. It maybe result to huge latency. In my case, it result to 20
seconds latency to download dependent jars(about 17M) when I lunch 32 executors
in one host(total 4 hosts).
This patch will cache downloaded files and jars for executors to reduce network
throughput and download latency. I my case, the latency was reduced from 20
seconds to less than 1 second.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
