[
https://issues.apache.org/jira/browse/SPARK-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075914#comment-14075914
]
Zhihui commented on SPARK-2713:
-------------------------------
PR https://github.com/apache/spark/pull/1616
> Executors of same application in same host should only download files & jars
> once
> ---------------------------------------------------------------------------------
>
> Key: SPARK-2713
> URL: https://issues.apache.org/jira/browse/SPARK-2713
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Zhihui
>
> If Spark lunched multiple executors in one host for one application, every
> executor would download it dependent files and jars (if not using local: url)
> independently. It maybe result in huge latency. In my case, it result in 20
> seconds latency to download dependent jars(about 17M) when I lunch 32
> executors in one host(total 4 hosts).
> This patch will cache downloaded files and jars for executors to reduce
> network throughput and download latency. I my case, the latency was reduced
> from 20 seconds to less than 1 second.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
