[
https://issues.apache.org/jira/browse/SPARK-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or closed SPARK-2713.
----------------------------
Resolution: Fixed
Fix Version/s: 1.2.0
Assignee: Zhihui
Target Version/s: 1.2.0
> Executors of same application in same host should only download files & jars
> once
> ---------------------------------------------------------------------------------
>
> Key: SPARK-2713
> URL: https://issues.apache.org/jira/browse/SPARK-2713
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Zhihui
> Assignee: Zhihui
> Fix For: 1.2.0
>
>
> If Spark lunched multiple executors in one host for one application, every
> executor would download it dependent files and jars (if not using local: url)
> independently. It maybe result in huge latency. In my case, it result in 20
> seconds latency to download dependent jars(about 17M) when I lunch 32
> executors in one host(total 4 hosts).
> This patch will cache downloaded files and jars for executors to reduce
> network throughput and download latency. I my case, the latency was reduced
> from 20 seconds to less than 1 second.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
