[
https://issues.apache.org/jira/browse/SPARK-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihui updated SPARK-2713:
--------------------------
Description:
If Spark lunched multiple executors in one host for one application, every
executor would download it dependent files and jars (if not using local: url)
independently. It maybe result in huge latency. In my case, it result in 20
seconds latency to download dependent jars(about 17M) when I lunch 32 executors
in one host(total 4 hosts).
This patch will cache downloaded files and jars for executors to reduce network
throughput and download latency. I my case, the latency was reduced from 20
seconds to less than 1 second.
was:
If spark lunched multiple executors in one host for one application, every
executor will download it dependent files and jars (if not using local: url)
independently. It maybe result to huge latency. In my case, it result to 20
seconds latency to download dependent jars(about 17M) when I lunch 32 executors
in one host(total 4 hosts).
This patch will cache downloaded files and jars for executors to reduce network
throughput and download latency. I my case, the latency was reduced from 20
seconds to less than 1 second.
> Executors of same application in same host should only download files & jars
> once
> ---------------------------------------------------------------------------------
>
> Key: SPARK-2713
> URL: https://issues.apache.org/jira/browse/SPARK-2713
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.0.0
> Reporter: Zhihui
>
> If Spark lunched multiple executors in one host for one application, every
> executor would download it dependent files and jars (if not using local: url)
> independently. It maybe result in huge latency. In my case, it result in 20
> seconds latency to download dependent jars(about 17M) when I lunch 32
> executors in one host(total 4 hosts).
> This patch will cache downloaded files and jars for executors to reduce
> network throughput and download latency. I my case, the latency was reduced
> from 20 seconds to less than 1 second.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
