[
https://issues.apache.org/jira/browse/SPARK-27651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052251#comment-17052251
]
Thomas Graves commented on SPARK-27651:
---------------------------------------
thanks, that makes sense.
I can look in more details at the code, but I assume the executors could ask
the other executors for the directory list rather than going to the external
shuffle service if we wanted to support it.
> Avoid the network when block manager fetches shuffle blocks from the same host
> ------------------------------------------------------------------------------
>
> Key: SPARK-27651
> URL: https://issues.apache.org/jira/browse/SPARK-27651
> Project: Spark
> Issue Type: Improvement
> Components: Block Manager
> Affects Versions: 3.0.0
> Reporter: Attila Zsolt Piros
> Assignee: Attila Zsolt Piros
> Priority: Major
> Fix For: 3.0.0
>
>
> When a shuffle block (content) is fetched the network is always used even
> when it is fetched from the external shuffle service running on the same
> host. This can be avoided by getting the local directories of the same host
> executors from the external shuffle service and accessing those blocks from
> the disk directly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
