[ https://issues.apache.org/jira/browse/SPARK-27651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052251#comment-17052251 ]
Thomas Graves commented on SPARK-27651: --------------------------------------- thanks, that makes sense. I can look in more details at the code, but I assume the executors could ask the other executors for the directory list rather than going to the external shuffle service if we wanted to support it. > Avoid the network when block manager fetches shuffle blocks from the same host > ------------------------------------------------------------------------------ > > Key: SPARK-27651 > URL: https://issues.apache.org/jira/browse/SPARK-27651 > Project: Spark > Issue Type: Improvement > Components: Block Manager > Affects Versions: 3.0.0 > Reporter: Attila Zsolt Piros > Assignee: Attila Zsolt Piros > Priority: Major > Fix For: 3.0.0 > > > When a shuffle block (content) is fetched the network is always used even > when it is fetched from the external shuffle service running on the same > host. This can be avoided by getting the local directories of the same host > executors from the external shuffle service and accessing those blocks from > the disk directly. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org