There is a JIRA about this thing ( https://issues.apache.org/jira/browse/SPARK-6521). In the current Spark shuffle fetch still leverages Netty even two executors are on the same node, but according to the test on the JIRA, the performance is close whether to bypass network or not. From my understanding, kernel will not transfer data into NIC if it is just a loopback communication (please correct me if I'm wrong).
On Wed, May 10, 2017 at 5:53 PM, raintung li <raintung...@gmail.com> wrote: > Hi all, > > Now Spark only think the executorId same that fetch local file, but for > same IP different ExecutorId will fetch using network that actually it can > be fetch in the local Or Loopback. > > Apparently fetch the local file that it is fast that can use the LVS > cache. > > How do you think? > > Regards > -Raintung >
