squito commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-526644966 Thanks for looking @cloud-fan , I think there actually is a very interesting idea about avoiding asking the driver. The executors could ask each other for the local dirs (or ask the external shuffle service, so we know those dirs even after the executor goes down, for the cases where that is useful). Another idea would be to have the executors communicate via the filesystem -- there is some agreed upon location where each exec writes its set of local dirs. This optimization is only useful where the executors share a filesystem. This may also indirectly help with the k8s case you mentioned. On k8s, its a good question. I don't think the performance benefit will work, as the pods generally are isolated from each other, and can't read each others filesystems. But we have to make sure things aren't broken by this because they try to read from a place they don't have access to.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
