hiboyang commented on pull request #30763: URL: https://github.com/apache/spark/pull/30763#issuecomment-794917127
Just see the discussion here. The location abstraction is a good idea. For different shuffle solutions, they could have different location implementation, e.g. Spark's default sort shuffle has BlockManagerId as the location, remote shuffle service has shuffle servers as the location, disaggregated shuffle storage (e.g. S3) has S3 bucket/path as the location. `MapOutputTracker#getMapSizesByExecutorId` may not need to throw exception? It could return a list of `Locations` and sizes. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
