holdenk commented on pull request #28331: URL: https://github.com/apache/spark/pull/28331#issuecomment-619258239
If you've got some cycles to take a look at the WIP PR @prakharjain09 I'd appreciate it. This builds on top of the cache block migration. Why this PR is still a work in progress: 1) Only works with indexed shuffle files 2) I'm not sure if this is the best way to copy shuffle files between executors Future work (not planned in this PR but in the future): 1) Supporting write back to some type of DFS 2) Updating the listener that tracks location of shuffle blocks for scale down to understand block migrations 3) Integrate decommissioning into our planned scale down, not just cluster manager/cloud triggered scale downs. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
