jealous commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks when executors are lost URL: https://github.com/apache/spark/pull/24462#issuecomment-489482553 Hi @Ngone51 , check our project page here: https://github.com/MemVerge/splash/ And the design document here: https://github.com/MemVerge/splash/blob/master/doc/Design.md The idea is to extract all file/network IO from the shuffle manager. We supply a shared file system implementation for the storage interface. But you could always implement your own plugin. And we have the same issue, we don't want to re-submit the tasks when the executors are lost. Because the shuffle files are persisted somewhere else.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
