squito commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks when executors are lost URL: https://github.com/apache/spark/pull/24462#issuecomment-516149439 why do you want to store the data files on hdfs, but the index files on the executors? This seems to have the worst of both worlds -- the (bad) resiliency of local storage, and the (bad) performance of remote reads. Or is the index file backed up somewhere as well? I definitely understand the general problem with knowing what to do about shuffle data when an executor is lost. So I understand why you want to do something *like* what this PR does. But it probably makes more sense to address this as part of the other shuffle api changes, if possible, not as another config. You might not be able to do everything you want -- in particular, the new api does not support multiple locations for shuffle data. We decided that was out-of-scope, for now (but maybe a future enhancement). Is that what you're looking for -- one copy on the executors local disk, and another copy on hdfs?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
