jerryshao commented on issue #24817: [WIP][SPARK-27963][core] Allow dynamic allocation without a shuffle service. URL: https://github.com/apache/spark/pull/24817#issuecomment-501656384 What if the executor is idle currently (no active job, can be removed), but later on the jobs depend one the previous shuffle stage? From my understanding that job will be failed to fetch the shuffle data and rerun the parent stages. My feeling is that if idle time is short, this exception will be quite common and confuse the users.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
