srowen commented on issue #24056: [SPARK-26152] Synchronize Worker Cleanup with Worker Shutdown URL: https://github.com/apache/spark/pull/24056#issuecomment-471589017 I see, the point is that `shutdownNow` tries to cancel the tasks. It seems like it doesn't intend to 'wait' for cleanup anyway, so I hesitate to block it on submission of the tasks. But won't it just try to forcibly terminate them once they're submitted here? If the point is that you want to submit both Futures atomically -- can we make them one Future? all the second one does is log something. If the point is just avoiding RejectedExecutionException, then your `isShutdown` check does that effectively. I'm hesitant to add locks in this complex racy scenario. If there's a problem here it's a more fundamental one of how onStop() happens before the cleanup tasks can happen. A band-aid seems OK to me, as I'm not sure this more complex approach actually resolves the 'real' problem.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
