khogeland commented on issue #26687: [SPARK-30055][k8s] Allow configuration of restart policy for Kubernetes pods URL: https://github.com/apache/spark/pull/26687#issuecomment-568070160 > Particularly, if it gets set to Always, what's the implication of that for batch run-to-completion jobs? I think this is a documentation concern, because it's strongly recommended you don't use kubelet restart with batch mode (due to lingering executors after driver completion). At least, if you're trying to execute at-most-once, you can't guarantee the deletion of the executor when the driver completes. >Further, I'm not 100% sure how kubelet sets the state of an executor pod if the executor container completes in between restarts The pod phase stays as "Running", it's only the container status that is updated (which Spark doesn't use). The operator will need to be updated to support that, which I have the code for, I've just been waiting until I know how this PR will end up. >Even for streaming jobs that need to continuously run, is it simply a matter of restarting the executor pods that inherit the same executor IDs? It looks like no backends currently recycle executor IDs, so I'm going to investigate if it's possible to avoid any issues by having the executors generate their own IDs during startup on k8s. >It would be up to the user to know how to use the restart policy in a manner that doesn't cause unwanted behavior. Agreed, I'll need to add some documentation here. The driver restart policy is self-explanatory, but the executor restart has subtleties that need explanation.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
