khogeland commented on issue #26687: [SPARK-30055][k8s] Allow configuration of restart policy for Kubernetes pods URL: https://github.com/apache/spark/pull/26687#issuecomment-596726593 Good call, @holdenk, I found a conflict here while digging through the dynamic allocation code. The restart policy itself shouldn't mess up the dynamic scaling, but the allocator maps executor IDs to pods 1:1 via the label selector, so randomizing the executor ID on startup doesn't work. I would need to add some hacky string manipulation on the ID to get the label back out. Considering all the tricky edge cases that executors presents, and with https://github.com/apache/spark/pull/27568 handling the silent failure scenarios, I think it's best to reduce this PR's scope to the much more straightforward driver restart policy.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
