ifilonenko edited a comment on issue #26789: [SPARK-30160][K8S] Introduce the ExecutorPodController API URL: https://github.com/apache/spark/pull/26789#issuecomment-563403681 @erikerlandson Our use-case is that users of our compute clusters are only allowed to create/update CRDs -- one of which is a `SparkJob` CRD and another which is a `PodScaler` CRD; users are not allowed to create or delete arbitrary pods as per our internal security policy and since the driver is using the same service account as the user, the driver is unable to create pods (which is currently a requirement in the implementation of the ExecutorPodAllocator i.e. `kubernetesClient.pods().create(pod)` Our internal ExecutorPodcontroller implementation batches a bunch of addPod() or deletePod() commands and upon a commit() message, updates a `PodScaler` CRD which an operator reacts to. The PodScaler can be thought of as an intelligent ReplicaSet that can have deletions of specific pods in the set.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
