[
https://issues.apache.org/jira/browse/SPARK-35702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377048#comment-17377048
]
Christian Thiel commented on SPARK-35702:
-----------------------------------------
[~holden] yes it does matter, as different namespaces can have different
policies. If working in a Notebook in Kubernetes, users typically don't have
CREATE / DELETE Verbs unlocked for Pods and Configmaps in the Notebook
namespace, in order to not interfere with other users Pods.
A Serviceaccount in another executor namespace however allows users to spawn
executors there.
There is acutally a workaround possible, which is however a bit messy and
uncomplete with the following settings:
spark.master: k8s://https://<kubernetes-api-ip>:443
spark.driver.host: Pod IP of Notebook / Driver in Namespace 1
spark.kubernetes.namespace: Namespace 2
spark.kubernetes.image: Image to use for executors
This however breakes the [Client Mode Executor Pod Garbage
Collection|https://spark.apache.org/docs/latest/running-on-kubernetes.html#client-mode-executor-pod-garbage-collection]
as the driver is expected to be in spark.kubernetes.namespace.
> Kubernetes: Allow different namespaces for Driver and Executors
> ---------------------------------------------------------------
>
> Key: SPARK-35702
> URL: https://issues.apache.org/jira/browse/SPARK-35702
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.1.2
> Reporter: Christian Thiel
> Priority: Major
>
> Currently the Kubernetes namespace of driver AND executors always needs to be
> identical - specified with {{spark.kubernetes.namespace}}.
> Especially when developing in Kubernetes, this is a harsh restriction: In a
> _notebook_ Namespace I want to run spark in driver in client mode for
> interaction, however executors should get their own dedicated resources in a
> dedicated namespace with different policies and different lifecycles.
> This could be solved by adding the following two additional configuration
> options:
> {code:java}
> spark.kubernetes.executor.pod.namespace
> spark.kubernetes.driver.pod.namespace
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
