Hi folks, Bringing up this discussion again for more inputs.
In addition, while preparing the changes I proposed below, I also noticed a few issues relating to *KubernetesExecutor*'s *multi_namespace_mode*, which made me question if this mode ever worked with complete functionalities. For example, the try_adopt_task_instances method of the Executor only checks pods in a single namespace (the namespace specified in config's kubernetes_executor section), not "multiple" namespaces. The same for clear_not_launched_queued_tasks method. Please correct me if I'm wrong or missed anything. If any folks have been running *KubernetesExecutor* with *multi_namespace_mode*, I would like to hear your experience. Again, looking forward to your inputs and let's work together to further enhance this great feature of Airflow😃 Regards, XD On Fri, Nov 18, 2022 at 11:46 AM Xiaodong Deng <[email protected]> wrote: > Hi folks, > > I'm writing to share some thoughts and changes we would like to propose, > in order to make *KubernetesExecutor*'s *multi_namespace_mode* more > flexible & truly enterprise-ready. > > *# Background & Issue* > If you check *KubernetesExecutor*'s *multi_namespace_mode *configuration > <https://github.com/apache/airflow/blob/2.4.3/airflow/config_templates/default_airflow.cfg#L1186-L1187>, > you will notice the statement "*Will require creating a cluster-role for > the scheduler*". That's because under the hood KubernetesExecutor is > using *list_pod_for_all_namespaces() > <https://github.com/apache/airflow/blob/2.4.3/airflow/executors/kubernetes_executor.py#L140-L142>* > when it is *multi_namespace_mode*. This is a cluster-wise query and would > surely need a cluster role. > > This would become a problem in some of the enterprise environments. For > example: > - The Kubernetes clusters may be managed by a central team. Each user only > gets full access to namespace(s), and is never allowed to have a > cluster-role. > - A Kuberntes cluster may be shared by multiple teams, and it's never a > good idea to grant anyone cluster-role. > > This does block some users from using Airflow's *KubernetesExecutor* in > the vanilla shape, and these users have to make certain customisations > (like what our team has to do). > > *# Proposal* > We would like to propose refactoring KubernetesExecutor so that users can > provide a list of namespaces in the configuration, and then > Scheduler/Executor only needs to have a namespace-role to these namespaces. > > We already have a draft ready to be shared really soon. We would like to > hear feedback & thoughts from the community about this idea, before we > proceed to create a PR. > > Looking forward to your thoughts, and wish everyone a good weekend (and > for some folks a nice Thanksgiving break) ahead! > > > Regards, > XD > >
