hterik opened a new issue, #36944: URL: https://github.com/apache/airflow/issues/36944
### Description Hi We have a need for running the `KubernetesExecutor` to schedule tasks on two different Kubernetes clusters. If i'm not mistaken, today the `KubernetesExecutor` is very tied to one particular configuration context. It seems like one can use `KubernetesOperator` instead but it is not as convenient as having the executor and requires a celery executor to launch the pods. I have an idea that one can extend the way `CeleryKubernetesExecutor` routes tasks and instead of just supporting 2 fixed executors, it can support N number of executors. (https://en.wikipedia.org/wiki/Zero_one_infinity_rule) This way you can instantiate 2x `KubernetesExecutor`, each with different context, and use the queue to route them accordingly. The biggest change needed for this will be in the configurations. By making the Executor support N sub-executors, likewise need to be done with how config is read and written. Something like this, yaml-pseudo-coding here: ```yaml [core] executors: - type: kubernetes_executor queue: onprem params: cluster_context: onprem - type: kubernetes_executor queue: production params: cluster_context: production - type: celery_executor queue: ^celery_.* params: broker_url: .... ``` Is this a good idea? ### Use case/motivation _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
