fengsi commented on pull request #8150: URL: https://github.com/apache/airflow/pull/8150#issuecomment-663393946
And I feel the larger question would be: is it possible to come up with an elegant solution such that Airflow could "bootstrap" itself in K8s world? If we take a look at Airflow in K8s, its manifests are usually handled by other tools (e.g., Helm). Now both `KubernetesExecutor` and `KubernetesPodOperator` do similar things but rely on Airflow (K8s Python client), causing inconsistency between ecosystems. My current set up: 1. Airflow scheduler & web server(s): managed by Helm 2. Airflow worker: managed by `KubernetesExecutor` through tons of `AIRFLOW__KUBERNETES__*` settings but still incomplete and require hacks (like the new `pod_template_file`) 3. Airflow operators: executed by `KubernetesExecutor`, some being `KubernetesPodOperator` that may require totally different settings, but are partially contaminated by `KubernetesExecutor` settings Guess what I'm trying to say is K8s support in Airflow could be improved for sure. I understand Airflow was not originally designed for K8s since day one, but it'll be great if there is a way to consolidate these use cases in Airflow. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
