mik-laj edited a comment on pull request #8150: URL: https://github.com/apache/airflow/pull/8150#issuecomment-662792653
> I've noticed you've added several PRs recently to add missing functionality to the `airflow.cfg` for configuring the KubernetesExecutor. There is also a [new field](https://github.com/apache/airflow/blob/746d8de2fcaa9554b4ce7dbf261e4ab148233222/airflow/config_templates/default_airflow.cfg#L756) which allows you to pass the YAML directly if that helps at all. > @davlum, I've tried it and It did not appear to help—specifically, for default worker pod creation under the K8s Executor. It required too much specificity in the template and seemed to deactivate most other configuration options. Can you give an example of how it can be used to add volumes in exactly this way without sacrificing any of the existing functionality/configurability of worker pod creation? > In my opinion, our mistake was to add a large number of configuration options to Airflow. This makes it very difficult to maintain. I am trying to change the approach and recommend using the `pod_template_file` option. I know that changing habits can be problematic for many users and there were even command that would facilitate migrations, but it is not finished yet. https://github.com/apache/airflow/pull/8009 Would you like to help develop this command? In my opinion, this is a better solution and more forward-looking. The use of the Kubernetes API Object means we have better documentation as we can refer to the official documentation and it also makes it easier to use install KubernetesExecution as you can easily copy code snippets from the internet. You always have access to the latest Kubernetes feature too. If we do not take steps in this direction, we will be sad all the time because we will be missing new Kubernetes features in Airfllow. Project maintainers are also bored as we add/review every new feature the user may need. Just for a simple git sync we have 17 different configuration options: - git_repo - git_branch - git_sync_depth - git_subpath - git_sync_rev - git_user - git_password - git_sync_root - git_sync_dest - git_dags_folder_mount_point - git_ssh_key_secret_name - git_ssh_known_hosts_configmap_name - git_sync_credentials_secret - git_sync_container_repository - git_sync_container_tag - git_sync_init_container_name - git_sync_run_as_user And we don't address all the sync options that users expect with these options e.g. [some users need to login to the repository with gcloud](https://cloud.google.com/sdk/gcloud/reference/source/repos/clone) The Kubernetes world is too dynamic to rely on a predefined list of configuration options. In addition to adding new options, we also have to delete options that have already been dropped by the community. https://lists.apache.org/thread.html/a68ec0a7e1535ec2011a05cf697a97a6b44345a331fa5d4f721c3391%40%3Cdev.airflow.apache.org%3E https://cwiki.apache.org/confluence/display/AIRFLOW/Managing+Per-task+Credentials+in+KubernetesExecutor https://github.com/apache/airflow/pull/6768 This option is a good example of why we shouldn't add new Kubernetes-related options to Airflow.. Before the option was added to Airflow, FengLu sent a few messages on the mailing list, met with other people, prepare PR. After a long time this option didn't even come out of the beta phase, and then it was depreciated. Then another person would have to find the options, check if they were supported, discuss if they could be deleted, then prepare a fix and delete this option. It's a lot of work to keep up to date with every new idea that hits the Kubernetes world. On the other hand, we have a solution that is flexible enough for each user to adapt it to their needs. I hope you will understand the situation we are in now. Do you have any other idea to solve this problem? I am open to discussion. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
