[
https://issues.apache.org/jira/browse/AIRFLOW-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163189#comment-17163189
]
ASF GitHub Bot commented on AIRFLOW-3126:
-----------------------------------------
mik-laj edited a comment on pull request #8150:
URL: https://github.com/apache/airflow/pull/8150#issuecomment-662792653
> I've noticed you've added several PRs recently to add missing
functionality to the `airflow.cfg` for configuring the KubernetesExecutor.
There is also a [new
field](https://github.com/apache/airflow/blob/746d8de2fcaa9554b4ce7dbf261e4ab148233222/airflow/config_templates/default_airflow.cfg#L756)
which allows you to pass the YAML directly if that helps at all.
> @davlum, I've tried it and It did not appear to help—specifically,
for default worker pod creation under the K8s Executor. It required too much
specificity in the template and seemed to deactivate most other configuration
options. Can you give an example of how it can be used to add volumes in
exactly this way without sacrificing any of the existing
functionality/configurability of worker pod creation?
>
In my opinion, our mistake was to add a large number of configuration
options to Airflow. This makes it very difficult to maintain. I am trying to
change the approach and recommend using the `pod_template_file` option. I know
that changing habits can be problematic for many users and there were even
command that would facilitate migrations, but it is not finished yet.
https://github.com/apache/airflow/pull/8009
Would you like to help develop this command? In my opinion, this is a better
solution and more forward-looking. The use of the Kubernetes API Object means
we have better documentation as we can refer to the official documentation and
it also makes it easier to use install KubernetesExecution as you can easily
copy code snippets from the internet. You always have access to the latest
Kubernetes feature too.
If we do not take steps in this direction, we will be sad all the time
because we will be missing new Kubernetes features in Airfllow. Project
maintainers are also bored as we add/review every new feature the user may
need.
Just for a simple git sync we have 17 different configuration options:
- git_repo
- git_branch
- git_sync_depth
- git_subpath
- git_sync_rev
- git_user
- git_password
- git_sync_root
- git_sync_dest
- git_dags_folder_mount_point
- git_ssh_key_secret_name
- git_ssh_known_hosts_configmap_name
- git_sync_credentials_secret
- git_sync_container_repository
- git_sync_container_tag
- git_sync_init_container_name
- git_sync_run_as_user
And we don't address all the sync options that users expect with these
options e.g. [some users need to login to the repository with
gcloud](https://cloud.google.com/sdk/gcloud/reference/source/repos/clone)
The Kubernetes world is too dynamic to rely on a predefined list of
configuration options.
In addition to adding new options, we also have to delete options that have
already been dropped by the community.
https://lists.apache.org/thread.html/a68ec0a7e1535ec2011a05cf697a97a6b44345a331fa5d4f721c3391%40%3Cdev.airflow.apache.org%3E
https://cwiki.apache.org/confluence/display/AIRFLOW/Managing+Per-task+Credentials+in+KubernetesExecutor
https://github.com/apache/airflow/pull/6768
This option is a good example of why we shouldn't add new Kubernetes-related
options to Airflow.. Before the option was added to Airflow, FengLu sent a few
messages on the mailing list, met with other people, prepare PR. After a long
time this option didn't even come out of the beta phase, and then it was
depreciated. Then another person would have to find the options, check if they
were supported, discuss if they could be deleted, then prepare a fix and delete
this option.
It's a lot of work to keep up to date with every new idea that hits the
Kubernetes world.
On the other hand, we have a solution that is flexible enough for each user
to adapt it to their needs.
I hope you will understand the situation we are in now.
Do you have any other idea to solve this problem? I am open to discussion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> kubernetes executor not allowing new PV
> ---------------------------------------
>
> Key: AIRFLOW-3126
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3126
> Project: Apache Airflow
> Issue Type: Bug
> Components: executors
> Affects Versions: 1.10.0
> Reporter: Rahul Singh
> Assignee: Brandon Willard
> Priority: Major
>
> Kubernetes executor allows only two PV one for dag and other for logs , in
> project scenario there are multiple PV needed , like separate PV to store
> data , PV to store common script files etc . Current implementation is very
> static to only two PV.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
