[
https://issues.apache.org/jira/browse/AIRFLOW-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163180#comment-17163180
]
ASF GitHub Bot commented on AIRFLOW-3126:
-----------------------------------------
mik-laj commented on pull request #8150:
URL: https://github.com/apache/airflow/pull/8150#issuecomment-662792653
> I've noticed you've added several PRs recently to add missing
functionality to the `airflow.cfg` for configuring the KubernetesExecutor.
There is also a [new
field](https://github.com/apache/airflow/blob/746d8de2fcaa9554b4ce7dbf261e4ab148233222/airflow/config_templates/default_airflow.cfg#L756)
which allows you to pass the YAML directly if that helps at all.
> @davlum, I've tried it and It did not appear to help—specifically,
for default worker pod creation under the K8s Executor. It required too much
specificity in the template and seemed to deactivate most other configuration
options. Can you give an example of how it can be used to add volumes in
exactly this way without sacrificing any of the existing
functionality/configurability of worker pod creation?
>
Our mistake was to add a large number of configuration options to Airfllow.
This makes it very difficult to maintain. We are now trying to change the
approach and recommend using the `pod_template_file` option. We know that
changing habits can be problematic for many users and there were even command
that would facilitate migrations, but it is not finished yet.
https://github.com/apache/airflow/pull/8009
Would you like to help develop this command? In my opinion, this is a better
solution and more forward-looking. The use of the Kubernetes API Object means
we have better documentation as we can refer to the official documentation and
it also makes it easier to use install KubernetesExecution as you can easily
copy code snippets from the internet. You always have access to the latest
Kubernetes feature too.
If we do not take steps in this direction, we will be sad all the time
because we will be missing new Kubernetes features in Airfllow. We as project
maintainers are also bored as we add/review every new feature the user may
need. The Kubernetes world is too dynamic to rely on a predefined list of
configuration options.
In addition to adding new options, we also have to delete options that have
already been dropped by the community.
https://lists.apache.org/thread.html/a68ec0a7e1535ec2011a05cf697a97a6b44345a331fa5d4f721c3391%40%3Cdev.airflow.apache.org%3E
https://cwiki.apache.org/confluence/display/AIRFLOW/Managing+Per-task+Credentials+in+KubernetesExecutor
https://github.com/apache/airflow/pull/6768
It's a lot of work to keep up to date with every new idea that hits the
Kubernetes world.
I hope you will understand the situation we are in now.
Do you have any other idea to solve this problem? I am open to discussion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> kubernetes executor not allowing new PV
> ---------------------------------------
>
> Key: AIRFLOW-3126
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3126
> Project: Apache Airflow
> Issue Type: Bug
> Components: executors
> Affects Versions: 1.10.0
> Reporter: Rahul Singh
> Assignee: Brandon Willard
> Priority: Major
>
> Kubernetes executor allows only two PV one for dag and other for logs , in
> project scenario there are multiple PV needed , like separate PV to store
> data , PV to store common script files etc . Current implementation is very
> static to only two PV.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
