[
https://issues.apache.org/jira/browse/AIRFLOW-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164235#comment-17164235
]
ASF GitHub Bot commented on AIRFLOW-3126:
-----------------------------------------
fengsi commented on pull request #8150:
URL: https://github.com/apache/airflow/pull/8150#issuecomment-663393946
And I feel the larger question would be: is it possible to come up with an
elegant solution such that Airflow could "bootstrap" itself in K8s world? If
we take a look at Airflow in K8s, its manifests are usually handled by other
tools (e.g., Helm). Now both `KubernetesExecutor` and `KubernetesPodOperator`
do similar things but rely on Airflow (K8s Python client), causing
inconsistency between ecosystems.
My current set up:
1. Airflow scheduler & web server(s): managed by Helm
2. Airflow worker: managed by `KubernetesExecutor` through tons of
`AIRFLOW__KUBERNETES__*` settings but still incomplete and require hacks (like
the new `pod_template_file`)
3. Airflow operators: executed by `KubernetesExecutor`, some being
`KubernetesPodOperator` that may require totally different settings, but are
partially contaminated by `KubernetesExecutor` settings
Guess what I'm trying to say is K8s support in Airflow could be improved for
sure. I understand Airflow was not originally designed for K8s since day one,
but it'll be great if there is a way to consolidate these use cases in Airflow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> kubernetes executor not allowing new PV
> ---------------------------------------
>
> Key: AIRFLOW-3126
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3126
> Project: Apache Airflow
> Issue Type: Bug
> Components: executors
> Affects Versions: 1.10.0
> Reporter: Rahul Singh
> Assignee: Brandon Willard
> Priority: Major
>
> Kubernetes executor allows only two PV one for dag and other for logs , in
> project scenario there are multiple PV needed , like separate PV to store
> data , PV to store common script files etc . Current implementation is very
> static to only two PV.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
