[
https://issues.apache.org/jira/browse/AIRFLOW-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570752#comment-16570752
]
ASF subversion and git services commented on AIRFLOW-2755:
----------------------------------------------------------
Commit 90e88dfe82d781236db39ce3b6dfd92d5f48f6e7 in incubator-airflow's branch
refs/heads/master from Aldo Giambelluca
[ https://gitbox.apache.org/repos/asf?p=incubator-airflow.git;h=90e88df ]
[AIRFLOW-2755] Added `kubernetes.worker_dags_folder` configuration (#3612)
It was previously hardcoded to `/tmp/dags`.
This causes problems with python import of modules in the DAGs folder.
> k8s workers think DAGs are always in `/tmp/dags`
> ------------------------------------------------
>
> Key: AIRFLOW-2755
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2755
> Project: Apache Airflow
> Issue Type: Bug
> Components: configuration, worker
> Reporter: Aldo
> Assignee: Aldo
> Priority: Minor
> Fix For: 2.0.0
>
>
> We have Airflow configured to use the `KubernetesExecutor` and run tasks in
> newly created pods.
> I tried to use the `PythonOperator` to import the python callable from a
> python module located in the DAGs directory as [that should be
> possible|https://github.com/apache/incubator-airflow/blob/c7a472ed6b0d8a4720f57ba1140c8cf665757167/airflow/__init__.py#L42].
> Airflow complained that the module was not found.
> After a fair amount of digging we found that the issue was that the workers
> have the `AIRFLOW__CORE__DAGS_FOLDER` environment variable set to `/tmp/dags`
> as [you can see from the
> code|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/worker_configuration.py#L84].
> Unset that environment variable from within the task's pod and running the
> task manually worked as expected. I think that this path should be
> configurable (I'll give it a try to add a `kubernetes.worker_dags_folder`
> configuration).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
