Aldo created AIRFLOW-2755:
-----------------------------
Summary: k8s workers think DAGs are always in `/tmp/dags`
Key: AIRFLOW-2755
URL: https://issues.apache.org/jira/browse/AIRFLOW-2755
Project: Apache Airflow
Issue Type: Bug
Components: configuration, worker
Reporter: Aldo
We have Airflow configured to use the `KubernetesExecutor` and run tasks in
newly created pods.
I tried to use the `PythonOperator` to import the python callable from a python
module located in the DAGs directory as [that should be
possible|https://github.com/apache/incubator-airflow/blob/c7a472ed6b0d8a4720f57ba1140c8cf665757167/airflow/__init__.py#L42].
Airflow complained that the module was not found.
After a fair amount of digging we found that the issue was that the workers
have the `AIRFLOW__CORE__DAGS_FOLDER` environment variable set to `/tmp/dags`
as [you can see from the
code|https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/worker_configuration.py#L84].
Unset that environment variable from within the task's pod and running the task
manually worked as expected. I think that this path should be configurable
(I'll give it a try to add a `kubernetes.worker_dags_folder` configuration).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
