gillbuchanan commented on issue #13680:
URL: https://github.com/apache/airflow/issues/13680#issuecomment-763847427
I'm having this issue too. After upgrading to 2.0.0, while running with
`KubernetesExecutor` in an Azure k8s cluster, the worker pods have been failing
with the same error message
```[2021-01-20 14:42:35,290] {dagbag.py:440} INFO - Filling up the DagBag
from /opt/airflow/dags/test_dag.py
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/__main__.py", line
40, in main
args.func(args)
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py",
line 48, in command
return func(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line
89, in wrapper
return f(*args, **kwargs)
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py",
line 216, in task_run
dag = get_dag(args.subdir, args.dag_id)
File
"/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line
189, in get_dag
'parse.'.format(dag_id)
airflow.exceptions.AirflowException: dag_id could not be found: test. Either
the dag did not exist or it failed to parse.
```
I access my DAGs via an Azure file store mounted as a volume using a pvc,
and have found that the worker pods are not being created with the same mounted
volume. It seems that the previously available `airflow.cfg` values
```
[kubernetes]
dags_volume_claim =
dags_volume_subpath =
```
no longer exist in version 2.0.0, and the worker pods therefore cannot be
created with the the proper volume mounts, leading to the above error.
Any way around this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
