kondla opened a new issue #8647:
URL: https://github.com/apache/airflow/issues/8647
**Apache Airflow version**:
apache-airflow = {extras = ["gcp"],version = "==1.10.9"}
**Environment**:
- **OS**: macOS Catalina
- **Install tools**: Pipenv
**What happened**:
Running DAGs from a subfolder under /dags, e.g. /dags/subfolder with the
**same name** as running plugins from a /plugins/subfolder breaks the import in
the sub dags folder (sorry for the convolution).
E.g. importing plugins from a sub directory in the plugins folder fails with
directories set up like this:
.
├── dags
│ └── sub_dag
│ ├── __init__.py
│ └── **dagaroni**
│ ├── __init__.py
│ └── dag_script.py
└── plugins
└── **dagaroni**
└── placeholder
├── __init__.py
└── operators.py
I.e. when the name of the sub directory under dags where I am importing my
plugin, has the same name as the sub directory within the plugins folder where
I put my plugin
But imports _succeed_ with directories named like this, when the dags
subfolder and the plugins subfolder are named differently:
.
├── dags
│ └── sub_dag
│ ├── __init__.py
│ └── **dagaroni**
│ ├── __init__.py
│ └── dag_script.py
└── plugins
└── **different_dagaroni**
└── placeholder
├── __init__.py
└── operators.py
<details><summary>Error messages</summary>
[2020-04-30 15:54:44,806] {dagbag.py:92} INFO - Filling up the DagBag from
/Users/that_guy/VisualStudioProjects/that_project/dags
[2020-04-30 15:54:44,818] {dagbag.py:207} ERROR - Failed to import:
/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py
Traceback (most recent call last):
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/site-packages/airflow/models/dagbag.py",
line 204, in process_file
m = imp.load_source(mod_name, filepath)
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/imp.py",
line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File
"/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py",
line 12, in <module>
from dagaroni.dummy_dag import raw
ModuleNotFoundError: No module named 'dagaroni.dummy_dag'
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/site-packages/airflow/configuration.py:627:
DeprecationWarning: You have two airflow.cfg files:
/Users/that_guy/airflow/airflow.cfg and
/Users/that_guy/VisualStudioProjects/that_project/airflow.cfg. Airflow used to
look at ~/airflow/airflow.cfg, even when AIRFLOW_HOME was set to a different
value. Airflow will now only read
/Users/that_guy/VisualStudioProjects/that_project/airflow.cfg, and you should
remove the other file
category=DeprecationWarning,
[2020-04-30 15:54:45 +0200] [8697] [INFO] Starting gunicorn 19.10.0
[2020-04-30 15:54:45 +0200] [8697] [INFO] Listening at: http://0.0.0.0:8080
(8697)
[2020-04-30 15:54:45 +0200] [8697] [INFO] Using worker: sync
[2020-04-30 15:54:45 +0200] [8706] [INFO] Booting worker with pid: 8706
[2020-04-30 15:54:45 +0200] [8707] [INFO] Booting worker with pid: 8707
[2020-04-30 15:54:45 +0200] [8708] [INFO] Booting worker with pid: 8708
[2020-04-30 15:54:45 +0200] [8709] [INFO] Booting worker with pid: 8709
[2020-04-30 15:54:46,167] {__init__.py:51} INFO - Using executor
SequentialExecutor
[2020-04-30 15:54:46,168] {dagbag.py:92} INFO - Filling up the DagBag from
/Users/that_guy/VisualStudioProjects/that_project/dags
[2020-04-30 15:54:46,204] {dagbag.py:207} ERROR - Failed to import:
/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py
Traceback (most recent call last):
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/site-packages/airflow/models/dagbag.py",
line 204, in process_file
m = imp.load_source(mod_name, filepath)
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/imp.py",
line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File
"/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py",
line 12, in <module>
from dagaroni.dummy_dag import raw
ModuleNotFoundError: No module named 'dagaroni.dummy_dag'
[2020-04-30 15:54:46,296] {__init__.py:51} INFO - Using executor
SequentialExecutor
[2020-04-30 15:54:46,297] {dagbag.py:92} INFO - Filling up the DagBag from
/Users/that_guy/VisualStudioProjects/that_project/dags
[2020-04-30 15:54:46,328] {__init__.py:51} INFO - Using executor
SequentialExecutor
[2020-04-30 15:54:46,331] {dagbag.py:92} INFO - Filling up the DagBag from
/Users/that_guy/VisualStudioProjects/that_project/dags
[2020-04-30 15:54:46,328] {dagbag.py:207} ERROR - Failed to import:
/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py
Traceback (most recent call last):
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/site-packages/airflow/models/dagbag.py",
line 204, in process_file
m = imp.load_source(mod_name, filepath)
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/imp.py",
line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File
"/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py",
line 12, in <module>
from dagaroni.dummy_dag import raw
ModuleNotFoundError: No module named 'dagaroni.dummy_dag'
[2020-04-30 15:54:46,371] {dagbag.py:207} ERROR - Failed to import:
/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py
Traceback (most recent call last):
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/site-packages/airflow/models/dagbag.py",
line 204, in process_file
m = imp.load_source(mod_name, filepath)
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/imp.py",
line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File
"/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py",
line 12, in <module>
from dagaroni.dummy_dag import raw
ModuleNotFoundError: No module named 'dagaroni.dummy_dag'
[2020-04-30 15:54:46,439] {__init__.py:51} INFO - Using executor
SequentialExecutor
[2020-04-30 15:54:46,440] {dagbag.py:92} INFO - Filling up the DagBag from
/Users/that_guy/VisualStudioProjects/that_project/dags
[2020-04-30 15:54:46,474] {dagbag.py:207} ERROR - Failed to import:
/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py
Traceback (most recent call last):
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/site-packages/airflow/models/dagbag.py",
line 204, in process_file
m = imp.load_source(mod_name, filepath)
File
"/Users/that_guy/.local/share/virtualenvs/that_environment/lib/python3.7/imp.py",
line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed
File
"/Users/that_guy/VisualStudioProjects/that_project/dags/sub_dag/dagaroni/__init__.py",
line 12, in <module>
from dagaroni.dummy_dag import raw
ModuleNotFoundError: No module named 'dagaroni.dummy_dag'
</details>
**What you expected to happen**:
Normally, I would expect that I can use subfolders under
`dags/subfolder/__init__.py` even if they have the same name as subfolders
under `plugins/subfolder/my_plugin`
I think the way airflow attaches plugins to its own module so they can be
called as `from airflow.operators.subfolder import ThatOperator` gets confused
when the names within search paths overlap
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
