[
https://issues.apache.org/jira/browse/AIRFLOW-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889485#comment-16889485
]
immerrr again edited comment on AIRFLOW-4982 at 7/20/19 11:58 AM:
------------------------------------------------------------------
Got bitten by this too without entrypoints, but rather because of the desire to
unit-test the custom plugins we use in our project. I put them into our python
package, and sure enough, they cannot be imported because of circular imports.
Here's a minimal repro:
{code:java}
mkdir tmp; cd tmp
mkdir -p airflow/plugins
cat <<EOF > airflow/plugins/myplugin.py
from mycode import MyPlugin
EOF
cat <<EOF > mycode.py
from airflow.plugins_manager import AirflowPlugin
class MyPlugin(AirflowPlugin):
name = 'my_plugin'
EOF
python3 -m venv ./venv
./venv/bin/pip install git+https://github.com/apache/airflow
PYTHONPATH=$PWD AIRFLOW_HOME=$PWD/airflow ./venv/bin/python -c 'import mycode'
{code}
And it raises a circular dependency import error {{mycode.py ->
airflow.plugins_manager -> airflow/plugins/myplugin.py -> mycode.py}}:
{code:java}
/home/immerrr/src/tmp/venv/lib/python3.7/site-packages/airflow/models/dagbag.py:21:
DeprecationWarning: the imp module is deprecated in favour of importlib; see
the module's documentation for alternative uses
import imp
[2019-07-20 13:28:46,340] {plugins_manager.py:148} ERROR - cannot import name
'MyPlugin' from 'mycode' (/home/immerrr/src/tmp/mycode.py)
Traceback (most recent call last):
File
"/home/immerrr/src/tmp/venv/lib/python3.7/site-packages/airflow/plugins_manager.py",
line 142, in <module>
m = imp.load_source(namespace, filepath)
File "/home/immerrr/src/pyenv/versions/3.7.4/lib/python3.7/imp.py", line 171,
in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/immerrr/src/tmp/airflow/plugins/myplugin.py", line 1, in <module>
from mycode import MyPlugin
ImportError: cannot import name 'MyPlugin' from 'mycode'
(/home/immerrr/src/tmp/mycode.py)
[2019-07-20 13:28:46,341] {plugins_manager.py:149} ERROR - Failed to import
plugin /home/immerrr/src/tmp/airflow/plugins/myplugin.py
{code}
It works OK if you import stuff going from airflow side of things, i.e. import
things from `$AIRFLOW_HOME/plugins` first.
Similar thing happens when you build custom operators: {{from airflow.models
import BaseOperator}} import creates a circular dependency via
{code:java}
myoperator.py -> airflow.models -> airflow/_init_.py ->
airflow.operators._integrate_plugins() ->
-> airflow.plugins_manager -> $AIRFLOW_HOME/plugins/myplugin.py -> mycode.py ->
myoperator.py{code}
was (Author: immerrr):
Got bitten by this too without entrypoints, but rather because of the desire to
unit-test the custom plugins we use in our project. I put them into our python
package, and sure enough, they cannot be imported because of circular imports.
Here's a minimal repro:
{code:java}
mkdir tmp; cd tmp
mkdir -p airflow/plugins
cat <<EOF > airflow/plugins/myplugin.py
from mycode import MyPlugin
EOF
cat <<EOF > mycode.py
from airflow.plugins_manager import AirflowPlugin
class MyPlugin(AirflowPlugin):
name = 'my_plugin'
EOF
python3 -m venv ./venv
./venv/bin/pip install git+https://github.com/apache/airflow
PYTHONPATH=$PWD AIRFLOW_HOME=$PWD/airflow ./venv/bin/python -c 'import mycode'
{code}
And it raises a circular dependency import error {{mycode.py ->
airflow.plugins_manager -> airflow/plugins/myplugin.py -> mycode.py}}:
{code:java}
/home/immerrr/src/tmp/venv/lib/python3.7/site-packages/airflow/models/dagbag.py:21:
DeprecationWarning: the imp module is deprecated in favour of importlib; see
the module's documentation for alternative uses
import imp
[2019-07-20 13:28:46,340] {plugins_manager.py:148} ERROR - cannot import name
'MyPlugin' from 'mycode' (/home/immerrr/src/tmp/mycode.py)
Traceback (most recent call last):
File
"/home/immerrr/src/tmp/venv/lib/python3.7/site-packages/airflow/plugins_manager.py",
line 142, in <module>
m = imp.load_source(namespace, filepath)
File "/home/immerrr/src/pyenv/versions/3.7.4/lib/python3.7/imp.py", line 171,
in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/immerrr/src/tmp/airflow/plugins/myplugin.py", line 1, in <module>
from mycode import MyPlugin
ImportError: cannot import name 'MyPlugin' from 'mycode'
(/home/immerrr/src/tmp/mycode.py)
[2019-07-20 13:28:46,341] {plugins_manager.py:149} ERROR - Failed to import
plugin /home/immerrr/src/tmp/airflow/plugins/myplugin.py
{code}
It works OK if you import stuff going from airflow side of things, i.e. import
things from `$AIRFLOW_HOME/plugins` first.
Similar thing happens when you build custom operators: {{from airflow.models
import BaseOperator}} import creates a circular dependency via {{myoperator.py
-> airflow.models -> airflow/__init__.py -> operators._integrate_plugins() ->
airflow.plugins_manager -> }}{{airflow/plugins/myplugin.py ->}}{{ mycode.py ->
myoperator.py}}
> New entry_point load causes circular dependency
> -----------------------------------------------
>
> Key: AIRFLOW-4982
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4982
> Project: Apache Airflow
> Issue Type: Bug
> Components: plugins
> Affects Versions: 1.10.3
> Reporter: Ryan Pennell
> Priority: Minor
>
> I believe there is a bug with the current method used to load airflow plugins
> with setuptools entry_points.
> When creating a plugin, the current recommended method is (in myplugin.py):
> {code:python}
> from airflow.plugin_manager import AirflowPlugin
> class MyCustomPlugin(AirflowPlugin):
> name='myplugin'
> hooks = []
> {code}
> Also in plugins manager (~line 149) is the code to load the entry point.
> Unless I'm missing something, doesn't this lead to a circular load? If I try
> to use import airflow.operators.myplugin, the entry point will be loaded. So
> the entry point loads myplugin.py, which imports airflow.plugin_manager
> because it needs to inherit from AirflowPlugin. But when myplugin.py imports
> plugin_manager, it tries to load the entry points again. An because it
> hasn't completed the import of AirflowPlugin, the class MyCustomPlugin does
> not exists yet, so the import fails.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
