Nick Ivanov created AIRFLOW-860:
-----------------------------------
Summary: Circular module dependency prevents loading of custom
executor plugin
Key: AIRFLOW-860
URL: https://issues.apache.org/jira/browse/AIRFLOW-860
Project: Apache Airflow
Issue Type: Bug
Components: executor, plugins
Affects Versions: Airflow 1.8
Environment: Linux RHEL 7.6, Python 2.7.13
Reporter: Nick Ivanov
Custom plugins cannot be loaded, which prevents airflow from running, due to
apparent cyclic dependency in plugins_manager called in
{{executors.\_\_init\_\_}} -- the top-level {{\_\_init\_\_}} attempts to load
the default executor, which then goes back to plugins_manager etc.
{noformat}
File
"/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
line 17, in <module>
from airflow import configuration
File
"/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
line 31, in <module>
from airflow.models import DAG
File
"/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
line 62, in <module>
from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
File
"/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
line 55, in <module>
raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
airflow.exceptions.AirflowException: Executor TestExecutor not supported.
{noformat}
While attempting {{from airflow.plugins_manager import executors_modules}}
cycles right back where it came from:
{noformat}
>>> from airflow.plugins_manager import executor_modules
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File
"/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
line 31, in <module>
from airflow.models import DAG
File
"/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
line 62, in <module>
from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
File
"/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
line 55, in <module>
raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
airflow.exceptions.AirflowException: Executor LSFExecutor not supported.
{noformat}
To reproduce:
# Create a custom executor, e.g. by copying {{local_executor.py}} into
{{$AIRFLOW_HOME/plugins/test_executor.py}} and replacing occurrences of "Local"
with "Test" in the code, which will produce {{TestExecutor}}
# Update {{$AIRFLOW_HOME/airflow.cfg}} by setting in the {{\[core\]}} section:
{{executor = TestExecutor}}
# Run any airflow command, e.g. "airflow version"
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
