[
https://issues.apache.org/jira/browse/AIRFLOW-860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009264#comment-16009264
]
ASF subversion and git services commented on AIRFLOW-860:
---------------------------------------------------------
Commit 08a784ede6ffe1a2389255e1c6d597e9b2131080 in incubator-airflow's branch
refs/heads/master from [~stverhae]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=08a784e ]
[AIRFLOW-860][AIRFLOW-935] Fix plugin executor import cycle and executor
selection
When a plugin is made with a custom Operator and
executor, an import cycle occurs when the executor
is chosen in airflow.cfg because the
executors/__init__.py starts loading plugins too
early.
changed DEFAULT_EXECUTOR use to a function call
which returns the default executor. This lazy
approach fixes the import cycle.
revision eb5982d (included in 1.8) breaks plugin
executors altogether. It makes a new module for
every plugin, so import statements need to be
adapted, but the executor selection is left
unchanged, so it ends up assigning the plugin
module as an executor.
fixed executor selection to work with the new
plugin modules system introduced in 1.8. in
Airflow.cfg a executor can now be specified as
{plugin_name}.{executor_name}
Fixes:
-
https://issues.apache.org/jira/browse/AIRFLOW-860
-
https://issues.apache.org/jira/browse/AIRFLOW-935
Closes #2120 from stverhae/master
> Circular module dependency prevents loading of custom executor plugin
> ---------------------------------------------------------------------
>
> Key: AIRFLOW-860
> URL: https://issues.apache.org/jira/browse/AIRFLOW-860
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor, plugins
> Affects Versions: Airflow 1.8
> Environment: Linux RHEL 7.6, Python 2.7.13
> Reporter: Nick Ivanov
>
> Custom plugins cannot be loaded, which prevents airflow from running, due to
> apparent cyclic dependency in plugins_manager called in
> {{executors.\_\_init\_\_}} -- the top-level {{\_\_init\_\_}} attempts to load
> the default executor, which then goes back to plugins_manager etc.
> {noformat}
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> line 17, in <module>
> from airflow import configuration
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
> line 31, in <module>
> from airflow.models import DAG
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
> line 62, in <module>
> from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
> line 55, in <module>
> raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor TestExecutor not supported.
> {noformat}
> While attempting {{from airflow.plugins_manager import executors_modules}}
> cycles right back where it came from:
> {noformat}
> >>> from airflow.plugins_manager import executor_modules
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
> line 31, in <module>
> from airflow.models import DAG
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
> line 62, in <module>
> from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
> line 55, in <module>
> raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor LSFExecutor not supported.
> {noformat}
> To reproduce:
> # Create a custom executor, e.g. by copying {{local_executor.py}} into
> {{$AIRFLOW_HOME/plugins/test_executor.py}} and replacing occurrences of
> "Local" with "Test" in the code, which will produce {{TestExecutor}}
> # Update {{$AIRFLOW_HOME/airflow.cfg}} by setting in the {{\[core\]}}
> section: {{executor = TestExecutor}}
> # Run any airflow command, e.g. "airflow version"
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
