Jarek Potiuk created AIRFLOW-6003:
-------------------------------------
Summary: Cyclic imports detected by pylint randomly
Key: AIRFLOW-6003
URL: https://issues.apache.org/jira/browse/AIRFLOW-6003
Project: Apache Airflow
Issue Type: Bug
Components: ci
Affects Versions: 2.0.0
Reporter: Jarek Potiuk
There are cyclic imports detected seemingly randomly by pylint checks when some
of the PRs are run in CI:
An example of such pylint problem:
[https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification&utm_source=github_status]
************* Module airflow.utils.log.json_formatter
1492airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_generator) (cyclic-import)
1493airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow ->
airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_launcher) (cyclic-import)
1494airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator)
(cyclic-import)
The problem is that airflow's __init__ contains a few convenience imports
(AirflowException, Executors etc.) but it also imports a number of packages
(for example kubernetes_executor) that in turn import the airflow package
objects - for example airflow.Executor. This leads to cyclic imports if you
import first the executors before airflow. Similar problem happens with
executor.__init__.py containing class "Executors" imported by all executors but
at the same time some of the executors (for example KubernetesExecutor) import
the very same Executor class.
This might happen in pylint checks in pre-commit because they split a number of
files they process between the multiple threads you have at your machine and
sometimes it might happen that the files are imported in different order.
As a solution, the executors "list" should be moved to a separate module and a
number of classes (such as AirflowException) should be moved out from __init__
package and imported from where they belong (so for example AirflowException
should be universally imported from airflow.exceptions).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
