potiuk commented on a change in pull request #6596: [AIRFLOW-6004] Untangle
Executors class to avoid cyclic imports
URL: https://github.com/apache/airflow/pull/6596#discussion_r352307296
##########
File path: airflow/kubernetes/pod_generator.py
##########
@@ -26,35 +26,7 @@
import kubernetes.client.models as k8s
-from airflow.executors import Executors
-
-
-class PodDefaults:
- """
- Static defaults for the PodGenerator
- """
- XCOM_MOUNT_PATH = '/airflow/xcom'
- SIDECAR_CONTAINER_NAME = 'airflow-xcom-sidecar'
- XCOM_CMD = 'trap "exit 0" INT; while true; do sleep 30; done;'
- VOLUME_MOUNT = k8s.V1VolumeMount(
- name='xcom',
- mount_path=XCOM_MOUNT_PATH
- )
- VOLUME = k8s.V1Volume(
- name='xcom',
- empty_dir=k8s.V1EmptyDirVolumeSource()
- )
- SIDECAR_CONTAINER = k8s.V1Container(
- name=SIDECAR_CONTAINER_NAME,
- command=['sh', '-c', XCOM_CMD],
- image='alpine',
- volume_mounts=[VOLUME_MOUNT],
- resources=k8s.V1ResourceRequirements(
- requests={
- "cpu": "1m",
- }
- ),
- )
+from airflow.kubernetes.pod_defaults import PodDefaults
Review comment:
The problem here was more than PodDefaults. The podDefaults had originally
dependency on Executors
```
from airflow.executors import Executors
```
And that was the real problem. It is very often problem with dependencies
that you create some common code (defaults) that you use internally in your
class (Pod Generator). Then you create another class to use those common
settings (from PodGenerator) and then in your PodGenerator you refer the other
class that uses defaults (PodLauncher in this case) and PodLauncher uses
PodGenerator eventually - causing cyclic dependency..
This is classic 'cyclic dependency' problem. And the way to solve it is to
move the common code to a separate class (PodDefaults) that can be used from
two places (PodGenerator and PodLauncher. This means that both PodGenerator
and PodLauncher can use PodDefaults without depending on each other.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
