potiuk commented on a change in pull request #6596: [AIRFLOW-6004] Untangle
Executors class to avoid cyclic imports
URL: https://github.com/apache/airflow/pull/6596#discussion_r351280476
##########
File path: airflow/models/dagbag.py
##########
@@ -317,9 +317,7 @@ def kill_zombies(self, zombies, session=None):
had a heartbeat for too long, in the current DagBag.
:param zombies: zombie task instances to kill.
- :type zombies: airflow.utils.dag_processing.SimpleTaskInstance
Review comment:
That's what I tried to do, but when you start doing this, you uncover a
whole host of further cyclic dependencies, which will take quite some time to
untangle (like DagBag using TaskInstances which are using DAGs which are using
... DagBag).
When you define those types in return or parameter values, you no longer can
use local imports because the types become part of the public API of the class.
And pretty much hell breaks loose. Some of these changes will require some more
"internal architecture" changes - moving methods between tasks, separating out
constants which are shared with logic, joining several separate but tightly
coupled classes in single module because they currently refer each other. But
doing it in this PR is not a good idea.
I already have a WIP where I added more types
(https://github.com/PolideaInternal/airflow/tree/WIP-more-types) but this has
so many changes that I decided to use it as a base to introduce more types
later. There are a number of small things that we can do to untangle all of
that (but I prefer to have steady stream of separate PRs for that) for example
in order to avoid some cyclic dependencies between BaseOperator and
TaskInstance we will have to move clear()/clear_task_instances() from
BaseOperator (this actually makes sense and can be safely done but it's quite a
change on it's own)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
