jedcunningham commented on code in PR #40696:
URL: https://github.com/apache/airflow/pull/40696#discussion_r1672574284
##########
airflow/models/dagrun.py:
##########
@@ -948,9 +948,12 @@ def _filter_tis_and_exclude_removed(dag: DAG, tis:
list[TI]) -> Iterable[TI]:
tis = list(_filter_tis_and_exclude_removed(self.get_dag(), tis))
- unfinished_tis = [t for t in tis if t.state in State.unfinished]
+ unfinished_tis = [t for t in tis if t.state in State.unfinished if not
t.blocked_by_upstream]
finished_tis = [t for t in tis if t.state in State.finished]
- if unfinished_tis:
+ blocked_tis = [t for t in tis if t.blocked_by_upstream]
Review Comment:
Should this only be unfinished blocked tis (as in, with `if t.state in
State.unfinished` in there)?
##########
airflow/models/dagrun.py:
##########
@@ -966,6 +969,14 @@ def _filter_tis_and_exclude_removed(dag: DAG, tis:
list[TI]) -> Iterable[TI]:
new_unfinished_tis = [t for t in unfinished_tis if t.state in
State.unfinished]
finished_tis.extend(t for t in unfinished_tis if t.state in
State.finished)
unfinished_tis = new_unfinished_tis
+
+ return schedulable_tis, changed_tis, unfinished_tis
+
+ if unfinished_tis:
+ schedulable_tis, changed_tis, unfinished_tis =
_decide_on_tis(unfinished_tis)
+ elif not unfinished_tis and finished_tis:
+ schedulable_tis, changed_tis, unfinished_tis =
_decide_on_tis(blocked_tis)
+ unfinished_tis = [t for t in unfinished_tis if t.state in
State.unfinished]
Review Comment:
I'm a little unsure about this approach.
- I think we can simplify this and include blocked tasks if there are no
unfinished/non-blocked tasks, instead of needing a helper function.
- It feels like waiting to unblock tasks until there are no remaining
non-blocked tasks might be inefficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
