Hi all,
In the cyclic TaskGroup thread [1], it was suggested that any dependency
from a task inside a TaskGroup to a task outside it, except from the
group's exit, should be deprecated in 3.4 and removed in 3.5, because
Task Loops would invalidate those dependencies. That is a much bigger
change than group-level cycles, so I'm moving it to its own thread.
My position: new TaskGroup features, such as Task Loops (AIP-111) [2]
and Dynamic Task Groups (AIP-113) [3], should not break existing Dags
that use plain TaskGroups this way. Any boundary rule those features
need should apply only to the groups that opt into them.
WHAT THIS IS ABOUT
A task-level edge that leaves a TaskGroup from a task other than its
exit, or enters it at a task other than its entry:
with TaskGroup("load"):
extract = ...
transform = ...
publish = ...
extract >> transform >> publish
extract >> audit # audit is outside the group
Group-level cycles (a >> ext >> b, with a and b in the same group) are
not part of this. The other thread covers those.
WHY THIS IS A VALID AND COMMON PATTERN
- Our docs have always described TaskGroups as grouping, not execution
semantics. The 2.x docs said: "TaskGroups are purely a UI grouping
concept. All tasks within the TaskGroup still behave as any other
tasks outside of the TaskGroup." [4] That sentence left the 3.x docs
only because the SubDAG section around it was removed (#41390). The
3.x overview still says TaskGroups "let you visually group tasks in
the UI". AIP-34 also says TaskGroups make no change to how the
scheduler works.
- Our best-practices page depends on it. The watcher pattern [5] says
the watcher "needs also to be a downstream task for all other tasks
in the Dag", and the edges have to be direct because trigger rules
only look at direct parents. Any Dag that uses the watcher together
with a TaskGroup has non-exit tasks feeding a task outside the
group. 11 of our own provider system-test Dags do this today.
- We fixed the Graph view for exactly this pattern in June. #67714 [6]
reported a Dag where tasks in the middle of a @task_group also feed
a task outside the group, and noted that the Dag executes correctly.
#67720 [7] fixed the rendering to preserve "the author's explicit
dependency intent".
- It has been a tested case since TaskGroups were introduced.
test_task_group.py has had an edge into a non-entry task of another
group (task5 >> task8) since the original AIP-34 PR (#10153).
- TaskFlow produces it naturally. Passing an outside XComArg into a
@task_group function creates an edge to whichever task consumes it,
not necessarily the group's entry. And @task_group forwards whatever
task the function returns, so downstream tasks bind to that task,
not necessarily the group's last one.
- Third-party guides teach it. Astronomer's dependencies guide says:
"You can also set dependencies between task groups, between tasks
inside and out of task groups, and even between tasks in different
(nested) task groups", with example code that does this [8].
- Users describe their TaskGroups this way. A user on #73087 wrote that
their "task groups are logical groupings, per database schema, with
no bearing on Airflow dependencies" [9].
WHAT THE AIPs ALREADY SAY
Both AIPs are opt-in. AIP-111 (accepted) adds .loop() to a task or
TaskGroup and says no existing Dag is affected. AIP-113 adds a separate
DynamicTaskGroup construct and says "No existing Dag is affected."
Both route downstream edges through a synthesized completion or merge
node. Neither needs plain TaskGroups to change.
PROPOSAL
1. Plain TaskGroups keep today's semantics: any acyclic task-level edge
across a group boundary stays valid. Group-level cycles are handled
in the other thread.
2. A looped group or a DynamicTaskGroup may restrict edges across its
own boundary (for example, only through its entry and its completion
or merge node). That check runs at parse time, only for that group,
and the error names the offending edge.
3. AIP-111 and AIP-113 state their boundary rules explicitly, including
what happens to a direct edge between a loop-body or catalog task
and a task outside the group.
4. If we later want to restrict plain TaskGroups, that is a separate
breaking-change proposal, with data on user impact and a deprecation
period under our policy [10].
Does anyone see a case where loops or dynamic groups need plain
TaskGroups to change?
[1] https://lists.apache.org/thread/sossl7b2w2ftyk4028qrhps2tcdxj2px
[2]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=440303747
[3]
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440304823/AIP-113+Dynamic+Task+Groups
[4]
https://airflow.apache.org/docs/apache-airflow/2.10.5/core-concepts/dags.html
[5]
https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#example-of-watcher-pattern-with-trigger-rules
[6] https://github.com/apache/airflow/issues/67714
[7] https://github.com/apache/airflow/pull/67720
[8] https://www.astronomer.io/docs/learn/managing-dependencies
https://github.com/astronomer/webinar-task-groups/blob/main/dags/example_complex_dependencies_2.py
[9] https://github.com/apache/airflow/pull/73087#issuecomment-5808925697
[10]
https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#deprecation-policy
Thanks,
Dheeraj
