Hi Jarek, Agreed on 2a and on rejecting group-level cycles in 3.5.
On disallowing dependencies out of a TaskGroup except its exit, -1 from me. Tasks in the middle of a group feeding tasks outside it is a normal pattern, not a bug, and restricting it would need a broad discussion of its own. Let's keep this thread to cycles between TaskGroups. Thanks, Dheeraj On Fri, Sep 25, 2026 at 1:36 PM Pierre Jeambrun <[email protected]> wrote: > I’m also +1 on 2a) > > On Thu 24 Sep 2026 at 23:46, Damian Shaw <[email protected]> > wrote: > > > Anecdotally as a user who was impacted by this, it was pretty annoying > and > > had it been intentional it would have been very frustrating to not have a > > deprecation notice. > > > > That said, it was also trivial to fix (with LLM assistance) as, for me, > it > > was simply caused by implicit edges that had to be made explicit. > > > > I'm a big +1 for warning users before upgrading, and let users know what > > if they call "check_cycle(dag)" in their CI for exceptions, they should > > also call something like "group.topological_sort() for group in > > dag.task_group.get_task_group_dict().values()" in CI for exceptions. > > > > Damian > > > > -----Original Message----- > > From: Jarek Potiuk <[email protected]> > > Sent: Thursday, September 24, 2026 5:16 PM > > To: [email protected] > > Subject: Re: [DISCUSS] Handling cyclic TaskGroup dependencies (Grid/Graph > > 500 since 3.3.1) > > > > I also lean towards 2a) - with rejection in 3.5 - especially that task > > group loop behaviour is not going to land in 3.4 it seems. > > > > And my proposals to answer the questions: > > > > > > 1. Should Dags with group-level cycles be allowed at all, long term? > > > > Generally - I think no. Task Group - when loops are implemented, will > > invalidate all those kinds of dependencies. In fact **any** dependency > for > > tasks in Task Group going out (except the "exit" of the TaskGroup) should > > be deprecated in 3.4 and removed in 3.5. > > However - I think we should survey or query a large number of users' Dags > > to see the potential impact. And we should revisit this in case we find > > that it will break a lot of Dags. > > One option if we find this is a "common" scenario will be to add a Dag or > > "per airflow instance" option to either allow this or Task Group loops. > > > > 2. If we decide to disallow them, how should we roll that out? Some > > options: > > a) Fix the Grid/Graph 500 in 3.4.x and emit a deprecation / Dag > warning > > at parse time. Reject in 3.5. > > > > I would also treat it as a bug that it was allowed before. I don't think > > we ever intended such "task groups" behaviours. > > > > 3. If we decide to allow them, is a fallback ordering in Grid/Graph (as > in > > #72822) acceptable? How should the planned TaskGroup features handle > these > > Dags? > > > > As above. Ideally not allowing is "cleanest" but we should consider > > migration pain if we have some indication this pattern is common. And > > either ease migration or have a flag to disable Task Group loops. > > > > 4. Communication: should we add a known-issue note to the 3.3.1/3.3.2 > > release notes, and publish upgrade guidance on how to restructure > affected > > Dags? > > > > I would even say that this might be a reason to have 3.3.3 and fix it > > there (and add deprecation there as well) - if we decide to re-enable it > > (with > > deprecation) in 3.4.0. > > > > J. > > > > On Thu, Sep 24, 2026 at 8:42 PM Blain David <[email protected]> > > wrote: > > > > > Hi Dheeraj, > > > > > > Thanks for bringing this to the list, I remember this issue being > > > mentioned in the previous devcall ;-) > > > > > > I'd lean towards option 2a. > > > > > > If TaskGroups were only a visual grouping, I'd say the fix belongs in > > > the UI alone: the task graph is a valid DAG, so Grid and Graph should > > > just tolerate the group-level cycle and pick a fallback order, as > #72822 > > does. > > > Rendering an imperfect order is better than a 500 on a Dag that runs > > fine. > > > > > > But from earlier discussions my understanding is that TaskGroups are > > > meant to become part of the execution logic (loops, dynamic groups, > > > clearing or retrying a group, waiting on a group) in the future? Those > > > features need an unambiguous group order, so I agree these Dags should > > > eventually be rejected. > > > That doesn't have to be a parse error right away though. I'd propose: > > > > > > 1. > > > Short term (3.3.x / 3.4.x): fix the 500 in Grid/Graph with a fallback > > > ordering, and raise a Dag warning at parse time naming the TaskGroups > > > involved and stating that group-level cycles are deprecated. A Dag > > > warning is visible in the UI, which is where affected users will > > > actually notice it. Their Dags keep parsing and running. > > > 2. > > > Later: turn the warning into an import error. Given our deprecation > > > policy that's a breaking change, so realistically 4.0 rather than a > > > 3.x minor, but this is just speculation and thinking out loud. In the > > > meantime the planned TaskGroup features can refuse to operate on a > > > group that is part of a cycle, without affecting plain scheduling. > > > > > > Just my thoughts. > > > > > > Kind regards, > > > David > > > > > > ________________________________ > > > From: Dheeraj Turaga <[email protected]> > > > Sent: Thursday, September 24, 2026 20:00 > > > To: [email protected] <[email protected]> > > > Subject: [DISCUSS] Handling cyclic TaskGroup dependencies (Grid/Graph > > > 500 since 3.3.1) > > > > > > [You don't often get email from [email protected]. Learn why this is > > > important at https://aka.ms/LearnAboutSenderIdentification ] > > > > > > EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze > > > niet vertrouwt, klik niet op een link of open geen bijlages. Bij > > > twijfel, stuur deze e-mail als bijlage naar [email protected]<mailto: > > > [email protected]>. > > > > > > Hi all, > > > > > > I'd like to start a discussion about a regression introduced in > > > Airflow 3.3.1. The fix we choose will be a breaking change for some > > > users either way. > > > I'm affected by this on my own Dags too, so I'd like us to agree on a > > > direction before anything lands. > > > > > > Tracking issue: > > > https://github.com/apache/airflow/issues/73678 > > > <https://github.com/apache/airflow/issues/73678> > > > > > > WHAT HAPPENED > > > > > > Since 3.3.1, the Grid and Graph views return HTTP 500 for Dags whose > > > TaskGroup dependencies form a cycle when each group is treated as a > > > single unit. > > > At the task level these Dags are acyclic, and they parse, schedule and > > > run normally. > > > > > > A minimal example: > > > > > > with TaskGroup("group1"): > > > a1 = EmptyOperator(task_id="a1") > > > a2 = EmptyOperator(task_id="a2") > > > with TaskGroup("group2"): > > > b1 = EmptyOperator(task_id="b1") > > > b2 = EmptyOperator(task_id="b2") > > > > > > a1 >> b1 # group1 -> group2 > > > b2 >> a2 # group2 -> group1 > > > > > > A simpler variant also triggers it: two tasks in the same group > > > connected through a task outside the group (a >> ext >> b). > > > > > > This is common when TaskGroups are used for logical or visual grouping > > > (for example one group per database schema). Users have hit it when > > > upgrading from 2.x to 3.3.1. > > > > > > The cause is a bug fix. Grid/Graph used to ignore group-to-group and > > > cross-group edges when ordering TaskGroups, and silently rendered an > > > arbitrary order. The fix made the sort account for those edges, so > > > group-level cycles now raise an error: > > > - > > > https://github.com/apache/airflow/pull/69933 > > > <https://github.com/apache/airflow/pull/69933> > > > - > > > https://github.com/apache/airflow/pull/70591 > > > <https://github.com/apache/airflow/pull/70591> (backport, released in > > > 3.3.1) > > > > > > APPROACHES SO FAR > > > > > > - > > > https://github.com/apache/airflow/pull/72822 > > > <https://github.com/apache/airflow/pull/72822> (closed): keep these > > > Dags valid and make Grid/Graph fall back to a > > > strongly-connected-component ordering. > > > - > > > https://github.com/apache/airflow/pull/73087 > > > <https://github.com/apache/airflow/pull/73087> (open): reject these > > > Dags at parse time with an import error that names the TaskGroups > > involved. > > > > > > The arguments on each side, from the PR discussions: > > > > > > - For treating them as invalid: planned TaskGroup features such as > > > task loops, dynamic TaskGroups, retrying or clearing a TaskGroup, and > > > waiting for a TaskGroup to complete need an unambiguous group > > > ordering. They would have undefined behaviour on these Dags. > > > - For allowing them: the task graph itself is valid. Many users use > > > TaskGroups only for grouping. Rejecting these Dags in a patch release > > > would turn a UI-only problem into Dags failing to load, which > > > conflicts with our deprecation policy ( > > > > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process. > > > html#deprecation-policy > > > < > > > https://airflow.apache.org/docs/apache-airflow/stable/release-process. > > > html#deprecation-policy > > > > > > > ). > > > > > > QUESTIONS > > > > > > 1. Should Dags with group-level cycles be allowed at all, long term? > > > > > > 2. If we decide to disallow them, how should we roll that out? Some > > > options: > > > a) Fix the Grid/Graph 500 in 3.4.x and emit a deprecation / Dag > > > warning at parse time. Reject in a later minor or major release. > > > b) Deprecate it starting 3.4.0 > > > > > > 3. If we decide to allow them, is a fallback ordering in Grid/Graph > > > (as in > > > #72822) acceptable? How should the planned TaskGroup features handle > > > these Dags? > > > > > > 4. Communication: should we add a known-issue note to the 3.3.1/3.3.2 > > > release notes, and publish upgrade guidance on how to restructure > > > affected Dags? > > > > > > > > > Thanks, > > > Dheeraj > > > > > > General (Internal Property) > > > > > ________________________________ > > Strike Technologies, LLC (“Strike”) is part of the GTS corporate family. > > Strike is a technology solutions provider, and is not a broker or dealer > > and does not engage in securities transactions. This communication does > not > > constitute an offer to sell or the solicitation of an offer to buy any > > security in any jurisdiction.. > > ________________________________ > > > > CONFIDENTIALITY / PRIVILEGE NOTICE: This communication and any > attachments > > are intended solely for the addressee. The information contained in this > > communication may be proprietary, privileged, confidential, and/or > > protected from unauthorized use or disclosure under applicable law. If > you > > are not the intended recipient, you are hereby notified that any review, > > dissemination, distribution, copying, or other use or disclosure of this > > information is strictly prohibited, and may be unlawful. If you have > > received this transmission in error, please delete this message and all > > copies from your system and notify the sender via return transmittal. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > >
