Hi Kaxil and Vikram, Huge thanks for putting this together! It’s a fantastic document and addresses a concept many of us have been dreaming of for a long time. It reads really well and is a very exciting step forward for Airflow.
I’ve shared some detailed comments in the document to help clarify a few points, but here is a quick overview of what I’m thinking: - Task vs. Task Group loops: I’d love to see both included, especially for the potential optimizations Task.loop might offer. - "Fill-to-cap" vs. "on-demand": To keep things manageable, I’d suggest starting with one. "Fill-to-cap" feels like a safer first step, whereas "on-demand" is a bold but higher-risk goal. - Comparing alternatives: It would be great to highlight the specific advantages of .loop() versus things like Deferrable Operators, while loops, or AIP-104. - Edge cases: We might want to dive deeper into handling skipped tasks, loop exhaustion, and how the "condition" execution works. - Naming: A few terms like the "loop" parameter and "gate" task could perhaps be polished to be even clearer. Thanks again for the hard work on this! Best, Jarek On Thu, Jul 9, 2026 at 7:38 AM Kaxil Naik <[email protected]> wrote: > Thanks Vikram. > > Looking forward to feedback from all of you. > > On Tue, 7 Jul 2026 at 08:42, Vikram Koka via dev <[email protected]> > wrote: > > > Dear Airflowers, > > > > I am sending this note out on behalf of Kaxil Naik, who is temporarily > out > > of the office and asked me to start this discussion. > > > > Over the last several months, we have discussed expanding Airflow > > capabilities towards more dynamic graph structures. > > > > Airflow already runs your tasks deterministicly and reliably. Loops let a > > runtime condition decide *how many times* to run a section of the DAG, > such > > as a task or a task group. The dynamism focuses on the loop count and > stop > > condition. Additionally, this is always bounded by a hard cap. > > > > The full draft is on Confluence as AIP-111 Task Loops > > < > > > https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/440303747/AIP-111+Task+Loops > > > > > . > > > > We would love to hear your thoughts. Please comment on the AIP doc > itself. > > > > Vikram > > (on behalf of Kaxil) > > -- > > > > Vikram Koka > > Chief Strategy Officer > > Email: [email protected] > > > > > > <https://www.astronomer.io/> > > >
