Thanks for that list @[email protected], could you create a GH issue for the following please:
- Commands like airflow dags list and list-import-errors now rely on the DB, so users can’t easily validate DAGs without running a processor or standalone mode. We built a On Wed, 15 Oct 2025 at 07:27, Amogh Desai <[email protected]> wrote: > Huge +1 from me too. This will significantly improve the quality of the > release > and give more confidence to enterprises trying to upgrade. > > Thanks for sharing your insights, Dheeraj. Feel free to create issues / fix > the > problems that caused onboarding challenges for you. Glad to see your > migration story, good job. > > About de-siloing, yes, we should spend some time on Github issues so that > there > are no SPOF's per project. I would encourage contributors to comment on the > issues > they would like to work on and some of us can guide / provide the right > direction and > references for folks to get the knowledge they are looking for. > > > Thanks & Regards, > Amogh Desai > > > On Wed, Oct 15, 2025 at 5:10 AM Ferruzzi, Dennis <[email protected]> > wrote: > > > Great to see everyone last week. Big +1 on this from me as well. 3.0 > and > > 3.1 were pretty hectic with folks making big changes real fast. We'd > > benefit greatly from some polishing and cleaning with the caveats which > > Jens and Rajesh pointed out for features that didn't get fully > implemented > > before the 3.1 cutoff. > > > > Also big +1 on Jarek's comments about us (I suppose I can only speak for > > myself really, but I suspect it's not just me) needing to de-silo a > bit. I > > know I was focused pretty intently on a small chunk of the code while > > others were making big changes and I need to peek my head up a bit to > get a > > better understanding of what all those changes are/were. TaskSDK comes > to > > mind as a particular blind spot for me, for example; a big sweeping > change > > that I was a bit to overwhelmed to keep track of while also trying to > > deliver the Deadlines work. > > > > > > > > > > - ferruzzi > > > > > > ________________________________ > > From: Dheeraj Turaga <[email protected]> > > Sent: Sunday, October 12, 2025 8:22 PM > > To: [email protected] > > Subject: RE: [EXT] [DISCUSS]: Proposal to focus on polishing Airflow 3 > > > > CAUTION: This email originated from outside of the organization. Do not > > click links or open attachments unless you can confirm the sender and > know > > the content is safe. > > > > > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > > le contenu ne présente aucun risque. > > > > > > > > As promised during the Airflow Summit, I’m sharing our experience > migrating > > from Airflow 2.10 to 3.1 and overall, we’re thrilled with the upgrade. > > EdgeExecutor and Triggering User Name were the main drivers for > us—they’ve > > solved big pain points in our workflows. Thank you for making these > > features available! > > > > How We Migrated > > > > We spun up a new Airflow 3 instance alongside Airflow 2, cleaned up DAG > > parsing issues, ran UAT, and then retired Airflow 2 during a maintenance > > window. > > > > Why We Upgraded > > > > - EdgeExecutor: Critical for running tasks across multiple data > centers. > > - Triggering User Name: Huge for us—most runs are manual, and knowing > > who triggered a DAG is essential. In Airflow 2, we had to duplicate > DAGs > > per user, which was messy and resource-heavy. > > > > Migration Challenges > > > > - Almost all DAGs needed updates. Import changes were easy, but I > wasn’t > > aware of Ruff helper scripts at the time and migrated manually. > > - Lack of user-facing DAG debug tools was tough. Commands like airflow > > dags list and list-import-errors now rely on the DB, so users can’t > > easily > > validate DAGs without running a processor or standalone mode. We > built a > > custom CI script using DagBag, but it broke during 3.0.6–3.1 due to > > class > > changes. > > - No way to estimate DAG parse time without running the dag > > processor—large dynamic DAGs (1000+ tasks) caused bottlenecks. > > > > Issues Observed > > > > - Retry-delay bug: Scheduler crashed during 3.0.6 → 3.1 migration; > > required manual patch. > > - Grid View: Very slow for large DAGs (2000+ tasks); improved in 3.1 > but > > still slower than 2.10. > > - Mobile UI: DAG page, logs, clear/try actions broken in Airflow 3; > > critical for global teams working odd hours. > > - Task Ordering: Broken in 3.1; I have an open PR to fix this. > > - Graph View: Unusable for large DAGs with task groups; tasks > scattered > > infinitely (fixed in 3.0.6). > > - Rendered Template: Formatting issues in 3.0; fixed in 3.1. > > - Logs: > > - Scrolling large logs is slow; 3.1 added “jump to end” but still > > sluggish. > > - Minified React issue on overview page with large logs (fixed in > > 3.0.6). > > - Missing option to remove “source” line (fixed in 3.1). > > > > Suggestions for Improvement > > > > - DAG Validation Tools: Provide user-facing utilities to check import > > errors and estimate parse time without running a processor. > > - Mobile UI: Restore functionality for DAG actions and log viewing. > > - Grid View Performance: Optimize for large DAGs with thousands of > > tasks. > > - Log Viewer: Improve scrolling speed and usability for large logs. > > - Git DAG Bundles: Add robust retry logic for connectivity issues; > bare > > clone corruption currently breaks all tasks. > > - Mapped Index & Rendered Templates: Show mapped index and rendered > name > > before task runs; render even if task fails. > > - Search Capabilities: Add more filters for DAGRun and Task Instances > > (e.g., hostname) to help diagnose host-specific failures. > > - API Server Metrics: Expose metrics for API request load to help plan > > horizontal scaling. > > > > > > > > Airflow 3 is a big step forward for us—these suggestions are aimed at > > making it even better for large-scale deployments. Thanks for all the > hard > > work! > > > > > > > > Cheers, > > Dheeraj Turaga > > > > On Mon, Oct 6, 2025 at 3:11 PM Kaxil Naik <[email protected]> wrote: > > > > > That's awesome, let's catchup Dheeraj - looking forward to hearing > about > > > your migration experience > > > > > > On Mon, 6 Oct 2025 at 10:59, Dheeraj Turaga <[email protected]> > > > wrote: > > > > > > > +1 I strongly agree, as someone who migrated our org to 3.1 last week > > > (and > > > > initial adopters for 3.0 aswell) , I have been pushing bug fixes as I > > see > > > > and as my user base reports them. I felt the migration was bumpy and > > > have > > > > some notes regarding this > > > > > > > > Looking forward to share my experience with you all at the summit! > > > > > > > > On Mon, Oct 6, 2025 at 7:53 PM Pavankumar Gopidesu < > > > > [email protected]> > > > > wrote: > > > > > > > > > Yeah very strong +1 lets make next release super stable version. > > > > > > > > > > Regards > > > > > Pavan > > > > > > > > > > On Mon, 6 Oct 2025 at 18:36, Kaxil Naik <[email protected]> > wrote: > > > > > > > > > > > Strong +1, thanks Vikram for the proposal. > > > > > > > > > > > > Dedicated time for this is essential. > > > > > > > > > > > > On Mon, 6 Oct 2025 at 01:22, Aritra Basu < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > > > Very valid points from both of you! I am all in on this as > well, > > I > > > > > think > > > > > > > it's been all cylinders firing for a little while now with > making > > > > > > airflow 3 > > > > > > > feature rich. Taking some time to clean up the features would > be > > > > great! > > > > > > > > > > > > > > -- > > > > > > > Regards, > > > > > > > Aritra Basu > > > > > > > > > > > > > > On Mon, 6 Oct 2025, 10:23 am Jarek Potiuk, <[email protected]> > > > wrote: > > > > > > > > > > > > > > > I am very much for it! > > > > > > > > > > > > > > > > And I would even add a little - it would be great that pretty > > > much > > > > > all > > > > > > of > > > > > > > > us got involved - even (and especially) in the areas that > they > > > have > > > > > not > > > > > > > > been involved so far. > > > > > > > > > > > > > > > > There are a number of areas - both new, and "changed old" > that > > I > > > > > think > > > > > > > > there is a small number of "experts" (basically those who > > worked > > > on > > > > > > it) - > > > > > > > > but others have limited visibility of understanding of a) new > > > areas > > > > > b) > > > > > > > > scope of changes (I am speaking from my own experience here > as > > > > well). > > > > > > And > > > > > > > > we are somewhat shying away not even in attempting to fix > > things, > > > > but > > > > > > > also > > > > > > > > even in triaging and responding and interacting with users > who > > > are > > > > > > > raising > > > > > > > > issues. Thus many issues are untriaged. I think we got a bit > > more > > > > > > > "siloed" > > > > > > > > in our part with Airflow 3 development and we need to break > the > > > > > silos a > > > > > > > > bit. > > > > > > > > > > > > > > > > There might be few reasons: > > > > > > > > > > > > > > > > - we feel not competent enough to help > > > > > > > > - somehow we feel "the others who implemented it" are > > responsible > > > > for > > > > > > > > fixing those > > > > > > > > - we have "our" parts that we are looking at and focusing on > > > (this > > > > > is I > > > > > > > > think the biggest part especially for those "experts" who > might > > > > feel > > > > > > > > overwhelmed - if we look elsewhere, we might have a feeling > > that > > > > > "our" > > > > > > > > part will be lagging behind) > > > > > > > > - those "experts" on the other hand might feel overloaded > with > > a > > > > > number > > > > > > > of > > > > > > > > issues in their specific area and have hard time in getting > > > someone > > > > > to > > > > > > > help > > > > > > > > them > > > > > > > > > > > > > > > > I think ideally, we need more of the community engagement > here > > - > > > > and > > > > > > > likely > > > > > > > > "experts" taking more of a role of brainstorming and guiding > > > other > > > > > > > > contributors, committers, PMC members to help following their > > > > advice > > > > > > and > > > > > > > > oversight in solving the issues. That would not only be > > > opportunity > > > > > to > > > > > > > efix > > > > > > > > things potentially faster (after initial ramp-up time) but > also > > > > turn > > > > > > such > > > > > > > > "polishing" period into a knowledge transfer. Ultimately it's > > not > > > > one > > > > > > or > > > > > > > > two person who is responsible for some "areas" in Airflow, > but > > > > whole > > > > > > > > community is. And those "experts" might even find time to > help > > in > > > > > > "other" > > > > > > > > areas if they are less burdened with working on solutions > down > > > to a > > > > > > green > > > > > > > > PR in their area of expertise. > > > > > > > > > > > > > > > > And also I think that "help" thing comes to the users who > > raised > > > > > their > > > > > > > > issues (some of them undoubtedly listening here) - we will > need > > > > their > > > > > > > help > > > > > > > > in at least testing solutions and commenting on hypotheses. > > > > > > > > > > > > > > > > Maybe we can figure out a way of working (commenting on > issues, > > > > > > triaging > > > > > > > > approach, issue solving attempt, way of asking for help)? > that > > > will > > > > > > > > "catalyse" such knowledge transfer. > > > > > > > > > > > > > > > > But I also might be wrong in my assesment - so I'd love to > hear > > > > what > > > > > > > others > > > > > > > > might say here - maybe also have some proposals how we could > > > > > reorganise > > > > > > > to > > > > > > > > handle open issues better (and to handle some of the > challenges > > > > > > > involved). > > > > > > > > Undoubtedly such knowledge transfer has some risks that > solving > > > > > issues > > > > > > > will > > > > > > > > slow down - at least initially, so we have to be rather > careful > > > > with > > > > > > this > > > > > > > > approach and have clear boundary of trust from the experts > that > > > > > things > > > > > > > will > > > > > > > > be solved when they are guiding somoene. > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Oct 5, 2025 at 8:13 PM Vikram Koka via dev < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Dear Airflowers, > > > > > > > > > > > > > > > > > > I am looking forward to meeting many of you this coming > week > > at > > > > the > > > > > > > > Airflow > > > > > > > > > Summit. It will be wonderful to connect in person after a > > year > > > of > > > > > > > online > > > > > > > > > collaboration since the last Summit. > > > > > > > > > > > > > > > > > > I’d like to put a proposal in front of all of you. We’re > sure > > > to > > > > > hear > > > > > > > > > valuable feedback from users who have adopted or are > adopting > > > > > Airflow > > > > > > > 3. > > > > > > > > My > > > > > > > > > proposal is that we dedicate October, the four weeks > > following > > > > the > > > > > > > > Summit, > > > > > > > > > to polishing work rather than new feature development. > > > > > > > > > > > > > > > > > > This would mean focusing on smoothing out any rough edges > in > > > the > > > > > > > adoption > > > > > > > > > journey and making it easier for users to take full > advantage > > > of > > > > > the > > > > > > > new > > > > > > > > > capabilities we’ve released. Depending on the aggregated > > > > feedback, > > > > > we > > > > > > > can > > > > > > > > > also consider multiple patch releases during this period to > > > > quickly > > > > > > > > > incorporate improvements. > > > > > > > > > > > > > > > > > > As part of this, let's make sure feedback is easy to track: > > > > > > > > > > > > > > > > > > - System of record: Use Github issues > > > > > > > > > <https://github.com/apache/airflow/issues> as the > source > > of > > > > > > truth, > > > > > > > > even > > > > > > > > > if there is a conversation over slack or on the dev > list. > > > > > > > > > - Version labelling: Include the Airflow version so it > can > > > be > > > > > > > labeled > > > > > > > > > appropriated (label:affected_version either 3.0 or 3.1), > > > > easily > > > > > > > > > reproduced > > > > > > > > > and resolved. > > > > > > > > > - Upgrade blockers: Indicate if this affects upgrades > from > > > > 2.x. > > > > > We > > > > > > > > have > > > > > > > > > been labeling and tracking these separately. > > > > > > > > > - Documentation vs. code: Indicate if this is a > > > documentation > > > > > gap, > > > > > > > > > rather than a code problem. > > > > > > > > > - Context: Airflow's flexibility allows for a wide range > > of > > > > > > > behavior. > > > > > > > > > With Airflow 3's architectural changes, especially the > new > > > > > TaskSDK > > > > > > > > > model, > > > > > > > > > some implicit behaviors may now need to be explicitly > > > > specified. > > > > > > If > > > > > > > > you > > > > > > > > > found anything confusing or frustrating, please let us > > know > > > > if a > > > > > > > > > documentation update, upgrade script change, or a > > clarifying > > > > > > example > > > > > > > > > would > > > > > > > > > be helpful. > > > > > > > > > > > > > > > > > > We are looking for active participation from everyone, > > > including > > > > > > those > > > > > > > > who > > > > > > > > > haven't contributed before. Even a small contribution such > > as a > > > > > clear > > > > > > > > > reproduction scenario, a documentation improvement, or a > > simple > > > > > > upgrade > > > > > > > > > script update can make a big difference. > > > > > > > > > > > > > > > > > > Thank you and best regards, > > > > > > > > > Vikram > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Vikram Koka > > > > > > > > > Chief Strategy Officer > > > > > > > > > Email: [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > <https://www.astronomer.io/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
