Hey everyone,
I updated our meeting notes document in the Airflow wiki to capture the
notes from our dev call yesterday. The link for those notes is here
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-17April2025>
Thank you all for attending the meeting. The discussion was very good, as
evidenced by the long summary and list of action items :) I hope I captured
them all.
To everyone who attended the meeting, please check the summary and add
anything that I may have missed. For those who could not join, please let
us know if you disagree with anything discussed and agreed upon in the
meeting. Also, please do ask questions if something is unclear.
Best regards,
Vikram
--
Below is the summary from the call:
- Development Updates:
- Update on the 3.0.1 release Kaxil Naik
<https://cwiki.apache.org/confluence/display/~kaxilnaik>
- Kaxil mentioned that over 200 commits are part of 3.0.1 RC1
including many usability improvements.
- He said that the voting for RC1 ends on Monday, the 12th of May.
- Jarek added that we would probably need an RC2 because of the
inclusion of an older version of an Amazon provider.
- 2.11 release scope / timing Kaxil Naik
<https://cwiki.apache.org/confluence/display/~kaxilnaik>
- Kaxil mentioned that the decision between 2.11 vs. 2.10.x was
made because of several features and changes in the 2.x
branch, which meant
beyond a patch.
- Specifically, he mentioned the new Timetable addition as well as
the PR for making sure that the metrics are aligned for consistent
reporting.
- Another reason for the feature release was also to add support
for "airflow config lint" which enables validation of the airflow
configuration before upgrading to Airflow 3.
- Discussion Topics:
- 3.0 release feedback / discussion topics (All)
- Vikram shared a quick update on the Airflow downloads by version
over the last two weeks since Airflow 3 was released.
- Vikram highlighted that Airflow 3 has seen significant adoption
with about 7.5% of the total Airflow downloads. The highest downloaded
version continues to be 2.10, which is the last release on
the 2.x release
stream, with about a third of all downloads. Of course, these metrics
should be taken with the usual grain of salt, because they
are susceptible
to image caching and CI processes.
- Elad asked where features "missing" in Airflow 3 were
documented. Vikram said that these were documented as breaking changes
within the Upgrading to Airflow 3 doc
<https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html#breaking-changes>.
The team identified a couple of missing elements, which Elad
volunteered to
add.
- Jarek said that there was significant user confusion around
upgrading to 3.0 with respect to DAG writing practices,
specifically around
imports. After much discussion, the conclusion was that one
of the core
problems here was the UX around doc versions on the website, being too
subtle and not obvious to users about the differences between
2.x and 3.0
syntax. The action item for Jarek and Elad was to look at the
docs UX to
see if this could be made more obvious.
- 3.1 release scope / timing discussion Vikram Koka
<https://cwiki.apache.org/confluence/display/~vikramkoka>
- In the follow-up discussion about the scope of 3.1, Vikram
explained that the Airflow 3.1 scope was largely based on
user feedback of
3.0 including the prior discussion about missing features.
- The early draft of Airflow 3.1 scope
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153069#Airflow3.0-DraftAirflow3.1scope>
is documented on the wiki for discussion.
- Vikram mentioned that Dennis couldn't make the call today, but
asked for reviews on his PR which was early work towards Deadline
Alerts
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488182>,
which replaces SLA alerts in Airflow 2.x and is currently scoped for
Airflow 3.1. Rajesh added that he was confident with the timing, since
Dennis and others were actively working on this and would be ready to
update the team in two weeks' time.
- Jarek raised the question a
- Other language support
- Ash shared an update on the Go SDK. He shared a link to the
branch with the draft incarnation and said that a PR would be
coming soon.
- Vikram shared an update with the early Java SDK branch. Jens
chimed in referencing the offline discussion with Vikram
around leveraging
the Edge API for these other languages.
- The action item was for Jens, Ash, Kaxil, and Vikram to discuss
async on any changes needed to the Edge provider.
- Ash raised a concern that we may not see significant adoption
until the DAGs were in other languages as well. Elad and Jarek argued
against the viewpoint feeling that a single task in a
different language
would be very powerful for application integration. Elad said
that "The
notion of executing a task directly without a wrapper DAG is
a request I
hear from time to time (already)".
- FAB upgrades (4.6.2 first, 5.0.0 later) Jarek Potiuk
<https://cwiki.apache.org/confluence/display/~potiuk>
- Jarek raised the FAB upgrade issue which we had deferred a
couple of months ago to post 3.0. He said that we are in a much better
situation now, but that he needed help with the SqlAlchemy
migrations and
would follow up on the dev list.
- As part of this conversation, there was a discussion about
dropping support for Python 3.9 early. However, there was no
appetite for
that from the attendees because of all the complexity around
doing this
selectively for Core Airflow vs. Providers. This was going to
be conveyed
back to the dev list discussion by Jens.
- Example DAGS Jens Scheffler
<https://cwiki.apache.org/confluence/display/~jscheffl> Jarek Potiuk
<https://cwiki.apache.org/confluence/display/~potiuk>
- Jens and Jarek brought up the issue with the Example DAGs which
have grown to the point of having become too confusing,
because they are
used for two different purposes: One for developers testing
Airflow before
release, and second for users to learn how to use Airflow.
- There was clear agreement on separating the examples for those
two use cases and Jarek was going to take the lead on that.
- Jens was going to take the lead on making the true end user
example DAGs more useful with a story telling concept for a broad use
case.
- Continuing dev calls
- Vikram suggested that we make these dev calls every two weeks
going forward, rather than weekly.
- There was immediate agreement on this.
- Action items:
- Airflow docs UX update to make versions more obvious (Elad, Jarek)
- Update on Deadline Alerts AIP progress and Github board (Dennis,
Rajesh)
- Edge executor for other languages (Jens, Ash, Kaxil, Vikram)
- Get help on SqlAlchemy upgrade changes (Jarek)
- Example Dags update (Jens, Jarek)
- Make dev calls to be every two weeks (Vikram)
--
Vikram Koka
Chief Strategy Officer
Email: vik...@astronomer.io
<https://www.astronomer.io/>
