Hey everyone,
I updated our meeting notes document in the Airflow wiki to capture the
notes from our dev call yesterday. The link for those notes is here
<https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.17>
I was unable to make the dev call yesterday because I was unwell. Thank you
Ash for leading the call.
It was great to hear the progress on the Event Driven Scheduling. Great
work Vincent and team! It was also great to see how the UI has been coming
along with the updates around DAG Triggering and Task Status, as well as
the GridView.
The overall positivity and enthusiasm from all the AIP owners and
participants is awesome. Can't wait to see the alpha builds soon!
To everyone who attended the meeting, please check the summary and add
anything that I may have missed.
For those who could not join, please let us know if you disagree with
anything discussed and agreed upon in the meeting. Also, please do ask
questions if something is unclear.
Best regards,
Vikram
--
Below is the summary from the call on Thursday:
- Follow-up on action items from the last call:
- AIP-83 amendment aka Data completeness discussion follow-up Daniel
Standish <https://cwiki.apache.org/confluence/display/~dstandish>
Tzu-ping
Chung <https://cwiki.apache.org/confluence/display/~uranusjr>
- Daniel Standish confirmed that the vote had been started on dev
list and encouraged the team to respond.
- Development Updates:
- Update on AIP-82 External Event Driven Scheduling
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-82+External+event+driven+scheduling+in+Airflow>
Vincent BECK
<https://cwiki.apache.org/confluence/display/~vin100.beck>
- Vincent demonstrated the implementation of the AssetWatcher
concept built on top of FileOperator, which enables a DAG to
be triggered
based on the arrival of a file in an external blob storage.
- Vincent also did a second demo based on watching for two
different files and that the DAG would be triggered based on
either file
arriving in blob storage.
- Vincent also showed a DAG which could be triggered based on a
message arriving in an SQS queue.
- Vincent highlighted that DAG authors need to be careful on what
triggers to use for invoking DAGs, since it is possible to
have additional
DAG runs because of persistent events. There was discussion
around this and
the agreement was to take that discussion offline.
- Update on AIP-72 Task Execution Interface aka Task SDK
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK>
Ash Berlin-Taylor
<https://cwiki.apache.org/confluence/display/~ash> Kaxil
Naik <https://cwiki.apache.org/confluence/display/~kaxilnaik> Amogh
Desai <https://cwiki.apache.org/confluence/display/~amoghdesai>
- Ash highlighted that the team was chipping away on the work. He
highlighted that Mapped Tasks was close to completely working
based on the
TaskSDK.
- He said that once Mapped Tasks was done, Celery and Kubernetes
executors would be moved over to use the TaskSDK.
- He also said that this was the last "big feature" to be done and
then there was going to be the long tail of fixes.
- Update on AIP-38 Modern Web Application
<https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application>
Pierre
Jeambrun Brent Bovenzi
<https://cwiki.apache.org/confluence/display/~bbovenzi>
- Pierre demonstrated the triggering and clearing of DAG runs and
Tasks.
- Pierre said that Brent is working on the Grid view and it isn't
merged yet, but close
- Pierre said that after the alpha, we would be able to use the
new API exclusively and then we could delete the old UI and API.
- Pierre also said that they were making progress and no
particular blockers.
- Brent demonstrated the GridView and said how they were looking
forward to getting feedback as part of the Alphas.
- Update on Provider packages new structure Jarek Potiuk
<https://cwiki.apache.org/confluence/display/~potiuk>
- Jarek did a quick walk through about moving the providers to the
new structure as a prelude to the new packaging work upcoming
in February.
- He said that he would ask for help from the larger team on this
work, since it could not be 100% automated. He said that his script
automated 90+% of the work, but needed human intervention
for the last
little bit. He walked the team through a quick run of this script to
illustrate.
- Discussion Topics
- Milestone update and progress Vikram Koka
<https://cwiki.apache.org/confluence/display/~vikramkoka> (actually Ash
Berlin-Taylor <https://cwiki.apache.org/confluence/display/~ash> )
- Ash walked through the overall progress update, since Vikram was
out sick and couldn't make the call.
- Ash gave a quick overview of where we are based on the dev
milestone wiki page. He highlighted that we are shooting for
getting the
first alpha builds out next week.
- Elad asked for clarification about the "feature freeze" beyond
the alpha milestone. Ash took the action item to post a
message to the dev
list about it.
- Rajesh asked about where we are on the timeline and different
AIP owners went through their progress updates as compared to
the published
timeline
- Jed, Vincent, Pierre, Vincent, Bugra, and Daniel Standish all
went through their AIPs and expressed confidence in the
schedule as they
saw it.
- Ash summarized the discussion to express that the first alpha
would have rough edges, but that the team felt positive about
the progress
and where we are at this point.
- AIP-81 CLI Discussion: Task Commands Remote vs. Local Buğra Öztürk
<https://cwiki.apache.org/confluence/display/~bugraoz93> (Doc link
<https://docs.google.com/document/d/1BSEQxKEOArPRCclfCrAjAN5hjC56vG0R5nZPc3pxF9Y/edit?usp=sharing>
)
- Bugra wanted to have a discussion around some of the CLI Task
commands which would not be possible through the Task API.
- Because of a lack of time, the proposal was to have that
discussion async.
- Bugra suggested reviewing the doc also for a discussion around
local vs. remote commands.
- Airflow 2.11 release discussions Jens Scheffler
<https://cwiki.apache.org/confluence/display/~jscheffl> (Jarek Potiuk
<https://cwiki.apache.org/confluence/display/~potiuk> stepped in
because of bad Wifi connection for Jens)
- Jed said that Utkarsh is planning to cut 2.10.5 next week.
- Topics raised demanded/needed for 2.11
- airflow config lint command (but still can be added in
2.10-line or independent as well, no big driver)
- Migration tooling (which is still WIP)
- No other things raised
- Generation notion: Not critical at the moment. At least nobody
raised a deadline or critical feature (yet)
- Action items on/before next dev call:
- Ash to send out an email to the dev list regarding "feature freeze"
guidelines, both to developers and committers about "merging PRs".
- Team to review Bugra's doc around CLI changes
--
Vikram Koka
Chief Strategy Officer
Email: vik...@astronomer.io
<https://www.astronomer.io/>
