Hey everyone, I updated our meeting notes document in the Airflow wiki to capture the notes from our dev call on Thursday, the 19th of December. The link for those notes is here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-Summary.15> It was inspiring to see the progress on the UI modernization project (AIP-38) and the Task SDK project (AIP-72). Wonderful progress, team!
To everyone who attended the meeting, please check the summary and add anything that I may have missed. For those who could not join, please let us know if you disagree with anything discussed and agreed upon in the meeting. Also, please do ask questions if something is unclear. There's already an initial agenda for our next dev call, which is scheduled for 9th Jan. If you would like something to be added to the proposed agenda for that meeting, please add it here <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=308153072#Airflow3Devcall:MeetingNotes-(Proposed)Agenda.5> or let me know. Happy Holidays! Vikram -- Below is the summary from the call on Thursday: -- - Follow-up on action items from the last call: - Update on the Data completeness discussion (DanielS): - Daniel presented the problem around ambiguous behavior for backfills, when logical dates are no longer unique. - He spoke about how the traditional Airflow behaviour using execution date based on the DAG schedule implicitly established a date-based partitioning scheme for DAGs. - He then advocated making this behaviour explicit, by defining this to be a partition scheme of "implicit" and also supporting a partition scheme of "none" for non-date-based partitioned DAGs which could be run multiple times for the exact same logical date, which is a common use case both for model training as well as for inference execution. - There was a significant level of debate around this topic, with two counter-points. The first counter-point was that the "partition" term was confusing, especially with "Data Asset Partitions" upcoming in AIP-76 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-76+Asset+Partitions>, and if this was the same, different, or related. The second counter-point was around whether a new DAG-level parameter should be added for this, as opposed to either one or more combinations of DAG API/UI invocation parameters and existing DAG parameters such as "catchup". - Since a conclusion could not be reached in the call, the action item for Daniel, was to write up a document as an amendment to the existing AIP-83 Remove logical date uniqueness and share it with the mailing list. Daniel created and shared this doc <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-83+amendment+to+support+classic+Airflow+authoring+style> on Friday. - Development updates and presentations - Update on AIP-38 Modern Web Application <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-38+Modern+Web+Application> (Brent) - Brent demonstrated the progress on the UI specifically including the DagsList and DagDetails pages. - This included the ability to trigger DAGs and view DagRuns. - Brent then drilled into viewing DAG Run details such as Task logs and XCom data. - It was very encouraging to see the core DAG development and test workflows in action as part of Airflow 3. - Update on AIP-72 Task Execution Interface aka Task SDK <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK> (Amogh, Kaxil, Ash) - Amogh started the update on AIP-72 by walking through how deferred tasks were being executed based on the Task SDK. - The demo was based on a DAG including a date time sensor and the interaction between the supervisor process and the task runner for different task states. - Amogh also explained the task execution states which were currently implemented and the ones which were still to be done. - Kaxil then walked through the implementation of the TaskContext, a key concept within the Task Execution Interface. - This included the interface and the implementation of Connections, and how that was fetched from the API server. - Ash briefly shared a PR that removed the database dependency for DAG parsing and how that leveraged the supervisor-task runner communication interface. - Because we were running out of time in the dev call, he said that would subsequently post a video of the demo. Ash posted this demo video to the dev list later. - Discussion topics: - PR strategy for 3.1+ feature work (RajeshB) - Rajesh wanted to discuss how to handle PRs for post-3.0 feature work. Since the dev call was already running over, this was to be discussed async. - Scheduling for next dev call - Since a lot of committers are out of holidays, the next dev call will be on the 9th of January, 2025 instead of the 2nd of January. -- Vikram Koka Chief Strategy Officer Email: vik...@astronomer.io <https://www.astronomer.io/>
