FYI: We likely won't have time in tomorrow's meeting for a discussion on this unless we bump a number of other things off the agenda
On 10 July 2024 19:05:38 BST, "Michał Modras" <michalmod...@google.com.INVALID> wrote: >Good discussion here, few points from our perspective: > >- +1 for separating DAG submission into an API endpoint or SDK method. Not >only it creates a clean interface boundary for a parsed DAG, but also >enables other modes of using that endpoint, e.g. event-driven, which could >be a prelude for on-demand DAG processing without the parsing >loop, executing DAG parsing remotely, etc. > >- With the interface for DAG submission, the DAG lifecycle of parsing / >serialisation / execution could be decoupled, both technically and in terms >of the AIPs & workstreams of Airflow 3. Having aligned on the boundary >shape, the Google team would be happy to contribute to the DAG parsing / >processing part (which would be a separate AIP, potentially feeding DAGs >into AIP-72). > >- DAG parser/processor should assume identity and related accesses in a >similar way to executor described in AIP-72. I don't think they should be >the same component, but could reuse the same parts. The exact shape (and >flow) of DAG parser/processor should be decoupled into a separate AIP >(we're happy to drive this). > >- Moving callback execution to workers seems like the right move. Since >some callbacks might have 'central' dependencies, we could have two types >of callbacks: 1) regular callbacks, executed on worker, and 2) remote >callbacks, executed wherever, based on a predefined interface. > >Let's discuss further in tomorrow's meeting! > >On Wed, Jul 10, 2024 at 6:56 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > Sound good? >> >> Absolutely. I agree with all that. Even more - I think it's OK that >> the implementation part of the DAGFileProcessor changes might be >> shifted to AIP-67. While AIP-72 design should take into account (and >> define) DAG serialization and callback passing, implementation of it >> could be done as part of AIP-67 - so that the team implementing AIP-67 >> can build on the foundation of what AIP-72 implement, >> >> This has the added advantage that the AIP-72 "ways" will be known more >> intimately by more people. >> >> J. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >>
