I would favor moving towards Airflow 3 and delivering a fully-baked feature.
On Thu, 11 Jul 2024 at 18:28, Jarek Potiuk <ja...@potiuk.com> wrote: > Following up after today's Airflow 3 call, I wanted to start > discussion on whether we should continue implementing AIP-44 and > release it in Airflow 2 or whether we should focus on AIP-72 and > Airflow 3. > > We could go both ways. > > The implementation is somewhat 80% done - what we really miss there is > automation of testing it in CI to make sure that all DB access is > going through RPC (and complete remaining things that aren't). Daniel > particularly put a lot of effort into implementing some of the > remaining pieces and manually got the "task" working. also Jens had > done some manual testing with "remote executor" (AIP-69) and got it to > "work". But thorough testing - especially "forward-looking-testing" is > needed. Also wiring the authentication is needed (following the > webserver/log authentication we already have). > > On one hand - I know Google Team is particularly interested in having > it completed, also Jens would like to use it to run the "AIP-69" > prototype in Airflow 2. if we don't implement AIP-44, AIP-69 will only > land in Airflow 3. > > On the other hand, it will drag out some efforts from Airflow 3 and > will keep people out of implementing some of the changes we need. Also > if we implement and release it now - we will have to support and solve > issues Also in Airflow 2. Users who will deploy it, will have to redo > their deployment for Airflow 3. And without multi-team AIP-67 it will > only partially separate out the DB access (there will be no team > isolation, DAG authors will still be able to access "all other" DAGs, > tasks, connections, variables etc., but they won't be able to run > arbitrary DB operations). > > We could go both ways, and yes - there is a lot of "Sunken cost". > > I'd love to hear what others think about it. > > J. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
