Following up after today's Airflow 3 call, I wanted to start discussion on whether we should continue implementing AIP-44 and release it in Airflow 2 or whether we should focus on AIP-72 and Airflow 3.
We could go both ways. The implementation is somewhat 80% done - what we really miss there is automation of testing it in CI to make sure that all DB access is going through RPC (and complete remaining things that aren't). Daniel particularly put a lot of effort into implementing some of the remaining pieces and manually got the "task" working. also Jens had done some manual testing with "remote executor" (AIP-69) and got it to "work". But thorough testing - especially "forward-looking-testing" is needed. Also wiring the authentication is needed (following the webserver/log authentication we already have). On one hand - I know Google Team is particularly interested in having it completed, also Jens would like to use it to run the "AIP-69" prototype in Airflow 2. if we don't implement AIP-44, AIP-69 will only land in Airflow 3. On the other hand, it will drag out some efforts from Airflow 3 and will keep people out of implementing some of the changes we need. Also if we implement and release it now - we will have to support and solve issues Also in Airflow 2. Users who will deploy it, will have to redo their deployment for Airflow 3. And without multi-team AIP-67 it will only partially separate out the DB access (there will be no team isolation, DAG authors will still be able to access "all other" DAGs, tasks, connections, variables etc., but they won't be able to run arbitrary DB operations). We could go both ways, and yes - there is a lot of "Sunken cost". I'd love to hear what others think about it. J. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
