Hi Developers, After some further discussion time I’d like to call for a vote for AIP-69. All details are described in: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor
Note: * Compared to first VOTE in https://lists.apache.org/thread/tyfsrpjn12sz9dw50pbg16dsv6lmj610 more details have been added * A PoC PR is available in https://github.com/apache/airflow/pull/40224 * Status of progress in https://github.com/jscheffl/airflow/blob/feature/aip-69-poc/airflow/providers/remote/TODO.md * Q&A session was hosted, Notes in https://lists.apache.org/thread/h2nxkto0lxgjnqj8yps0qsh7ppbccx6g Remote Executor should be a special executor for use cases where a distributed (non central) setup across different security perimeters need to be achieved and a worker accesses the central site only via HTTP(s). It will leverage AIP-61 (Hybrid Execution) as well as builds on-top of AIP-44 (at least the parts needed for the worker, see PoC PR, it is already working on existing structures). Target is to deliver it with Airflow 2.10 as a Pre-Release. There it can be experienced/tested and incrementally be improved. It will integrate in Airflow 3 with AIP-72 and replace AIP-44 task communication with this. From the Q&A meeting main consent was elaborated in a direction of: - Remote Executor will be marked experimental, not contained in default release in 2.10 line - Even if installed, remote endpoint will be disabled by default to minimize risk of exposure - We would release the provider package only with a version suffix "pre0" to PyPi such that an user must explicitly install a pre-release version as manual install - Support and maintenance in Airflow 2.10++ will end with the feature being available in Airflow 3 to reduce double maintenance and as motivation to migrate Why already in 2.10? With the existing structures in Airflow 2.10 we can get started, it is already working with limitations. From there we can use it, learn on a running system and incrementally enhance and improve. The vote will run for 6 days and last till next Tuesday 23nd of July 2024 8:00 UTC. Everyone is encouraged to vote, although only PMC members and Committer's votes are considered binding. This is my +1. Mit freundlichen Grüßen / Best regards Jens Scheffler Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY | www.bosch.com Tel. +49 711 811-91508 | Mobil +49 160 90417410 | jens.scheff...@de.bosch.com<mailto:jens.scheff...@de.bosch.com> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus Forschner, Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
