Overall I'm +1 NOTE: I still strongly believe we should _not_ brand this "Remote Executors" we already use Remote Executors (to mean CeleryExecutor, K8sExecutor, etc) in many many contexts as a contrast to Local Executors (LocalExecutor, SequentialExecutor). It's in our docs, blog posts, Airflow Summit talks, everywhere. Overloading this term will confuse users who understand the existing terminology. Instead we should go with other terms (also present in your description) such as "Distributed Executors", "Decentralized Executors", or something else similar.
Great work on this one and it's exciting that it might make it for 2.10! ________________________________ From: Scheffler Jens (XC-AS/EAE-ADA-T) <jens.scheff...@de.bosch.com.INVALID> Sent: Tuesday, July 16, 2024 10:36:48 PM To: dev@airflow.apache.org Subject: [EXT] [VOTE] (v2) AIP-69 Remote Executor CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le contenu ne présente aucun risque. Hi Developers, After some further discussion time I’d like to call for a vote for AIP-69. All details are described in: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor Note: * Compared to first VOTE in https://lists.apache.org/thread/tyfsrpjn12sz9dw50pbg16dsv6lmj610 more details have been added * A PoC PR is available in https://github.com/apache/airflow/pull/40224 * Status of progress in https://github.com/jscheffl/airflow/blob/feature/aip-69-poc/airflow/providers/remote/TODO.md * Q&A session was hosted, Notes in https://lists.apache.org/thread/h2nxkto0lxgjnqj8yps0qsh7ppbccx6g Remote Executor should be a special executor for use cases where a distributed (non central) setup across different security perimeters need to be achieved and a worker accesses the central site only via HTTP(s). It will leverage AIP-61 (Hybrid Execution) as well as builds on-top of AIP-44 (at least the parts needed for the worker, see PoC PR, it is already working on existing structures). Target is to deliver it with Airflow 2.10 as a Pre-Release. There it can be experienced/tested and incrementally be improved. It will integrate in Airflow 3 with AIP-72 and replace AIP-44 task communication with this. From the Q&A meeting main consent was elaborated in a direction of: - Remote Executor will be marked experimental, not contained in default release in 2.10 line - Even if installed, remote endpoint will be disabled by default to minimize risk of exposure - We would release the provider package only with a version suffix "pre0" to PyPi such that an user must explicitly install a pre-release version as manual install - Support and maintenance in Airflow 2.10++ will end with the feature being available in Airflow 3 to reduce double maintenance and as motivation to migrate Why already in 2.10? With the existing structures in Airflow 2.10 we can get started, it is already working with limitations. From there we can use it, learn on a running system and incrementally enhance and improve. The vote will run for 6 days and last till next Tuesday 23nd of July 2024 8:00 UTC. Everyone is encouraged to vote, although only PMC members and Committer's votes are considered binding. This is my +1. Mit freundlichen Grüßen / Best regards Jens Scheffler Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | GERMANY | www.bosch.com<http://www.bosch.com> Tel. +49 711 811-91508 | Mobil +49 160 90417410 | jens.scheff...@de.bosch.com<mailto:jens.scheff...@de.bosch.com> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus Forschner, Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
