I agree that this AIP is well thought out and needs to be worked on. Just making a note that this is dependent on AIP-44 which was discussed in an earlier thread on the dev list, which is currently proposed to be marked "experimental". I approve this AIP with the explicit caveat that this also be marked as "Experimental use only" as it is dependent on AIP-44. Once AIP-72 is released and this AIP is updated to depend on AIP-72, this can then be marked as "GA".
I believe this is consistent with Jen's notes above, but wanted to reiterate that. +1 (binding) On Thu, Jul 18, 2024 at 2:00 PM Oliveira, Niko <oniko...@amazon.com.invalid> wrote: > Overall I'm +1 > > NOTE: I still strongly believe we should _not_ brand this "Remote > Executors" we already use Remote Executors (to mean CeleryExecutor, > K8sExecutor, etc) in many many contexts as a contrast to Local Executors > (LocalExecutor, SequentialExecutor). It's in our docs, blog posts, Airflow > Summit talks, everywhere. Overloading this term will confuse users who > understand the existing terminology. Instead we should go with other terms > (also present in your description) such as "Distributed Executors", > "Decentralized Executors", or something else similar. > > Great work on this one and it's exciting that it might make it for 2.10! > > ________________________________ > From: Scheffler Jens (XC-AS/EAE-ADA-T) <jens.scheff...@de.bosch.com > .INVALID> > Sent: Tuesday, July 16, 2024 10:36:48 PM > To: dev@airflow.apache.org > Subject: [EXT] [VOTE] (v2) AIP-69 Remote Executor > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > Hi Developers, > > After some further discussion time I’d like to call for a vote for AIP-69. > All details are described in: > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-69+Remote+Executor > > Note: > > * Compared to first VOTE in > https://lists.apache.org/thread/tyfsrpjn12sz9dw50pbg16dsv6lmj610 more > details have been added > * A PoC PR is available in > https://github.com/apache/airflow/pull/40224 > * Status of progress in > https://github.com/jscheffl/airflow/blob/feature/aip-69-poc/airflow/providers/remote/TODO.md > * Q&A session was hosted, Notes in > https://lists.apache.org/thread/h2nxkto0lxgjnqj8yps0qsh7ppbccx6g > > Remote Executor should be a special executor for use cases where a > distributed (non central) setup across different security perimeters need > to be achieved and a worker accesses the central site only via HTTP(s). It > will leverage AIP-61 (Hybrid Execution) as well as builds on-top of AIP-44 > (at least the parts needed for the worker, see PoC PR, it is already > working on existing structures). > Target is to deliver it with Airflow 2.10 as a Pre-Release. There it can > be experienced/tested and incrementally be improved. It will integrate in > Airflow 3 with AIP-72 and replace AIP-44 task communication with this. > > > From the Q&A meeting main consent was elaborated in a direction of: > > - Remote Executor will be marked experimental, not contained in default > release in 2.10 line > > - Even if installed, remote endpoint will be disabled by default to > minimize risk of exposure > > - We would release the provider package only with a version suffix "pre0" > to PyPi such that an user must explicitly install a pre-release version as > manual install > > - Support and maintenance in Airflow 2.10++ will end with the feature > being available in Airflow 3 to reduce double maintenance and as motivation > to migrate > > > > Why already in 2.10? With the existing structures in Airflow 2.10 we can > get started, it is already working with limitations. From there we can use > it, learn on a running system and incrementally enhance and improve. > > > > The vote will run for 6 days and last till next Tuesday 23nd of July 2024 > 8:00 UTC. > > > > Everyone is encouraged to vote, although only PMC members and Committer's > votes are considered binding. > > > > This is my +1. > > Mit freundlichen Grüßen / Best regards > > Jens Scheffler > > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen | > GERMANY | www.bosch.com<http://www.bosch.com> > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > jens.scheff...@de.bosch.com<mailto:jens.scheff...@de.bosch.com> > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus > Forschner, > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert > >
