I am very relieved with the "experimental, don't try this at home" label on this body of work, since this will be replaced shortly.
If there is belief that this will help accelerate learning towards AIP-69 (Remote Executor), I understand the rationale for proceeding forward with this given the caveats above. Given that, I don't have strong feelings either way on this. I am ok with the conclusion above of releasing it largely "as is" to be used as a test bed before AIP-72 is released as part of 3.0. On Thu, Jul 18, 2024 at 11:54 AM Jarek Potiuk <ja...@potiuk.com> wrote: > Remote Executor not worker. > > On Thu, Jul 18, 2024 at 8:52 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > My conclusion is (I spoke to Jens about it today) that we should attempt > > to get a "workable, not necessarily optimized for performance, and > > potentially having a few things not working version of it" and label it > " > > highly experimental, don't do it at home" - rather similar to what Jens' > > approach is for the "pre" release of remote worker. Jens already has a > > working Remote vorker on it, so having the base implemented in Airflow > 2.10 > > might be for him (and few other brave souls) who would want to test > remote > > worker on "2.10" version. And maybe we will also find a thing or two that > > might be useful for AIP-72 implementation while doing it. > > > > J. > > > > On Thu, Jul 18, 2024 at 8:44 PM Daniel Standish > > <daniel.stand...@astronomer.io.invalid> wrote: > > > >> I don't have strong feelings here either way, but we should probably try > >> to > >> arrive at a conclusion here sooner than later because meanwhile work on > >> this continues. > >> > >> I think the question is sort of, is anyone going to essentially veto it, > >> or > >> perhaps veto it unless certain conditions are met. E.g. it has been > >> proposed that the feature must be marked as experimental so as to not > >> increase support burden on what is perhaps a dead end feature. > >> > >> Any other sort of proposals or questions outstanding on this one? > >> > >> > >> > >> On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai <amoghdesai....@gmail.com> > >> wrote: > >> > >> > I'd prefer to move it completely to Airflow 3, because I don't really > >> see > >> > spending cycles > >> > on developing something for the short term here. > >> > > >> > With Airflow 3, it would come out more complete and something that > >> could be > >> > usable by users without > >> > having to put a halt to it and starting it again. > >> > > >> > I understand that it would affect the AIP 69 but as I whole i don't > see > >> > doing AIP 44 for Airflow 2 as beneficial. > >> > > >> > > >> > Thanks & Regards, > >> > Amogh Desai > >> > > >> > > >> > On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <a...@apache.org> > >> wrote: > >> > > >> > > -0.25 to me continuing - I don't see the value long term in putting > >> more > >> > > effort into this, and any lessons from this have already been > learnt, > >> as > >> > > development on Airflow 3 has already started, so this would > literally > >> be > >> > > happening in parallel. > >> > > > >> > > Given we are planning, at most, 1 or two more releases of Airflow 2 > >> this > >> > > feature will have to stay marked experimental so that it doesn't > >> increase > >> > > our support burden. > >> > > > >> > > My TL;Dr: I see this feature as a dead end now, but if you want to > >> work > >> > on > >> > > it by my guest, as long as it doesn't cause us more (support) work > in > >> the > >> > > long term. > >> > > > >> > > -ash > >> > > > >> > > On 12 July 2024 14:20:14 BST, "Michał Modras" < > >> michalmod...@google.com > >> > .INVALID> > >> > > wrote: > >> > > >I think we should finish the AIP within Airflow 2 - it will take > time > >> > > until > >> > > >Airflow 3 is out, and I believe some learnings from finishing and > >> > running > >> > > >this AIP might be useful for Airflow 3. We plan to contribute to > >> > finishing > >> > > >this AIP. > >> > > > > >> > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T) > >> > > ><jens.scheff...@de.bosch.com.invalid> wrote: > >> > > > > >> > > >> Hi, > >> > > >> > >> > > >> I'd favor to make it usable - especially as we are at 80%. > >> > > >> > >> > > >> Main motivation is that with our environment we see stability > >> problems > >> > > >> with the distributed setup and using Celery, which was the main > >> > > motivation > >> > > >> to spin the discussion about AIP-69. AIP-69 is depending on the > >> > feature. > >> > > >> Waiting another 12-18 months to be able to host a stable > >> distributed > >> > > setup > >> > > >> based on Airflow 3 is something hard to argue. And I can confirm > >> it is > >> > > >> working already in my AIP-69 PoC. > >> > > >> > >> > > >> In this light I could offer to move it to at least the level that > >> it > >> > can > >> > > >> be used and is properly CI tested as using it for AIP-69 as first > >> > > consumer > >> > > >> (which could reduce the scope to task execution, DAG parsing and > >> > > triggered > >> > > >> could be taken out-of-scope for AIP-69 dependency for example). I > >> > could > >> > > >> offer supporting to close the gaps to completion. > >> > > >> > >> > > >> In regards of workload the completion should be a target before > the > >> > > >> cut-off to Airflow 3, so I would assume only "keeping the lights > >> on" > >> > > would > >> > > >> be a distraction while developing Airflow 3. > >> > > >> > >> > > >> Mit freundlichen Grüßen / Best regards > >> > > >> > >> > > >> Jens Scheffler > >> > > >> > >> > > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > >> > > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 > >> Stuttgart-Vaihingen | > >> > > >> GERMANY | www.bosch.com > >> > > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 > >> 90417410 > >> > > >> <+49%20160%2090417410> | jens.scheff...@de.bosch.com > >> > > >> > >> > > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB > 14000; > >> > > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > >> > > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. > >> > Markus > >> > > >> Forschner, > >> > > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja > Rückert > >> > > >> > >> > > >> -----Original Message----- > >> > > >> From: Jed Cunningham <jedcunning...@apache.org> > >> > > >> Sent: Thursday, July 11, 2024 10:41 PM > >> > > >> To: dev@airflow.apache.org > >> > > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44 > >> > > >> > >> > > >> It feels a little weird to add a new "forever" experimental > >> feature in > >> > > >> Airflow 2 that we already know won't be there in Airflow 3. Not > >> > > something > >> > > >> I'd want to be really user facing at this point in time either. > >> > > >> > >> > > >> Given the short timeline for Airflow 3, I imagine we'd be better > >> off > >> > > >> spending those cycles elsewhere. My 2c - not my cycles :) > >> > > >> > >> > > > >> > > > --------------------------------------------------------------------- > >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> > > For additional commands, e-mail: dev-h...@airflow.apache.org > >> > > > >> > > > >> > > >> > > >
