Remote Executor not worker. On Thu, Jul 18, 2024 at 8:52 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> My conclusion is (I spoke to Jens about it today) that we should attempt > to get a "workable, not necessarily optimized for performance, and > potentially having a few things not working version of it" and label it " > highly experimental, don't do it at home" - rather similar to what Jens' > approach is for the "pre" release of remote worker. Jens already has a > working Remote vorker on it, so having the base implemented in Airflow 2.10 > might be for him (and few other brave souls) who would want to test remote > worker on "2.10" version. And maybe we will also find a thing or two that > might be useful for AIP-72 implementation while doing it. > > J. > > On Thu, Jul 18, 2024 at 8:44 PM Daniel Standish > <daniel.stand...@astronomer.io.invalid> wrote: > >> I don't have strong feelings here either way, but we should probably try >> to >> arrive at a conclusion here sooner than later because meanwhile work on >> this continues. >> >> I think the question is sort of, is anyone going to essentially veto it, >> or >> perhaps veto it unless certain conditions are met. E.g. it has been >> proposed that the feature must be marked as experimental so as to not >> increase support burden on what is perhaps a dead end feature. >> >> Any other sort of proposals or questions outstanding on this one? >> >> >> >> On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai <amoghdesai....@gmail.com> >> wrote: >> >> > I'd prefer to move it completely to Airflow 3, because I don't really >> see >> > spending cycles >> > on developing something for the short term here. >> > >> > With Airflow 3, it would come out more complete and something that >> could be >> > usable by users without >> > having to put a halt to it and starting it again. >> > >> > I understand that it would affect the AIP 69 but as I whole i don't see >> > doing AIP 44 for Airflow 2 as beneficial. >> > >> > >> > Thanks & Regards, >> > Amogh Desai >> > >> > >> > On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <a...@apache.org> >> wrote: >> > >> > > -0.25 to me continuing - I don't see the value long term in putting >> more >> > > effort into this, and any lessons from this have already been learnt, >> as >> > > development on Airflow 3 has already started, so this would literally >> be >> > > happening in parallel. >> > > >> > > Given we are planning, at most, 1 or two more releases of Airflow 2 >> this >> > > feature will have to stay marked experimental so that it doesn't >> increase >> > > our support burden. >> > > >> > > My TL;Dr: I see this feature as a dead end now, but if you want to >> work >> > on >> > > it by my guest, as long as it doesn't cause us more (support) work in >> the >> > > long term. >> > > >> > > -ash >> > > >> > > On 12 July 2024 14:20:14 BST, "Michał Modras" < >> michalmod...@google.com >> > .INVALID> >> > > wrote: >> > > >I think we should finish the AIP within Airflow 2 - it will take time >> > > until >> > > >Airflow 3 is out, and I believe some learnings from finishing and >> > running >> > > >this AIP might be useful for Airflow 3. We plan to contribute to >> > finishing >> > > >this AIP. >> > > > >> > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T) >> > > ><jens.scheff...@de.bosch.com.invalid> wrote: >> > > > >> > > >> Hi, >> > > >> >> > > >> I'd favor to make it usable - especially as we are at 80%. >> > > >> >> > > >> Main motivation is that with our environment we see stability >> problems >> > > >> with the distributed setup and using Celery, which was the main >> > > motivation >> > > >> to spin the discussion about AIP-69. AIP-69 is depending on the >> > feature. >> > > >> Waiting another 12-18 months to be able to host a stable >> distributed >> > > setup >> > > >> based on Airflow 3 is something hard to argue. And I can confirm >> it is >> > > >> working already in my AIP-69 PoC. >> > > >> >> > > >> In this light I could offer to move it to at least the level that >> it >> > can >> > > >> be used and is properly CI tested as using it for AIP-69 as first >> > > consumer >> > > >> (which could reduce the scope to task execution, DAG parsing and >> > > triggered >> > > >> could be taken out-of-scope for AIP-69 dependency for example). I >> > could >> > > >> offer supporting to close the gaps to completion. >> > > >> >> > > >> In regards of workload the completion should be a target before the >> > > >> cut-off to Airflow 3, so I would assume only "keeping the lights >> on" >> > > would >> > > >> be a distraction while developing Airflow 3. >> > > >> >> > > >> Mit freundlichen Grüßen / Best regards >> > > >> >> > > >> Jens Scheffler >> > > >> >> > > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) >> > > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 >> Stuttgart-Vaihingen | >> > > >> GERMANY | www.bosch.com >> > > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 >> 90417410 >> > > >> <+49%20160%2090417410> | jens.scheff...@de.bosch.com >> > > >> >> > > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; >> > > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; >> > > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. >> > Markus >> > > >> Forschner, >> > > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert >> > > >> >> > > >> -----Original Message----- >> > > >> From: Jed Cunningham <jedcunning...@apache.org> >> > > >> Sent: Thursday, July 11, 2024 10:41 PM >> > > >> To: dev@airflow.apache.org >> > > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44 >> > > >> >> > > >> It feels a little weird to add a new "forever" experimental >> feature in >> > > >> Airflow 2 that we already know won't be there in Airflow 3. Not >> > > something >> > > >> I'd want to be really user facing at this point in time either. >> > > >> >> > > >> Given the short timeline for Airflow 3, I imagine we'd be better >> off >> > > >> spending those cycles elsewhere. My 2c - not my cycles :) >> > > >> >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> > > For additional commands, e-mail: dev-h...@airflow.apache.org >> > > >> > > >> > >> >
