Also one thing after spending some time on it - after discussion with Jens I already made the "DB Isolation" mode works end-to-end in breeze ( https://github.com/apache/airflow/pull/40894) - which was fairly easy (and it seems to work pretty well/stable with some limitations (no mini-scheduler for now). Bt I am also working through making the standalone DAGFileProcessor working (it had missing bits and pieces) and I strongly believe that work will be extremely useful for AIP-72 as well because it turns out that there is also quite a some "spaggetish" DB / access at places so whatever refactoring and changes are implemented now, they should basically be a base for AIP-72 DAGFileProcessing API. It's not "com[plex" but a bit tedious - and it will show immediately where API calls will be needed as well.
So that work is not **entirely* useless. J. On Fri, Jul 19, 2024 at 6:59 PM Vikram Koka <vik...@astronomer.io.invalid> wrote: > I am very relieved with the "experimental, don't try this at home" label on > this body of work, since this will be replaced shortly. > > If there is belief that this will help accelerate learning towards AIP-69 > (Remote Executor), I understand the rationale for proceeding forward with > this given the caveats above. > > Given that, I don't have strong feelings either way on this. I am ok with > the conclusion above of releasing it largely "as is" to be used as a test > bed before AIP-72 is released as part of 3.0. > > > > On Thu, Jul 18, 2024 at 11:54 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Remote Executor not worker. > > > > On Thu, Jul 18, 2024 at 8:52 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > My conclusion is (I spoke to Jens about it today) that we should > attempt > > > to get a "workable, not necessarily optimized for performance, and > > > potentially having a few things not working version of it" and label it > > " > > > highly experimental, don't do it at home" - rather similar to what > Jens' > > > approach is for the "pre" release of remote worker. Jens already has a > > > working Remote vorker on it, so having the base implemented in Airflow > > 2.10 > > > might be for him (and few other brave souls) who would want to test > > remote > > > worker on "2.10" version. And maybe we will also find a thing or two > that > > > might be useful for AIP-72 implementation while doing it. > > > > > > J. > > > > > > On Thu, Jul 18, 2024 at 8:44 PM Daniel Standish > > > <daniel.stand...@astronomer.io.invalid> wrote: > > > > > >> I don't have strong feelings here either way, but we should probably > try > > >> to > > >> arrive at a conclusion here sooner than later because meanwhile work > on > > >> this continues. > > >> > > >> I think the question is sort of, is anyone going to essentially veto > it, > > >> or > > >> perhaps veto it unless certain conditions are met. E.g. it has been > > >> proposed that the feature must be marked as experimental so as to not > > >> increase support burden on what is perhaps a dead end feature. > > >> > > >> Any other sort of proposals or questions outstanding on this one? > > >> > > >> > > >> > > >> On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai < > amoghdesai....@gmail.com> > > >> wrote: > > >> > > >> > I'd prefer to move it completely to Airflow 3, because I don't > really > > >> see > > >> > spending cycles > > >> > on developing something for the short term here. > > >> > > > >> > With Airflow 3, it would come out more complete and something that > > >> could be > > >> > usable by users without > > >> > having to put a halt to it and starting it again. > > >> > > > >> > I understand that it would affect the AIP 69 but as I whole i don't > > see > > >> > doing AIP 44 for Airflow 2 as beneficial. > > >> > > > >> > > > >> > Thanks & Regards, > > >> > Amogh Desai > > >> > > > >> > > > >> > On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <a...@apache.org> > > >> wrote: > > >> > > > >> > > -0.25 to me continuing - I don't see the value long term in > putting > > >> more > > >> > > effort into this, and any lessons from this have already been > > learnt, > > >> as > > >> > > development on Airflow 3 has already started, so this would > > literally > > >> be > > >> > > happening in parallel. > > >> > > > > >> > > Given we are planning, at most, 1 or two more releases of Airflow > 2 > > >> this > > >> > > feature will have to stay marked experimental so that it doesn't > > >> increase > > >> > > our support burden. > > >> > > > > >> > > My TL;Dr: I see this feature as a dead end now, but if you want to > > >> work > > >> > on > > >> > > it by my guest, as long as it doesn't cause us more (support) work > > in > > >> the > > >> > > long term. > > >> > > > > >> > > -ash > > >> > > > > >> > > On 12 July 2024 14:20:14 BST, "Michał Modras" < > > >> michalmod...@google.com > > >> > .INVALID> > > >> > > wrote: > > >> > > >I think we should finish the AIP within Airflow 2 - it will take > > time > > >> > > until > > >> > > >Airflow 3 is out, and I believe some learnings from finishing and > > >> > running > > >> > > >this AIP might be useful for Airflow 3. We plan to contribute to > > >> > finishing > > >> > > >this AIP. > > >> > > > > > >> > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T) > > >> > > ><jens.scheff...@de.bosch.com.invalid> wrote: > > >> > > > > > >> > > >> Hi, > > >> > > >> > > >> > > >> I'd favor to make it usable - especially as we are at 80%. > > >> > > >> > > >> > > >> Main motivation is that with our environment we see stability > > >> problems > > >> > > >> with the distributed setup and using Celery, which was the main > > >> > > motivation > > >> > > >> to spin the discussion about AIP-69. AIP-69 is depending on the > > >> > feature. > > >> > > >> Waiting another 12-18 months to be able to host a stable > > >> distributed > > >> > > setup > > >> > > >> based on Airflow 3 is something hard to argue. And I can > confirm > > >> it is > > >> > > >> working already in my AIP-69 PoC. > > >> > > >> > > >> > > >> In this light I could offer to move it to at least the level > that > > >> it > > >> > can > > >> > > >> be used and is properly CI tested as using it for AIP-69 as > first > > >> > > consumer > > >> > > >> (which could reduce the scope to task execution, DAG parsing > and > > >> > > triggered > > >> > > >> could be taken out-of-scope for AIP-69 dependency for > example). I > > >> > could > > >> > > >> offer supporting to close the gaps to completion. > > >> > > >> > > >> > > >> In regards of workload the completion should be a target before > > the > > >> > > >> cut-off to Airflow 3, so I would assume only "keeping the > lights > > >> on" > > >> > > would > > >> > > >> be a distraction while developing Airflow 3. > > >> > > >> > > >> > > >> Mit freundlichen Grüßen / Best regards > > >> > > >> > > >> > > >> Jens Scheffler > > >> > > >> > > >> > > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) > > >> > > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 > > >> Stuttgart-Vaihingen | > > >> > > >> GERMANY | www.bosch.com > > >> > > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 > > >> 90417410 > > >> > > >> <+49%20160%2090417410> | jens.scheff...@de.bosch.com > > >> > > >> > > >> > > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB > > 14000; > > >> > > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > >> > > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, > Dr. > > >> > Markus > > >> > > >> Forschner, > > >> > > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja > > Rückert > > >> > > >> > > >> > > >> -----Original Message----- > > >> > > >> From: Jed Cunningham <jedcunning...@apache.org> > > >> > > >> Sent: Thursday, July 11, 2024 10:41 PM > > >> > > >> To: dev@airflow.apache.org > > >> > > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44 > > >> > > >> > > >> > > >> It feels a little weird to add a new "forever" experimental > > >> feature in > > >> > > >> Airflow 2 that we already know won't be there in Airflow 3. Not > > >> > > something > > >> > > >> I'd want to be really user facing at this point in time either. > > >> > > >> > > >> > > >> Given the short timeline for Airflow 3, I imagine we'd be > better > > >> off > > >> > > >> spending those cycles elsewhere. My 2c - not my cycles :) > > >> > > >> > > >> > > > > >> > > > > --------------------------------------------------------------------- > > >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >> > > For additional commands, e-mail: dev-h...@airflow.apache.org > > >> > > > > >> > > > > >> > > > >> > > > > > >
