All right - waiting for the last review on https://github.com/apache/airflow/pull/40916 but I think we are ready before the cut-off.
What nicely work there (at least manually checks): a) airflow in breeze works out-of-the-box with any combination of: * --db-isolation-mode * --standalone-dag-processor * Local and Celery Executor (Local Executor mostly because I wanted to test if it works when run K8S Executor that intenrnally uses LocalExecutor b) Authentication works nicely using separate internal_api_secret_key and JWT signer. c) Performance does not look bad actually at least on local machine - but I am sure it can be optimized further. d) we do not use https:// - but I think this should be described in the documentation that SSL terminating proxies should be used in front of internal-api to add SSL (usual practice). We might want to give a big WARNING if someone uses HTTP:// in the client What we will need more is: a) special tests for isolation mode b) documentation (stressing experimental status) But those can be even added after the cut-off for 2.10 branch and cherry-picked potentially. I think particularly the Dag File Processor was an interesting one in light of AIP-72 - I think it can be much more easily mapped almost 1-1 to whatever AIP-72 brings. And nice thing about it that we can tell adventurous users to use the internal API and report any issues they encounter with lack of DB access on the side of worker and parser long before AIP-72 will be out) J. On Fri, Jul 19, 2024 at 8:58 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Also one thing after spending some time on it - after discussion with Jens > I already made the "DB Isolation" mode works end-to-end in breeze ( > https://github.com/apache/airflow/pull/40894) - which was fairly easy > (and it seems to work pretty well/stable with some limitations (no > mini-scheduler for now). Bt I am also working through making the standalone > DAGFileProcessor working (it had missing bits and pieces) and I strongly > believe that work will be extremely useful for AIP-72 as well because it > turns out that there is also quite a some "spaggetish" DB / access at > places so whatever refactoring and changes are implemented now, they should > basically be a base for AIP-72 DAGFileProcessing API. It's not "com[plex" > but a bit tedious - and it will show immediately where API calls will be > needed as well. > > So that work is not **entirely* useless. > > J. > > On Fri, Jul 19, 2024 at 6:59 PM Vikram Koka <vik...@astronomer.io.invalid> > wrote: > >> I am very relieved with the "experimental, don't try this at home" label >> on >> this body of work, since this will be replaced shortly. >> >> If there is belief that this will help accelerate learning towards AIP-69 >> (Remote Executor), I understand the rationale for proceeding forward with >> this given the caveats above. >> >> Given that, I don't have strong feelings either way on this. I am ok with >> the conclusion above of releasing it largely "as is" to be used as a test >> bed before AIP-72 is released as part of 3.0. >> >> >> >> On Thu, Jul 18, 2024 at 11:54 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> >> > Remote Executor not worker. >> > >> > On Thu, Jul 18, 2024 at 8:52 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> > >> > > My conclusion is (I spoke to Jens about it today) that we should >> attempt >> > > to get a "workable, not necessarily optimized for performance, and >> > > potentially having a few things not working version of it" and label >> it >> > " >> > > highly experimental, don't do it at home" - rather similar to what >> Jens' >> > > approach is for the "pre" release of remote worker. Jens already has a >> > > working Remote vorker on it, so having the base implemented in Airflow >> > 2.10 >> > > might be for him (and few other brave souls) who would want to test >> > remote >> > > worker on "2.10" version. And maybe we will also find a thing or two >> that >> > > might be useful for AIP-72 implementation while doing it. >> > > >> > > J. >> > > >> > > On Thu, Jul 18, 2024 at 8:44 PM Daniel Standish >> > > <daniel.stand...@astronomer.io.invalid> wrote: >> > > >> > >> I don't have strong feelings here either way, but we should probably >> try >> > >> to >> > >> arrive at a conclusion here sooner than later because meanwhile work >> on >> > >> this continues. >> > >> >> > >> I think the question is sort of, is anyone going to essentially veto >> it, >> > >> or >> > >> perhaps veto it unless certain conditions are met. E.g. it has been >> > >> proposed that the feature must be marked as experimental so as to not >> > >> increase support burden on what is perhaps a dead end feature. >> > >> >> > >> Any other sort of proposals or questions outstanding on this one? >> > >> >> > >> >> > >> >> > >> On Mon, Jul 15, 2024 at 11:39 AM Amogh Desai < >> amoghdesai....@gmail.com> >> > >> wrote: >> > >> >> > >> > I'd prefer to move it completely to Airflow 3, because I don't >> really >> > >> see >> > >> > spending cycles >> > >> > on developing something for the short term here. >> > >> > >> > >> > With Airflow 3, it would come out more complete and something that >> > >> could be >> > >> > usable by users without >> > >> > having to put a halt to it and starting it again. >> > >> > >> > >> > I understand that it would affect the AIP 69 but as I whole i don't >> > see >> > >> > doing AIP 44 for Airflow 2 as beneficial. >> > >> > >> > >> > >> > >> > Thanks & Regards, >> > >> > Amogh Desai >> > >> > >> > >> > >> > >> > On Fri, Jul 12, 2024 at 10:39 PM Ash Berlin-Taylor <a...@apache.org >> > >> > >> wrote: >> > >> > >> > >> > > -0.25 to me continuing - I don't see the value long term in >> putting >> > >> more >> > >> > > effort into this, and any lessons from this have already been >> > learnt, >> > >> as >> > >> > > development on Airflow 3 has already started, so this would >> > literally >> > >> be >> > >> > > happening in parallel. >> > >> > > >> > >> > > Given we are planning, at most, 1 or two more releases of >> Airflow 2 >> > >> this >> > >> > > feature will have to stay marked experimental so that it doesn't >> > >> increase >> > >> > > our support burden. >> > >> > > >> > >> > > My TL;Dr: I see this feature as a dead end now, but if you want >> to >> > >> work >> > >> > on >> > >> > > it by my guest, as long as it doesn't cause us more (support) >> work >> > in >> > >> the >> > >> > > long term. >> > >> > > >> > >> > > -ash >> > >> > > >> > >> > > On 12 July 2024 14:20:14 BST, "Michał Modras" < >> > >> michalmod...@google.com >> > >> > .INVALID> >> > >> > > wrote: >> > >> > > >I think we should finish the AIP within Airflow 2 - it will take >> > time >> > >> > > until >> > >> > > >Airflow 3 is out, and I believe some learnings from finishing >> and >> > >> > running >> > >> > > >this AIP might be useful for Airflow 3. We plan to contribute to >> > >> > finishing >> > >> > > >this AIP. >> > >> > > > >> > >> > > >On Fri, Jul 12, 2024 at 7:52 AM Scheffler Jens (XC-AS/EAE-ADA-T) >> > >> > > ><jens.scheff...@de.bosch.com.invalid> wrote: >> > >> > > > >> > >> > > >> Hi, >> > >> > > >> >> > >> > > >> I'd favor to make it usable - especially as we are at 80%. >> > >> > > >> >> > >> > > >> Main motivation is that with our environment we see stability >> > >> problems >> > >> > > >> with the distributed setup and using Celery, which was the >> main >> > >> > > motivation >> > >> > > >> to spin the discussion about AIP-69. AIP-69 is depending on >> the >> > >> > feature. >> > >> > > >> Waiting another 12-18 months to be able to host a stable >> > >> distributed >> > >> > > setup >> > >> > > >> based on Airflow 3 is something hard to argue. And I can >> confirm >> > >> it is >> > >> > > >> working already in my AIP-69 PoC. >> > >> > > >> >> > >> > > >> In this light I could offer to move it to at least the level >> that >> > >> it >> > >> > can >> > >> > > >> be used and is properly CI tested as using it for AIP-69 as >> first >> > >> > > consumer >> > >> > > >> (which could reduce the scope to task execution, DAG parsing >> and >> > >> > > triggered >> > >> > > >> could be taken out-of-scope for AIP-69 dependency for >> example). I >> > >> > could >> > >> > > >> offer supporting to close the gaps to completion. >> > >> > > >> >> > >> > > >> In regards of workload the completion should be a target >> before >> > the >> > >> > > >> cut-off to Airflow 3, so I would assume only "keeping the >> lights >> > >> on" >> > >> > > would >> > >> > > >> be a distraction while developing Airflow 3. >> > >> > > >> >> > >> > > >> Mit freundlichen Grüßen / Best regards >> > >> > > >> >> > >> > > >> Jens Scheffler >> > >> > > >> >> > >> > > >> Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) >> > >> > > >> Robert Bosch GmbH | Hessbruehlstraße 21 | 70565 >> > >> Stuttgart-Vaihingen | >> > >> > > >> GERMANY | www.bosch.com >> > >> > > >> Tel. +49 711 811-91508 <+49%20711%2081191508> | Mobil +49 160 >> > >> 90417410 >> > >> > > >> <+49%20160%2090417410> | jens.scheff...@de.bosch.com >> > >> > > >> >> > >> > > >> Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB >> > 14000; >> > >> > > >> Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; >> > >> > > >> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, >> Dr. >> > >> > Markus >> > >> > > >> Forschner, >> > >> > > >> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja >> > Rückert >> > >> > > >> >> > >> > > >> -----Original Message----- >> > >> > > >> From: Jed Cunningham <jedcunning...@apache.org> >> > >> > > >> Sent: Thursday, July 11, 2024 10:41 PM >> > >> > > >> To: dev@airflow.apache.org >> > >> > > >> Subject: Re: [DISCUSS] To AIP-44 or not to AIP-44 >> > >> > > >> >> > >> > > >> It feels a little weird to add a new "forever" experimental >> > >> feature in >> > >> > > >> Airflow 2 that we already know won't be there in Airflow 3. >> Not >> > >> > > something >> > >> > > >> I'd want to be really user facing at this point in time >> either. >> > >> > > >> >> > >> > > >> Given the short timeline for Airflow 3, I imagine we'd be >> better >> > >> off >> > >> > > >> spending those cycles elsewhere. My 2c - not my cycles :) >> > >> > > >> >> > >> > > >> > >> > > >> > --------------------------------------------------------------------- >> > >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> > >> > > For additional commands, e-mail: dev-h...@airflow.apache.org >> > >> > > >> > >> > > >> > >> > >> > >> >> > > >> > >> >
