Happy New Year everyone: Just a short reminder we have the AIP-43 up for discussion - Mateusz is back in a few days, but I am happy to discuss details there: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation
J. On Sat, Dec 18, 2021 at 6:34 PM Jarek Potiuk <[email protected]> wrote: > > Just as a comment - I think a number of people expressed that they > want to comment on AIP-43 / AIP-44. Mateusz mentioned that also maybe > some people will have more time during the slower period around Xmas - > so feel free to make your comments still - we aim - if possible - to > have the discussions conclude ~ mid Jan and that should give enough > time for different contributors. Do not wait till the last moment > though and feel free to comment on both AIPs as soon as you can. I > will be around also to answer those. > > On Wed, Dec 15, 2021 at 9:47 PM Mateusz Henc <[email protected]> wrote: > > > > Hi, > > FYI I start my vacation on Monday and I am off for 3 weeks (back on 01/10) > > with limited access to my laptop. > > > > If you have any comments please add by the end of this week, so I could > > address them while I am still there. > > If you are fine then maybe we could start voting soon? > > > > // Leaving more time for AIP-44 as it is longer and more complicated. > > > > Best regards, > > Mateusz Henc > > > > > > On Tue, Dec 14, 2021 at 1:09 PM Jarek Potiuk <[email protected]> wrote: > >> > >> Comment on that one too - we also provided more details (including > >> possible deployment options). Similarly as in case of AIP-44 we had a > >> few goals - which also resulted from earlier discussions: > >> > >> * Keep current behavior practically untouched > >> * Allow for different deployments "no separation", "dag processor > >> separation", "multiple dag processors for tenant isolation" > >> * the "multiple dag processor" case does NOT change the ways dags are > >> stored or parsed or where dags are stored. They are all parts of > >> "DAG_FOLDER". The change is merely there to allow each dag processor > >> to be "responsible" for parsing only selected part of the "dag" tree. > >> This is purely to isolate code execution between tenants and has no > >> impact whatsoever on other components (and it has no impact on other > >> ideas there - such as DAG fetcher or DAG submission). It's purely to > >> allow the "deployment" to better isolate code execution of user code > >> coming from different tenants. > >> > >> I hope this one will be faster to discuss and raise less > >> controversies/doubts - but feel absolutely free to comment/criticize > >> etc. > >> > >> J. > >> > >> On Thu, Dec 9, 2021 at 4:36 PM Mateusz Henc <[email protected]> > >> wrote: > >> > > >> > Hi, > >> > We just updated the AIP-43 adding some details and deployment options. > >> > Please take a look. Any comments are appreciated. > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation > >> > > >> > BTW Update for AIP-44 is coming soon. > >> > > >> > Best regards, > >> > Mateusz Henc > >> > > >> > > >> > On Thu, Dec 2, 2021 at 3:04 PM Jarek Potiuk <[email protected]> wrote: > >> >> > >> >> Excited :) > >> >> > >> >> On Thu, Dec 2, 2021 at 2:20 PM Mateusz Henc <[email protected]> > >> >> wrote: > >> >> > > >> >> > Hi, > >> >> > I just submitted my proposal for separating the DagProcessor from > >> >> > Scheduler and running it as an independent component. This is part of > >> >> > AIP-1 Airflow multi-tenancy effort. > >> >> > > >> >> > PTAL > >> >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation > >> >> > > >> >> > Best regards, > >> >> > Mateusz Henc
