Comment on that one too - we also provided more details (including possible deployment options). Similarly as in case of AIP-44 we had a few goals - which also resulted from earlier discussions:
* Keep current behavior practically untouched * Allow for different deployments "no separation", "dag processor separation", "multiple dag processors for tenant isolation" * the "multiple dag processor" case does NOT change the ways dags are stored or parsed or where dags are stored. They are all parts of "DAG_FOLDER". The change is merely there to allow each dag processor to be "responsible" for parsing only selected part of the "dag" tree. This is purely to isolate code execution between tenants and has no impact whatsoever on other components (and it has no impact on other ideas there - such as DAG fetcher or DAG submission). It's purely to allow the "deployment" to better isolate code execution of user code coming from different tenants. I hope this one will be faster to discuss and raise less controversies/doubts - but feel absolutely free to comment/criticize etc. J. On Thu, Dec 9, 2021 at 4:36 PM Mateusz Henc <[email protected]> wrote: > > Hi, > We just updated the AIP-43 adding some details and deployment options. > Please take a look. Any comments are appreciated. > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation > > BTW Update for AIP-44 is coming soon. > > Best regards, > Mateusz Henc > > > On Thu, Dec 2, 2021 at 3:04 PM Jarek Potiuk <[email protected]> wrote: >> >> Excited :) >> >> On Thu, Dec 2, 2021 at 2:20 PM Mateusz Henc <[email protected]> wrote: >> > >> > Hi, >> > I just submitted my proposal for separating the DagProcessor from >> > Scheduler and running it as an independent component. This is part of >> > AIP-1 Airflow multi-tenancy effort. >> > >> > PTAL >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+Dag+processor+separation >> > >> > Best regards, >> > Mateusz Henc
