Thanks. I'll check the impact of DAG serialization.
The scalability issues I was referring to would be more likely during UI rendering, but I never tested that at scale. On Wed, 25 Nov 2020, 01:20 Kamil Breguła, <[email protected]> wrote: > Hello, > > I have a feeling this plugin doesn't work when DAG serialization is > enabled. > > https://github.com/ms32035/airflow-dag-dependencies/blob/30108e750e2008bc5f043210f8c7fa728fe5a630/dag-dependencies-plugin/dag_dependencies_plugin.py#L65 > In Airflow 2.0, serialization is required and cannot be disabled. > > > Be careful if you have 1000s of DAGs, might not scale. > > This is also problematic. It seems to me that we can come up with a > solution that will scale, e.g. when all information will be written to the > database by the scheduler and the webserver will read this information. > > Best. regards, > Kamil Breguła > > On Wed, Nov 25, 2020 at 1:22 AM Marcin Szymański <[email protected]> > wrote: > >> Hi all >> >> Just had a chat with Ry and I've seen popular demand in the past, so I >> guess it might be a good time and idea to contribute my DAG dependencies >> plugin (https://github.com/ms32035/airflow-dag-dependencies) to core >> Airflow. For that I'd need some community guidance on the following points: >> 1) I guess the feature is not large enough to require AIP and simple PR >> would be enough >> 2) Right now the plugin refreshes the dependencies using last view time + >> timeout. Any suggestions about better solutions? >> 3) Are there any upcoming scheduler changes that might impact the >> functionality, like partitioning etc? >> 4) Any other caveats I might be missing? >> >> Thanks >> Marcin >> >> >>
