Hello, I have a feeling this plugin doesn't work when DAG serialization is enabled. https://github.com/ms32035/airflow-dag-dependencies/blob/30108e750e2008bc5f043210f8c7fa728fe5a630/dag-dependencies-plugin/dag_dependencies_plugin.py#L65 In Airflow 2.0, serialization is required and cannot be disabled.
> Be careful if you have 1000s of DAGs, might not scale. This is also problematic. It seems to me that we can come up with a solution that will scale, e.g. when all information will be written to the database by the scheduler and the webserver will read this information. Best. regards, Kamil Breguła On Wed, Nov 25, 2020 at 1:22 AM Marcin Szymański <[email protected]> wrote: > Hi all > > Just had a chat with Ry and I've seen popular demand in the past, so I > guess it might be a good time and idea to contribute my DAG dependencies > plugin (https://github.com/ms32035/airflow-dag-dependencies) to core > Airflow. For that I'd need some community guidance on the following points: > 1) I guess the feature is not large enough to require AIP and simple PR > would be enough > 2) Right now the plugin refreshes the dependencies using last view time + > timeout. Any suggestions about better solutions? > 3) Are there any upcoming scheduler changes that might impact the > functionality, like partitioning etc? > 4) Any other caveats I might be missing? > > Thanks > Marcin > > >
