pingzh commented on PR #22917: URL: https://github.com/apache/airflow/pull/22917#issuecomment-1123056796
> Thoughts on deleting missing DAGs from the serialized DAGs table at the same time that it is marked inactive? > > https://github.com/apache/airflow/blob/cfa95af7e83b067787d8d6596caa3bc97f4b25bd/airflow/dag_processing/manager.py#L508-L515 > > There's some background in #21397 (Although I don't think the performance implications here are nearly as severe as in that case, as the serialized DAG table is properly indexed) > > This query is gonna run really frequently (once for every DAG per parse interval) and most of the time won't have any missing DAGs to delete, by performing this on an intermittent basis rather than constantly, we can reduce the volume of queries. > > This also would centralize similar pieces of code, rather than having similar functionality in completely separate parts of the codebase. @SamWheating thanks for the context. that is a good idea to keep them together. updating the pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
