Dear all, Last week we added AIP-19 (https://cwiki.apache.org/confluence/display/AIRFLOW/AIP+19+-+Making+the+webserver+stateless), which aims to address various stability issues in the Airflow webserver stemming from differences in DagBag state between the different processes of the webserver. These stability issues are illustrated in this video: https://youtu.be/sNrBruPS3r4.
Our AIP aims to solve these issues by moving DAG-related information into the database, rather than querying DAG metadata from the DagBag instance of the given webserver process. By obtaining this information from the database, we can ensure that there is a single-source-of-truth for DAG-related metadata, thus avoiding differences in state between webserver processes. To keep this AIP tractable, we propose to leverage the existing ORM models for storing and querying DAG metadata from the database. More information information on this AIP is available in cwiki. Feedback on the AIP is more than welcome! However, to keep the discussion centralized, I propose to discuss this AIP proposal in the comment section of cwiki. Best regards / met vriendelijke groet, Julian de Ruiter Machine learning engineer ▉▉▉▉▉▉▉ GoDataDriven Proudly part of the Xebia group M: +31 6 30 61 26 24 W: http://www.godatadriven.com
