shivanshs9 commented on issue #10341: URL: https://github.com/apache/airflow/issues/10341#issuecomment-675642866
> That will refresh DAGs from DB if DAG Serialization is enabled, if not it will refresh them from DAG files I understand that but I'd again like to confirm that it'll only refresh the DAGs for the in-memory instance of the DagBag for that specific gunicorn worker process that received the request. With more than 1 web worker, trying to open DAG details or trigger will still randomly fail since the "refresh all" request may be POSTed to some other worker. > I understand that waiting for all airflow workers to restart and tweaking `worker_refresh_interval` config would help here. After all, the issue is due to in-memory instances of DagBag not able to collect the new DAG beforehand. While a restart would help, I propose that there could be a configuration bool option like `attempt_refresh_dagbag` (by default it is `False` for backwards compatibility). If it is `True` and if DagBag doesn't have the DAG loaded (in this case, `DagBag.get_dag()` returns `None`), it would attempt to load the DAG directly by processing the file stored in the `DagModel`. This would be a better option for those who'd like to not wait for the new DAG to sync with all workers. Plus, they can focus on improving performance by increasing the `worker_refresh_interval` value and still work with the new DAGs ASAP. What I meant above for a catch-all handler (optional and off by default) to get around this randomness of the bug. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
