meel-ayush opened a new issue, #76552:
URL: https://github.com/apache/airflow/issues/76552

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.4.0rc3 (and main)
   
   ### What happened and how to reproduce it?
   
   ### Description
   
   When `bundle.refresh()` fails in the DAG processor manager (such as a git 
network timeout, GitHub/GitLab rate limiting, or expired credentials), 
`DagFileProcessorManager._refresh_dag_bundles()` catches the exception and logs 
it to daemon logs, but records nothing in the metadata database.
   
   Exactly 0 rows are written to `ParseImportError` or `DagWarning`. Because of 
this, the Airflow Web UI displays 0 import errors and reports the environment 
as healthy, even though DAG synchronization from the bundle has stopped 
completely.
   
   ### Affected Code
   
   In `airflow-core/src/airflow/dag_processing/manager.py:934-939`:
   
   ```python
   try:
       bundle.refresh()
       any_refreshed = True
   except Exception:
       self.log.exception("Error refreshing bundle %s", bundle.name)
       continue
   ```
   
   ### Steps to Reproduce
   
   1. Configure a `GitDagBundle` pointing to a remote repository.
   2. Trigger a connection failure (`ConnectionResetError`) or authentication 
failure during `bundle.refresh()`.
   3. The DAG processor logs `Error refreshing bundle ...` to stderr and 
continues its loop.
   4. Check the metadata DB: 0 rows in `import_error` (`ParseImportError`) and 
0 rows in `dag_warning` (`DagWarning`).
   5. The UI shows 0 import errors while the bundle remains out of date.
   
   ### Expected Behavior
   
   When a bundle fails to refresh, a `ParseImportError` or `DagWarning` (e.g. 
`DagWarningType.BUNDLE_REFRESH_FAILED`) should be recorded in the database for 
that bundle so operators can see the failure directly on the UI dashboard.
   
   ### Proposed Fix
   
   In `airflow-core/src/airflow/dag_processing/manager.py`, catch the exception 
in `_refresh_dag_bundles` and persist a warning or import error row to the 
database.
   
   I have verified this behavior with a local test and am working on a PR with 
regression tests to address this.
   
   
   ### What you think should happen instead?
   
   _No response_
   
   ### Operating System
   
   Kali GNU/Linux
   
   ### Deployment
   
   None
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   _No response_
   
   ### Helm Chart configuration
   
   _No response_
   
   ### Docker Image customizations
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to