kalluripradeep opened a new pull request, #61017: URL: https://github.com/apache/airflow/pull/61017
Fixes #60928 ## Problem The `dag_processing.import_errors` metric incorrectly reports errors when DAG callbacks (e.g., `on_failure_callback`) are executed in the DAG processor, even when there are no actual import errors. This causes false alerts for operators monitoring DAG health, as the metric is incremented during normal callback processing. **Root Cause:** When callback-only processing occurs (`is_callback_only=True`), the code creates a `DagFileStat` but doesn't explicitly set `import_errors=0`. The old value persists in `self._file_stats[file]`, and when metrics are emitted, it sums all `stat.import_errors` values, including stale ones from previous processing. ## Solution Explicitly set `import_errors=0` when creating `DagFileStat` for callback-only processing. This ensures that: 1. Callback execution doesn't increment the import_errors metric 2. Old import_errors values are cleared for callback-only runs 3. The metric accurately reflects only actual DAG parsing errors ## Changes - Updated `process_parse_results()` function in `airflow-core/src/airflow/dag_processing/manager.py` - Added `import_errors=0` parameter when creating `DagFileStat` for callback-only processing (line 1188) - Added comment explaining the fix ## Testing **Manual reproduction (as described in issue):** 1. Run 5-10 DAGs with multiple tasks in parallel 2. Ensure all tasks have failure callbacks 3. Kill celery workers and wait for callbacks to run in dag-processor 4. Check `dag_processing.import_errors` metric **Expected result:** Metric should remain 0 when callbacks run successfully **Before fix:** Metric incorrectly increments during callback processing **After fix:** Metric only increments for actual import errors ## Related - Addresses the issue reported by @deepak4babu - Improves accuracy of DAG processing metrics for monitoring -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
