AtharvaSarwate opened a new issue, #69983:
URL: https://github.com/apache/airflow/issues/69983
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.0
### What happened and how to reproduce it?
The `dag_processing.processes` metric in `DagFileProcessorManager` is
intended to
track the number of currently running Dag parsing processes, but it is
missing a
decrement on the normal (graceful) completion path.
### Current behaviour
`stats.incr("dag_processing.processes", tags={"action": "start", ...})` is
called
in `_start_new_processes()` when a child `DagFileProcessorProcess` is
spawned.
A matching `stats.decr(...)` is called in three abnormal exit paths:
- `terminate_orphan_processes()` — action: `"stop"` (file deleted mid-parse)
- `_kill_timed_out_processors()` — action: `"timeout"` (exceeded
`processor_timeout`)
- `terminate()` — action: `"terminate"` (manager shutting down)
However, `_collect_results()` — the **normal graceful exit path** called
every loop
iteration — pops the finished processor from `self._processors` and calls
`processor.close()` with **no corresponding `decr`**:
This means under normal operation (parsers completing successfully), the
metric value drifts upward indefinitely and never accurately reflects the
number of currently running processes
### What you think should happen instead?
A stats.decr("dag_processing.processes", tags={"action": "finish", ...})
should be emitted when a process completes gracefully, symmetric with the incr
in _start_new_processes().
### Operating System
_No response_
### Deployment
Docker-Compose
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
