eajimmy commented on issue #69048:
URL: https://github.com/apache/airflow/issues/69048#issuecomment-4916748456
Same problem here on Airflow 3.2.1 (KubernetesExecutor + CeleryExecutor).
I confirmed it with this query, running it repeatedly:
```sql
SELECT id, hostname, state,
latest_heartbeat,
EXTRACT(EPOCH FROM (now() - latest_heartbeat)) AS age_seconds
FROM job
WHERE job_type = 'DagProcessorJob'
ORDER BY latest_heartbeat DESC
LIMIT 30;
```
age_seconds oscillates continuously between ~30 and ~60s and never drops
close to 0.
Setting AIRFLOW__DAG_PROCESSOR__HEALTH_CHECK_THRESHOLD=120 fixes it, as
mentioned above.
I also tried increasing parsing_processes and min_file_process_interval, but
these had no effect on age_seconds.
It seems that the staleness is intrinsic to the heartbeat mechanism, not to
parsing load.
I even confirmed this with the processor fully idle
(min_file_process_interval=300, no files to parse) and the age still stayed in
the 30–60s range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
