This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 00ab45ffb7 Fix dag file processor hertbeat to run only if necessary
(#30899)
00ab45ffb7 is described below
commit 00ab45ffb7dee92030782f0d1496d95b593fd4a7
Author: Jarek Potiuk <[email protected]>
AuthorDate: Thu Apr 27 13:27:33 2023 +0200
Fix dag file processor hertbeat to run only if necessary (#30899)
The standalone file processor as of #30278 introduced accidentally
an artifficial delay between dag processing by adding heartbeat
but missing to set "only_if_necessary" flag to True.
If your dag file processing has been fast (faster than the
scheduler job_heartbeat_sec) this introduced unnecessary pause
between the next dag file processor loop (up until the time
passed), it also introduced inflation of the
dag_processing_last_duration metrics (it would always show minimum
job_heartbeat_sec)
Adding "only_if_necessary" flag fixes the problem.
Fixes: #30593
Fixes: #30884
---
airflow/jobs/dag_processor_job_runner.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/airflow/jobs/dag_processor_job_runner.py
b/airflow/jobs/dag_processor_job_runner.py
index 0fda9d9373..afc8f544da 100644
--- a/airflow/jobs/dag_processor_job_runner.py
+++ b/airflow/jobs/dag_processor_job_runner.py
@@ -57,7 +57,7 @@ class DagProcessorJobRunner(BaseJobRunner, LoggingMixin):
self.processor.heartbeat = lambda: perform_heartbeat(
job=self.job,
heartbeat_callback=empty_callback,
- only_if_necessary=False,
+ only_if_necessary=True,
)
def _execute(self) -> int | None:
