nickstenning commented on code in PR #73244:
URL: https://github.com/apache/airflow/pull/73244#discussion_r4027538899
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -2097,13 +2097,19 @@ workers:
max_failed_heartbeats:
description: |
The maximum number of consecutive failed heartbeats before terminating
the task instance process.
+
+ Unlike other calls to the execution API, heartbeats are not retried.
This setting and
+ ``min_heartbeat_interval`` together determine how long a worker
tolerates an unreachable API server
+ before killing running tasks.
version_added: 3.0.0
type: integer
example: ~
- default: "3"
+ default: "20"
Review Comment:
On `main` today, if the apiserver vanishes into thin air, the retry policy
combined with `execution_api_timeout` mean that one heartbeat attempt can block
the supervisor loop for about 40s. Multiply that by three (the default value of
`max_failed_heartbeats`) and you get a total of about `120s` before a task
stops running.
With this change, it would be `execution_api_timeout × max_failed_heartbeats
= 5s × 20 = 100s` before a task stops running.
Luckily, both of these are well under the default value of
`task_instance_heartbeat_timeout`, which is `300s`.
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -2097,13 +2097,19 @@ workers:
max_failed_heartbeats:
description: |
The maximum number of consecutive failed heartbeats before terminating
the task instance process.
+
+ Unlike other calls to the execution API, heartbeats are not retried.
This setting and
+ ``min_heartbeat_interval`` together determine how long a worker
tolerates an unreachable API server
+ before killing running tasks.
version_added: 3.0.0
type: integer
example: ~
- default: "3"
+ default: "20"
Review Comment:
On `main` today, if the apiserver vanishes into thin air, the retry policy
combined with `execution_api_timeout` mean that one heartbeat attempt can block
the supervisor loop for about `40s`. Multiply that by three (the default value
of `max_failed_heartbeats`) and you get a total of about `120s` before a task
stops running.
With this change, it would be `execution_api_timeout × max_failed_heartbeats
= 5s × 20 = 100s` before a task stops running.
Luckily, both of these are well under the default value of
`task_instance_heartbeat_timeout`, which is `300s`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
