nickstenning opened a new pull request, #73244:
URL: https://github.com/apache/airflow/pull/73244
Heartbeats had two stacked retry mechanisms: the Task SDK API client retried
every request, including heartbeats, and the supervisor separately counted
consecutive heartbeat failures against `[workers] max_failed_heartbeats`. The
client retries also had the unwanted side-effect of blocking the supervisor's
monitor loop while it backed off, so during an API server outage the supervisor
would be delayed forwarding task logs and/or noticing that the task subprocess
had exited.
Heartbeat retries are now governed solely by `max_failed_heartbeats`, and
I've adjusted the default value of that tunable from 3 to 20 to keep roughly
the same tolerance for an unreachable API server as the two mechanisms combined
gave before.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
