vincbeck commented on code in PR #76410:
URL: https://github.com/apache/airflow/pull/76410#discussion_r4643753830
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -2174,6 +2437,21 @@ def _run_scheduler_loop(self) -> None:
non_fatal=True,
)
+ timers.call_regular_interval(
+ # A floor, not just self.job.heartrate directly: that can be
configured down to 0 (and
+ # tests commonly do), and a zero delay re-queues itself at "now"
on every single pass
+ # through the event scheduler, which never lets a non-blocking
run() return.
+ delay=max(self.job.heartrate, 1.0),
+ action=self._touch_executor_heartbeats,
+ non_fatal=True,
+ )
+
+ timers.call_regular_interval(
+ delay=self._executor_health_check_interval,
Review Comment:
Setting `executor_health_check_interval` to 0 makes the scheduler loop spin
forever, same issue you fixed for the heartbeat timer above. Can we register
both timers only when `self._executor_health_keys` is not empty and the
interval is > 0, like `stale_bundle_cleanup_interval`?
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -456,6 +513,182 @@ def __init__(
def heartbeat_callback(self, *, session: Session = NEW_SESSION) -> None:
stats.incr("scheduler_heartbeat", 1, 1)
+ def _check_executor_health(self) -> None:
+ """
+ Kick off a health check for each reporting executor, in the background.
+
+ Called on its own timer (``[scheduler]
executor_health_check_interval``), independent of the
+ scheduling loop and of each executor's own
:meth:`~airflow.executors.base_executor.BaseExecutor.heartbeat`.
+ A new check for a given executor is skipped while its previous check
is still running, so a
+ slow or hung dependency can only ever delay that executor's own health
being updated, never
+ pile up work or block this (or any other) scheduler tick.
+
+ ``get_health()`` is passed a timeout, but an implementation that
ignores it can still leave its
+ future pending forever. Once a pending future is a few intervals old,
it's reported as stalled:
+ the scheduler records UNHEALTHY itself, since a check that hasn't
answered isn't still-healthy,
+ and logs and emits a metric so the stall is visible. If the stuck call
does eventually return,
+ its result overwrites this one.
+ """
+ for executor in self.executors:
+ key = _executor_health_key(executor)
+ if key is None or key not in self._executor_health_keys:
+ # No name, or doesn't override get_health(): there's nothing
to check or record.
+ continue
+ future, started = self._executor_health_check_futures.get(key,
(None, 0.0))
+ if future is not None and not future.done():
+ stall_after = (
+ self._executor_health_check_interval *
self._executor_health_check_stall_after_intervals
+ )
+ if time.monotonic() - started > stall_after and key not in
self._stalled_health_checks:
+ self._stalled_health_checks.add(key)
+ self.log.warning("Health check for executor %s has stopped
responding", key)
+ stats.incr(
+ "scheduler.executor_health_check.stalled",
Review Comment:
This metric needs an entry in
`shared/observability/src/airflow_shared/observability/metrics/metrics_template.yaml`,
otherwise the `check-metrics-synced-with-registry` hook fails.
##########
airflow-core/src/airflow/api_fastapi/core_api/datamodels/monitor.py:
##########
@@ -75,10 +86,25 @@ class DagProcessorInfoResponse(BaseInfoResponse):
instances: list[DagProcessorInstanceInfoResponse] | None = None
+class ExecutorInfoResponse(BaseInfoResponse):
+ """
+ Executor info serializer for responses.
+
+ Only present when at least one configured executor implements a health
check; an executor that
+ doesn't isn't a gap in coverage, so a deployment where nothing reports
simply has no ``executor``
+ key at all, rather than one that always reads unhealthy.
+ """
+
+ latest_executor_heartbeat: str | None
+ detailed_status: DetailedHealthStatus | None
+ instances: list[ExecutorInstanceInfoResponse] | None = None
+
+
class HealthInfoResponse(BaseModel):
"""Health serializer for responses."""
metadatabase: BaseInfoResponse
scheduler: SchedulerInfoResponse
triggerer: TriggererInfoResponse
dag_processor: DagProcessorInfoResponse | None = None
+ executor: ExecutorInfoResponse | None = None
Review Comment:
Same as the OpenAPI spec, the airflowctl datamodels
(`airflow-ctl/src/airflowctl/api/datamodels/generated.py`) need to be
regenerated. The `generate-airflowctl-datamodels` prek hook does it.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -435,6 +454,44 @@ def __init__(
self.executors: list[BaseExecutor] = executors if executors else
ExecutorLoader.init_executors()
self.executor: BaseExecutor = self.executors[0]
+ # Each executor's get_health() runs on its own daemon thread, never on
the scheduler loop's own
+ # thread: a check that talks to a broker or a cluster control plane
can be slow or hang, and
+ # must not be able to delay task scheduling. A pool isn't needed since
each executor is only
+ # ever checked by one in-flight call at a time (see
_check_executor_health); a daemon thread
+ # also means a hung check can't block process exit the way a
ThreadPoolExecutor worker would.
+ self._executor_health_check_futures: dict[str, tuple[Future, float]] =
{}
+ # A probe that ignores the timeout passed to it would otherwise occupy
its slot forever: once
+ # a pending future has been sitting this many intervals, it's reported
as stalled instead of
+ # silently skipped on every later tick (see _check_executor_health).
+ self._executor_health_check_stall_after_intervals = 3
+ self._stalled_health_checks: set[str] = set()
+
+ self._executor_health_check_interval = conf.getfloat(
+ "scheduler", "executor_health_check_interval", fallback=30.0
+ )
+ # Must be comfortably below the check interval so a probe that honours
it always finishes
+ # before the next tick fires; half the interval is the starting point.
+ self._executor_health_check_timeout =
self._executor_health_check_interval / 2
+
+ # Only executors that actually override get_health() get a Job row at
all: one that was never
+ # going to report anything isn't a gap in coverage (see
_configured_executor_keys in
+ # airflow_health.py, which applies this same rule from the
/monitor/health side). An executor
+ # with no name is also skipped: _executor_health_key has nothing
stable to key its row under.
+ # issubclass(type(executor), ...), not isinstance(executor, ...): a
test double standing in
+ # for an executor (a MagicMock, even one speced to BaseExecutor)
passes isinstance, since mock
+ # fakes that check, but calling a method on it never runs the real
classmethod body, it just
+ # hands back another mock, which this dict would then read as
"overrides it". type() bypasses
+ # that trick and correctly reports such a double as not a real
BaseExecutor subclass.
+ from airflow.executors.base_executor import BaseExecutor as
_BaseExecutor
+
+ self._executor_health_keys: dict[str, str | None] = {
+ key: executor.team_name
+ for executor in self.executors
+ if issubclass(type(executor), _BaseExecutor)
Review Comment:
This is still production code working around mocks. `self.executors` only
holds real executors, so can we call `executor.overrides_get_health()` here and
set `overrides_get_health.return_value = False` on the mocks in the tests? That
also removes the import inside `__init__`.
##########
airflow-core/tests/unit/jobs/test_scheduler_job_executor_health.py:
##########
@@ -0,0 +1,280 @@
+from __future__ import annotations
Review Comment:
Missing the license header
```suggestion
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
from __future__ import annotations
```
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -3912,6 +4190,23 @@ def adopt_or_reset_orphaned_tasks(self, *, session:
Session = NEW_SESSION) -> in
self.log.info("Marked %d SchedulerJob instances as
failed", num_failed)
stats.incr(self.__class__.__name__.lower() + "_end",
num_failed)
+ # A separate statement, not folded into the SchedulerJob
one above, so the
+ # "Marked %d SchedulerJob instances as failed" log line
and the _end metric keep
+ # counting only schedulers. These ExecutorJob rows are
left running with no
+ # end_date when their owning scheduler is SIGKILLed or
OOM-killed, since _execute's
+ # own cleanup (_end_executor_jobs) never gets to run for
that case; this also
+ # catches a row a health check still in flight at that
moment recreates after
+ # _end_executor_jobs has already closed the others.
+ session.execute(
Review Comment:
Can you add a test for this? Nothing checks that a stale `ExecutorJob` row
gets marked as failed.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -435,6 +454,44 @@ def __init__(
self.executors: list[BaseExecutor] = executors if executors else
ExecutorLoader.init_executors()
self.executor: BaseExecutor = self.executors[0]
+ # Each executor's get_health() runs on its own daemon thread, never on
the scheduler loop's own
Review Comment:
nit: most of the new comments are very long and explain the review history
rather than the code. Can you trim them? `AGENTS.md` asks for short comments,
only when the context is not obvious from the code.
##########
airflow-core/src/airflow/api/common/airflow_health.py:
##########
@@ -54,13 +58,15 @@ class DetailedHealthStatus(str, Enum):
@provide_session
-def get_jobs_health(job_runner_class, *, session: Session = NEW_SESSION) ->
list[Job]:
+def get_jobs_health(job_runner_class: type | str, *, session: Session =
NEW_SESSION) -> list[Job]:
"""
- Return live unfinished jobs and the latest unfinished job, ordered by
heartbeat.
+ Return live unfinished jobs and the latest unfinished job of the given job
type, ordered by heartbeat.
- Keep the latest job even when dead so health responses retain its
heartbeat.
+ Keep the latest job even when dead so health responses retain its
heartbeat. ``job_runner_class`` is
+ a job runner class that reports its own ``job_type`` (e.g.
``SchedulerJobRunner``), or a bare
+ job_type string for a job with no runner class of its own, like the
executor's ``EXECUTOR_JOB_TYPE``.
"""
- job_type = job_runner_class.job_type
+ job_type = getattr(job_runner_class, "job_type", job_runner_class)
Review Comment:
nit: why `getattr`? Taking a `job_type: str` and calling
`get_jobs_health(SchedulerJobRunner.job_type)` would be simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]