ferruzzi commented on code in PR #76410:
URL: https://github.com/apache/airflow/pull/76410#discussion_r4606228936


##########
airflow-core/src/airflow/api/common/airflow_health.py:
##########
@@ -236,6 +339,24 @@ def get_airflow_health() -> dict[str, Any]:
         dag_processor_status = HealthStatus.UNHEALTHY
         dag_processor_detailed_status = DetailedHealthStatus.DOWN
 
+    # Omitted entirely, not just reported unhealthy, when no configured 
executor implements a health
+    # check: a reporting mechanism nothing has opted into isn't a gap in 
coverage, and showing it as
+    # "unhealthy" on every deployment that hasn't adopted it yet would be a 
false alarm, not a signal.
+    report_executor_health = bool(_configured_executor_keys())

Review Comment:
   `_configured_executor_keys()` imports every configured executor class, and 
this runs every time `/monitor/health` is hit, even when nothing overrides 
`get_health()`.  The PR description says the change is inert until something 
overrides `get_health()`, but this part isn't: nothing else in the API server 
imports executor classes (`connections.py` only reads their names), and this 
does it on every request whether or not anything overrides the method.
   
   
   This is going to cause two problems:
   
   - If an import fails on the API server, the warning and traceback at line 
220 get logged on every request since Python retries a failed import each time. 
 Anything polling this endpoint turns that into a traceback every few seconds.
   - Each request computes the keys 3 times: here, `_executor_detailed_status`, 
and `_executor_legacy_status`.  The live-scheduler query in 
`_live_executor_jobs` also runs three times.
   
   The configured executors can't change while the process runs, so I'd cache 
the keys with `@lru_cache` (see `health_check_threshold()` in `job.py` for an 
example, and the `cleanup_jobs fixture` in `TestAirflowHealthFromDb` shows how 
the tests reset it) and call `get_executor_names(validate_teams=False)` the way 
`connections.py` does.  If we skip team validation, then a database hiccup on 
the first call can't get cached as "no executors".  Tests that patch 
`get_executor_names` will need `_configured_executor_keys.cache_clear()`.
   
   For the repeated status work, compute `detailed_status` once and derive the 
legacy status from it, which also lets `_executor_legacy_status` go away:
   
   ```python
             executor_detailed_status = _executor_detailed_status(executor_jobs)
             executor_status = (
                 HealthStatus.UNHEALTHY
                 if executor_detailed_status == DetailedHealthStatus.DOWN
                 else HealthStatus.HEALTHY
             )
   ```



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -425,6 +471,141 @@ def __init__(
     def heartbeat_callback(self, *, session: Session = NEW_SESSION) -> None:
         stats.incr("scheduler_heartbeat", 1, 1)
 
+    def _check_executor_health(self) -> None:
+        """
+        Kick off a health check for each reporting executor, in the background.
+
+        Called on its own timer (``[scheduler] 
executor_health_check_interval``), independent of the
+        scheduling loop and of each executor's own 
:meth:`~airflow.executors.base_executor.BaseExecutor.heartbeat`.
+        A new check for a given executor is skipped while its previous check 
is still running, so a
+        slow or hung dependency can only ever delay that executor's own health 
being updated, never
+        pile up work or block this (or any other) scheduler tick.
+
+        ``get_health()`` is passed a timeout, but an implementation that 
ignores it can still leave its
+        future pending forever. That future's own worker slot is never 
reclaimed -- the pool is sized
+        one worker per executor, so a stuck probe can only ever cost that 
executor's own slot -- but
+        without this check it would silently stop being checked again, with no 
sign anything was
+        wrong. Once a pending future is a few intervals old, it's reported as 
stalled instead.
+        """
+        for executor in self.executors:
+            key = _executor_health_key(executor)
+            if key not in self._executor_health_keys:
+                # Doesn't override get_health(); there's nothing to check or 
record.
+                continue
+            future, started = self._executor_health_check_futures.get(key, 
(None, 0.0))
+            if future is not None and not future.done():
+                stall_after = (
+                    self._executor_health_check_interval * 
self._executor_health_check_stall_after_intervals
+                )
+                if time.monotonic() - started > stall_after and key not in 
self._stalled_health_checks:
+                    self._stalled_health_checks.add(key)

Review Comment:
   When a check gets stuck, this warns once and emits the metric, which is 
good, but the row itself doesn't change.  `_touch_executor_heartbeats` keeps 
the heartbeat fresh, and `health_check_result` still holds whatever the last 
finished check returned.  If that was HEALTHY, then `/monitor/health` keeps 
reporting the executor as healthy for as long as the check stays stuck.
   
   For example, say a check ignores its timeout and hangs talking to a dead 
broker.  After three tries, the scheduler logs "has stopped responding", but 
anyone looking at `/monitor/health` or the dashboard still sees green.
   
   A stuck check hasn't answered, so I'd record it the same way as one that 
raised.  The scheduler already knows at this point, so it can write the result 
itself:
   
   ```
   
                 if time.monotonic() - started > stall_after and key not in 
self._stalled_health_checks:
                     self._stalled_health_checks.add(key)
                     self.log.warning("Health check for executor %s has stopped 
responding", key)
                     stats.incr("scheduler.executor_health_check.stalled", 
tags={"executor": key})
                     try:
                         self._record_executor_health_result(key, 
executor.team_name, HealthCheckResult.UNHEALTHY)
                     except Exception:
                         self.log.exception("Failed to record health check 
result for executor %s", key)
   ```
   
   If the stuck call does eventually return, its result overwrites this one, 
which is what we'd want.  A test that leaves a future pending past 
`stall_after` and checks that the row reads UNHEALTHY would cover it, and it 
would sit naturally with the `_check_executor_health` tests from the earlier 
thread.



##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -3018,6 +3018,18 @@ scheduler:
       type: float
       example: ~
       default: "120.0"
+    executor_health_check_interval:
+      description: |
+        How often, in seconds, the scheduler checks each configured executor's 
health via
+        ``get_health()``, reported on ``/monitor/health``. Each check runs in 
its own background
+        thread so a slow or unresponsive executor backend cannot delay task 
scheduling; this
+        setting controls how often a check is attempted, not how long a check 
is allowed to take.

Review Comment:
   Since the timeout became half the interval, the end of this sentence isn't 
true anymore; this setting now **does** control how long a check is allowed to 
take



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -425,6 +471,141 @@ def __init__(
     def heartbeat_callback(self, *, session: Session = NEW_SESSION) -> None:
         stats.incr("scheduler_heartbeat", 1, 1)
 
+    def _check_executor_health(self) -> None:
+        """
+        Kick off a health check for each reporting executor, in the background.
+
+        Called on its own timer (``[scheduler] 
executor_health_check_interval``), independent of the
+        scheduling loop and of each executor's own 
:meth:`~airflow.executors.base_executor.BaseExecutor.heartbeat`.
+        A new check for a given executor is skipped while its previous check 
is still running, so a
+        slow or hung dependency can only ever delay that executor's own health 
being updated, never
+        pile up work or block this (or any other) scheduler tick.
+
+        ``get_health()`` is passed a timeout, but an implementation that 
ignores it can still leave its
+        future pending forever. That future's own worker slot is never 
reclaimed -- the pool is sized
+        one worker per executor, so a stuck probe can only ever cost that 
executor's own slot -- but
+        without this check it would silently stop being checked again, with no 
sign anything was
+        wrong. Once a pending future is a few intervals old, it's reported as 
stalled instead.
+        """
+        for executor in self.executors:
+            key = _executor_health_key(executor)
+            if key not in self._executor_health_keys:
+                # Doesn't override get_health(); there's nothing to check or 
record.
+                continue
+            future, started = self._executor_health_check_futures.get(key, 
(None, 0.0))
+            if future is not None and not future.done():
+                stall_after = (
+                    self._executor_health_check_interval * 
self._executor_health_check_stall_after_intervals
+                )
+                if time.monotonic() - started > stall_after and key not in 
self._stalled_health_checks:
+                    self._stalled_health_checks.add(key)
+                    self.log.warning("Health check for executor %s has stopped 
responding", key)
+                    stats.incr("scheduler.executor_health_check.stalled", 
tags={"executor": key})

Review Comment:
   Nit:  you are tagging with the repr key here, but  
`scheduler.executor_heartbeat_duration` below tags with the class name and 
team_name, and uses `prune_dict`.  They should be the same; I'd recommend they 
both use the other version.



##########
airflow-core/tests/unit/jobs/test_scheduler_job_executor_health.py:
##########
@@ -0,0 +1,196 @@
+from __future__ import annotations
+
+import pytest
+from sqlalchemy import select
+
+from airflow.executors.base_executor import BaseExecutor
+from airflow.executors.executor_utils import ExecutorName
+from airflow.jobs.job import EXECUTOR_JOB_TYPE, HealthCheckResult, Job
+from airflow.jobs.scheduler_job_runner import SchedulerJobRunner, 
_executor_health_key
+
+from tests_common.test_utils.config import conf_vars
+from tests_common.test_utils.db import clear_db_jobs
+
+pytestmark = pytest.mark.db_test
+
+
+class _StubExecutor(BaseExecutor):
+    mock_module_path = "stub.executor.path"
+    mock_alias = "stub_executor"
+
+    def __init__(self, result=True, team_name=None, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.name = ExecutorName(module_path=self.mock_module_path, 
alias=self.mock_alias)
+        self.team_name = team_name
+        self.result = result
+        self.calls: list[float] = []
+
+    def get_health(self, timeout: float) -> bool | None:
+        self.calls.append(timeout)
+        if isinstance(self.result, Exception):
+            raise self.result
+        return self.result
+
+
[email protected](autouse=True)
+def clean_jobs():
+    clear_db_jobs()
+    yield
+    clear_db_jobs()
+
+
+def _make_runner(executor, session):
+    scheduler_job = Job()
+    session.add(scheduler_job)
+    session.commit()
+    runner = SchedulerJobRunner(job=scheduler_job, executors=[executor])
+    return runner
+
+
+def _get_executor_job(session, owner_job_id, executor_key):
+    return session.scalar(
+        select(Job).where(
+            Job.job_type == EXECUTOR_JOB_TYPE,
+            Job.owner_job_id == owner_job_id,
+            Job.executor_class == executor_key,
+        )
+    )
+
+
+def 
test_touch_executor_heartbeats_creates_a_row_for_a_reporting_executor(session):
+    executor = _StubExecutor()
+    runner = _make_runner(executor, session)
+    key = _executor_health_key(executor)
+
+    runner._touch_executor_heartbeats(session=session)
+
+    job = _get_executor_job(session, runner.job.id, key)
+    assert job is not None
+    assert job.health_check_result == HealthCheckResult.UNKNOWN
+    assert job.latest_heartbeat is not None
+    assert job.hostname != key
+
+
+def 
test_touch_executor_heartbeats_skips_executors_that_do_not_override_get_health(session):
+    executor = BaseExecutor()
+    executor.name = 
ExecutorName(module_path="airflow.executors.local_executor.LocalExecutor", 
alias="Local")
+    runner = _make_runner(executor, session)
+
+    runner._touch_executor_heartbeats(session=session)
+
+    assert session.scalar(select(Job).where(Job.job_type == 
EXECUTOR_JOB_TYPE)) is None
+
+
+def 
test_touch_executor_heartbeats_moves_the_heartbeat_without_changing_the_result(session):
+    executor = _StubExecutor()
+    runner = _make_runner(executor, session)
+    key = _executor_health_key(executor)
+
+    runner._touch_executor_heartbeats(session=session)
+    first = _get_executor_job(session, runner.job.id, key)
+    first.health_check_result = HealthCheckResult.HEALTHY
+    first_heartbeat = first.latest_heartbeat
+    session.commit()
+
+    runner._touch_executor_heartbeats(session=session)
+    second = _get_executor_job(session, runner.job.id, key)
+
+    assert second.id == first.id
+    assert second.health_check_result == HealthCheckResult.HEALTHY
+    assert second.latest_heartbeat >= first_heartbeat
+
+
+def test_run_executor_health_check_records_healthy(session):
+    executor = _StubExecutor(result=True)
+    runner = _make_runner(executor, session)
+    key = _executor_health_key(executor)
+
+    runner._run_executor_health_check(executor, key)
+
+    job = _get_executor_job(session, runner.job.id, key)
+    assert job.health_check_result == HealthCheckResult.HEALTHY
+
+
+def test_run_executor_health_check_records_unhealthy(session):
+    executor = _StubExecutor(result=False)
+    runner = _make_runner(executor, session)
+    key = _executor_health_key(executor)
+
+    runner._run_executor_health_check(executor, key)
+
+    job = _get_executor_job(session, runner.job.id, key)
+    assert job.health_check_result == HealthCheckResult.UNHEALTHY
+
+
+def test_run_executor_health_check_records_unknown_on_none(session):
+    executor = _StubExecutor(result=None)
+    runner = _make_runner(executor, session)
+    key = _executor_health_key(executor)
+
+    runner._run_executor_health_check(executor, key)
+
+    job = _get_executor_job(session, runner.job.id, key)
+    assert job.health_check_result == HealthCheckResult.UNKNOWN
+
+
+def 
test_run_executor_health_check_records_unknown_and_does_not_raise_when_get_health_raises(session):
+    executor = _StubExecutor(result=RuntimeError("boom"))
+    runner = _make_runner(executor, session)
+    key = _executor_health_key(executor)
+
+    runner._run_executor_health_check(executor, key)
+
+    job = _get_executor_job(session, runner.job.id, key)
+    assert job.health_check_result == HealthCheckResult.UNKNOWN
+
+
+def 
test_run_executor_health_check_passes_half_the_configured_interval_as_timeout(session):
+    with conf_vars({("scheduler", "executor_health_check_interval"): "10.0"}):

Review Comment:
   Nit:  Can you move this to a decorator, please.  Functionally the same, but 
a style consistency thing:
   
   ```python
   @conf_vars({("scheduler", "executor_health_check_interval"): "10.0"})
   def 
test_run_executor_health_check_passes_half_the_configured_interval_as_timeout(session):
       ...
   ```



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -340,6 +341,19 @@ def _get_current_dr_task_concurrency(states: 
Iterable[TaskInstanceState]) -> Sub
     )
 
 
+def _executor_health_key(executor: BaseExecutor) -> str:
+    """
+    Return the key an executor's health is tracked and recorded under.
+
+    ``ExecutorName.__repr__()`` is the identity ``ExecutorName`` itself 
already uses for equality and
+    hashing, so using it here too keeps this side and 
``_configured_executor_keys()`` in
+    ``airflow_health.py`` from drifting apart: an executor configured by a 
bare module path has no
+    alias, and comparing ``alias or module_path`` against ``alias or class 
name`` doesn't agree on
+    one for that case.
+    """
+    return repr(executor.name) if executor.name is not None else 
type(executor).__name__

Review Comment:
   `load_executor()` always sets `executor.name`, so this fallback only runs 
for an executor passed in without a name.  When it does, the key it produces 
(CeleryExecutor) can never match `_configured_executor_keys()` 
(:CeleryExecutor:), so that executor would read DOWN forever with nothing to 
say why.  I'd drop the fallback and skip unnamed executors where
   `_executor_health_keys` is built, so they just don't get a row:
   
   
   ```python
     return repr(executor.name) if executor.name is not None and 
type(executor).get_health is not _BaseExecutor.get_health
   ```



##########
airflow-core/src/airflow/api/common/airflow_health.py:
##########
@@ -106,6 +107,42 @@ def _dag_processor_instance_health(job: Job) -> dict[str, 
Any]:
     }
 
 
+def _executor_instance_health(job: Job) -> dict[str, Any]:
+    return {
+        **_job_instance_health(job, "latest_executor_heartbeat"),
+        "executor": job.executor_class,

Review Comment:
   `job.executor_class` is going t resolve to something like `:CeleryExecutor:` 
or `team_a:my_exec:my_company.executors.MyExecutor`.  Since the team_name is 
right there, and that `executor_class` is kind of leaking an internal 
implementation detail, I'd suggest that `name.alias` or `name.module_path` 
might be a better fit here since that should be the same name the user provided 
in the `[core] executor:` config entry.  If we do that, then we're reporting it 
as the name they know it as instead of the name we know it as, and that feels 
like a better user experience.  maybe:
   
   ```
   "executor": (name.alias or name.module_path) if name else job.executor_class
   ```



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1912,6 +2093,10 @@ def _execute(self) -> int | None:
                 except Exception:
                     self.log.exception("Exception when executing Executor.end 
on %s", executor)
 
+            # Don't block scheduler shutdown on a health check that's still in 
flight; it has no
+            # effect once nothing is left to record its result.
+            self._executor_health_check_pool.shutdown(wait=False, 
cancel_futures=True)

Review Comment:
   I might be missing something on this one so feel free to push back, but I 
don't think anything ever closes an `ExecutorJob` row.  The scheduler's row 
gets an `end_date` and a final state when it stops (handled by `run_job()` and 
`execute_job()` in `job.py`), but these executor rows stay running with no 
`end_date` after their scheduler exits.  That means `/api/v2/jobs` and the 
`Jobs` page will keep showing them as running, and every scheduler restart adds 
one more row per reporting executor.
   
   I can think of two ways that a scheduler can stop, and each would need its 
own fix:
   
   1. Clean exit (including SIGTERM).  This `finally` runs no matter how the 
loop exits (`_exit_gracefully` calls `sys.exit()`), so it's the right place to 
close this scheduler's rows:
   
   ```python
       @provide_session
       def _end_executor_jobs(self, state: JobState, *, session: Session = 
NEW_SESSION) -> None:
           session.execute(
               update(Job)
               .where(
                   Job.job_type == EXECUTOR_JOB_TYPE,
                   Job.owner_job_id == self.job.id,
                   Job.end_date.is_(None),
               )
               .values(state=state, end_date=timezone.utcnow())
           )
   ```
   
   Use the same state `execute_job()` gives the scheduler's row (SUCCESS for a 
normal exit or SIGTERM, FAILED if the loop raised) and the `except Exception` 
block just above is where you find out which.  Wrap the call in a `try/except` 
that logs, like the `executor.end()` calls above it, tha way a database error 
during shutdown can't hide the original one.
   
   2. Crash (SIGKILL, OOM, a lost node).  `finally` never runs, so nothing here 
helps.  `adopt_or_reset_orphaned_tasks` already marks stale `SchedulerJob` rows 
FAILED.  A second `update(Job)` right after it for `EXECUTOR_JOB_TYPE`, with 
the same heartbeat cutoff would treat these rows the same way.  It also catches 
a row that a health check still running at shutdown recreates after this 
`finally` has closed the others.  Keep it as a separate statement so the 
"Marked %d SchedulerJob instances as failed" log line and the `_end` metric 
still only count schedulers.
   
   



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -404,6 +418,38 @@ def __init__(
         self.executors: list[BaseExecutor] = executors if executors else 
ExecutorLoader.init_executors()
         self.executor: BaseExecutor = self.executors[0]
 
+        # Each executor's get_health() runs here, never on the scheduler 
loop's own thread: a check
+        # that talks to a broker or a cluster control plane can be slow or 
hang, and must not be able
+        # to delay task scheduling. One worker per configured executor is 
enough since each executor
+        # is only ever checked by one in-flight call at a time (see 
_check_executor_health).
+        self._executor_health_check_pool = ThreadPoolExecutor(

Review Comment:
   Thread Pools aren't something I'm great with, but I think it's the wrong 
tool here.  My understanding is that Python waits for a ThreadPoolExecutor 
worker thread when the interpreter exits, so even though `shutdown()` returns 
right away, the process itself won't exit until  `get_health()` returns.
   
   But since we know that each executor can only ever have one check in flight, 
you don't actually need a pool.  A daemon thread per check can still hand back 
the Future you already track, so everything you built around Future() would 
stay the same:
   
   ```python
             future: Future = Future()
             threading.Thread(
                 target=self._run_executor_health_check_and_resolve,
                 args=(executor, key, future),
                 name=f"executor-health-check-{key}",
                 daemon=True,
             ).start()
             self._executor_health_check_futures[key] = (future, 
time.monotonic())
   
     def _run_executor_health_check_and_resolve(self, executor: BaseExecutor, 
key: str, future: Future) -> None:
         try:
             self._run_executor_health_check(executor, key)
         finally:
             future.set_result(None)
   ```
   
   Then the pool, its `shutdown()` call, and the `ThreadPoolExecutor` import 
can all go.
   
   `LocalExecutor` does the same thing for a call that might never return 
(`local_executor.py:387`, for example).  The trade-off is that Python stops a 
daemon thread abruptly at exit, so a check that's partway through just gets cut 
off.  I'd say that's fine in this case since the scheduler is shutting down 
anyway.



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -425,6 +471,141 @@ def __init__(
     def heartbeat_callback(self, *, session: Session = NEW_SESSION) -> None:
         stats.incr("scheduler_heartbeat", 1, 1)
 
+    def _check_executor_health(self) -> None:
+        """
+        Kick off a health check for each reporting executor, in the background.
+
+        Called on its own timer (``[scheduler] 
executor_health_check_interval``), independent of the
+        scheduling loop and of each executor's own 
:meth:`~airflow.executors.base_executor.BaseExecutor.heartbeat`.
+        A new check for a given executor is skipped while its previous check 
is still running, so a
+        slow or hung dependency can only ever delay that executor's own health 
being updated, never
+        pile up work or block this (or any other) scheduler tick.
+
+        ``get_health()`` is passed a timeout, but an implementation that 
ignores it can still leave its
+        future pending forever. That future's own worker slot is never 
reclaimed -- the pool is sized
+        one worker per executor, so a stuck probe can only ever cost that 
executor's own slot -- but
+        without this check it would silently stop being checked again, with no 
sign anything was
+        wrong. Once a pending future is a few intervals old, it's reported as 
stalled instead.
+        """
+        for executor in self.executors:
+            key = _executor_health_key(executor)
+            if key not in self._executor_health_keys:
+                # Doesn't override get_health(); there's nothing to check or 
record.
+                continue
+            future, started = self._executor_health_check_futures.get(key, 
(None, 0.0))
+            if future is not None and not future.done():
+                stall_after = (
+                    self._executor_health_check_interval * 
self._executor_health_check_stall_after_intervals
+                )
+                if time.monotonic() - started > stall_after and key not in 
self._stalled_health_checks:
+                    self._stalled_health_checks.add(key)
+                    self.log.warning("Health check for executor %s has stopped 
responding", key)
+                    stats.incr("scheduler.executor_health_check.stalled", 
tags={"executor": key})
+                else:
+                    self.log.debug(
+                        "Skipping health check for executor %s, the previous 
check hasn't finished", key
+                    )
+                continue
+            self._stalled_health_checks.discard(key)
+            future = 
self._executor_health_check_pool.submit(self._run_executor_health_check, 
executor, key)
+            self._executor_health_check_futures[key] = (future, 
time.monotonic())
+
+    def _run_executor_health_check(self, executor: BaseExecutor, key: str) -> 
None:
+        """Run one executor's get_health() off the scheduler thread and record 
an explicit result."""
+        try:
+            healthy = 
executor.get_health(timeout=self._executor_health_check_timeout)
+        except Exception:
+            self.log.exception("Health check for executor %s raised an 
exception", key)
+            healthy = None
+        if healthy is None:
+            result = HealthCheckResult.UNKNOWN
+        elif healthy:
+            result = HealthCheckResult.HEALTHY
+        else:
+            self.log.warning("Executor %s reported itself unhealthy", key)
+            result = HealthCheckResult.UNHEALTHY
+        try:
+            self._record_executor_health_result(key, executor.team_name, 
result)
+        except Exception:
+            self.log.exception("Failed to record health check result for 
executor %s", key)
+
+    @provide_session
+    def _touch_executor_heartbeats(self, *, session: Session = NEW_SESSION) -> 
None:
+        """
+        Keep every reporting executor's Job row heartbeat moving on the 
scheduler's own cadence.
+
+        ``get_health()`` itself only runs every 
``executor_health_check_interval`` (default 30s), far
+        slower than a Job row can go quiet for before 
``health_check_threshold`` calls it dead.
+        Updating the heartbeat only when a check finishes would make a 
perfectly healthy executor
+        flicker to UNHEALTHY between checks, every cycle. Keeping the 
heartbeat on the scheduler's own
+        short cadence instead, the same way it already does for its own Job 
row, means the row's
+        liveness reflects "the scheduler is still running and still intends to 
check this executor",
+        while ``health_check_result`` -- not this heartbeat -- carries 
whatever the last actual check
+        found.
+        """
+        if not self._executor_health_keys:
+            return
+        now = timezone.utcnow()
+        for key, team_name in self._executor_health_keys.items():
+            job = self._get_or_create_executor_job(key, team_name, 
session=session)
+            job.latest_heartbeat = now
+        session.commit()
+
+    def _get_or_create_executor_job(
+        self, executor_key: str, team_name: str | None, *, session: Session
+    ) -> Job:
+        """
+        Return this scheduler's own Job row for ``executor_key``, creating it 
if needed.
+
+        Keyed by ``owner_job_id`` (this scheduler's own Job id), not just 
``hostname``: two schedulers
+        running the same executor against shared infrastructure (e.g. the same 
Celery broker) each get
+        their own row, so one scheduler's healthy report can never paper over 
another's failing one.
+        A restarted scheduler gets a fresh row for the same reason its own Job 
row is fresh -- a new
+        id -- instead of quietly taking over a row left behind by its previous 
instance.
+        """
+        job = session.scalar(
+            select(Job).where(
+                Job.job_type == EXECUTOR_JOB_TYPE,
+                Job.executor_class == executor_key,
+                Job.owner_job_id == self.job.id,
+                Job.end_date.is_(None),
+            )
+        )
+        if job is None:
+            job = Job(

Review Comment:
   Creating the row with `Job(...)` runs `Job.__init__`, which fires the 
`on_starting` listener hook (`job.py:211`).  That hook means "a component is 
starting", but we are using it here every time an executor row is created; the 
scheduler is already running.  `before_stopping` never fires for these rows, so 
a listener is going to see starts with no matching stops.
   
   Core already has a pattern for creating a `Job` row on another component's 
behalf.  The Execution API's registration endpoint does a plain insert, with a 
comment saying why (`api_fastapi/execution_api/routes/jobs.py:206`).  The same 
should work here.  You do want the scheduler's host and user on these rows, so 
set them explicitly:
   
   ```python
         if job is None:
             now = timezone.utcnow()
             inserted = session.execute(
                 insert(Job).values(
                     job_type=EXECUTOR_JOB_TYPE,
                     executor_class=executor_key,
                     owner_job_id=self.job.id,
                     health_check_result=HealthCheckResult.UNKNOWN,
                     state=JobState.RUNNING,
                     start_date=now,
                     latest_heartbeat=now,
                     hostname=get_hostname(),
                     unixname=getuser(),
                 )
             )
             job_id = cast("CursorResult", inserted).inserted_primary_key[0]  # 
type: ignore[index]
             if team_name:
                 session.add(JobTeam(job_id=job_id, team_name=team_name))
             session.flush()
             job = session.get(Job, job_id)
   ```
   
   Since the team is set when the row is created, the `elif team_name:` branch 
below can go, too since it currently rebuilds `team_names` on every heartbeat.



##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -425,6 +471,136 @@ def __init__(
     def heartbeat_callback(self, *, session: Session = NEW_SESSION) -> None:
         stats.incr("scheduler_heartbeat", 1, 1)
 
+    def _check_executor_health(self) -> None:

Review Comment:
   Thanks for adding the new test file.  `_run_executor_health_check`, 
`_touch_executor_heartbeats` and `_get_or_create_executor_job` are covered now. 
 The one still missing is `_check_executor_health`,   the method this thread is 
on.
   
   Two smaller ones while you're in here:
   -` _run_executor_health_check` when `_record_executor_health_result` raises: 
it should log the error and not raise.
   - On the API side, the legacy status mapping: DEGRADED and all-UNKNOWN 
should both give HEALTHY.  There are tests for healthy, unhealthy and 
never-reported, but not those two.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to