ferruzzi commented on code in PR #76410:
URL: https://github.com/apache/airflow/pull/76410#discussion_r4606228936
##########
airflow-core/src/airflow/api/common/airflow_health.py:
##########
@@ -236,6 +339,24 @@ def get_airflow_health() -> dict[str, Any]:
dag_processor_status = HealthStatus.UNHEALTHY
dag_processor_detailed_status = DetailedHealthStatus.DOWN
+ # Omitted entirely, not just reported unhealthy, when no configured
executor implements a health
+ # check: a reporting mechanism nothing has opted into isn't a gap in
coverage, and showing it as
+ # "unhealthy" on every deployment that hasn't adopted it yet would be a
false alarm, not a signal.
+ report_executor_health = bool(_configured_executor_keys())
Review Comment:
`_configured_executor_keys()` imports every configured executor class, and
this runs every time `/monitor/health` is hit, even when nothing overrides
`get_health()`. The PR description says the change is inert until something
overrides `get_health()`, but this part isn't: nothing else in the API server
imports executor classes (`connections.py` only reads their names), and this
does it on every request whether or not anything overrides the method.
This is going to cause two problems:
- If an import fails on the API server, the warning and traceback at line
220 get logged on every request since Python retries a failed import each time.
Anything polling this endpoint turns that into a traceback every few seconds.
- Each request computes the keys 3 times: here, `_executor_detailed_status`,
and `_executor_legacy_status`. The live-scheduler query in
`_live_executor_jobs` also runs three times.
The configured executors can't change while the process runs, so I'd cache
the keys with `@lru_cache` (see `health_check_threshold()` in `job.py` for an
example, and the `cleanup_jobs fixture` in `TestAirflowHealthFromDb` shows how
the tests reset it) and call `get_executor_names(validate_teams=False)` the way
`connections.py` does. If we skip team validation, then a database hiccup on
the first call can't get cached as "no executors". Tests that patch
`get_executor_names` will need `_configured_executor_keys.cache_clear()`.
For the repeated status work, compute `detailed_status` once and derive the
legacy status from it, which also lets `_executor_legacy_status` go away:
```python
executor_detailed_status = _executor_detailed_status(executor_jobs)
executor_status = (
HealthStatus.UNHEALTHY
if executor_detailed_status == DetailedHealthStatus.DOWN
else HealthStatus.HEALTHY
)
```
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -425,6 +471,141 @@ def __init__(
def heartbeat_callback(self, *, session: Session = NEW_SESSION) -> None:
stats.incr("scheduler_heartbeat", 1, 1)
+ def _check_executor_health(self) -> None:
+ """
+ Kick off a health check for each reporting executor, in the background.
+
+ Called on its own timer (``[scheduler]
executor_health_check_interval``), independent of the
+ scheduling loop and of each executor's own
:meth:`~airflow.executors.base_executor.BaseExecutor.heartbeat`.
+ A new check for a given executor is skipped while its previous check
is still running, so a
+ slow or hung dependency can only ever delay that executor's own health
being updated, never
+ pile up work or block this (or any other) scheduler tick.
+
+ ``get_health()`` is passed a timeout, but an implementation that
ignores it can still leave its
+ future pending forever. That future's own worker slot is never
reclaimed -- the pool is sized
+ one worker per executor, so a stuck probe can only ever cost that
executor's own slot -- but
+ without this check it would silently stop being checked again, with no
sign anything was
+ wrong. Once a pending future is a few intervals old, it's reported as
stalled instead.
+ """
+ for executor in self.executors:
+ key = _executor_health_key(executor)
+ if key not in self._executor_health_keys:
+ # Doesn't override get_health(); there's nothing to check or
record.
+ continue
+ future, started = self._executor_health_check_futures.get(key,
(None, 0.0))
+ if future is not None and not future.done():
+ stall_after = (
+ self._executor_health_check_interval *
self._executor_health_check_stall_after_intervals
+ )
+ if time.monotonic() - started > stall_after and key not in
self._stalled_health_checks:
+ self._stalled_health_checks.add(key)
Review Comment:
When a check gets stuck, this warns once and emits the metric, which is
good, but the row itself doesn't change. `_touch_executor_heartbeats` keeps
the heartbeat fresh, and `health_check_result` still holds whatever the last
finished check returned. If that was HEALTHY, then `/monitor/health` keeps
reporting the executor as healthy for as long as the check stays stuck.
For example, say a check ignores its timeout and hangs talking to a dead
broker. After three tries, the scheduler logs "has stopped responding", but
anyone looking at `/monitor/health` or the dashboard still sees green.
A stuck check hasn't answered, so I'd record it the same way as one that
raised. The scheduler already knows at this point, so it can write the result
itself:
```
if time.monotonic() - started > stall_after and key not in
self._stalled_health_checks:
self._stalled_health_checks.add(key)
self.log.warning("Health check for executor %s has stopped
responding", key)
stats.incr("scheduler.executor_health_check.stalled",
tags={"executor": key})
try:
self._record_executor_health_result(key,
executor.team_name, HealthCheckResult.UNHEALTHY)
except Exception:
self.log.exception("Failed to record health check
result for executor %s", key)
```
If the stuck call does eventually return, its result overwrites this one,
which is what we'd want. A test that leaves a future pending past
`stall_after` and checks that the row reads UNHEALTHY would cover it, and it
would sit naturally with the `_check_executor_health` tests from the earlier
thread.
##########
airflow-core/src/airflow/config_templates/config.yml:
##########
@@ -3018,6 +3018,18 @@ scheduler:
type: float
example: ~
default: "120.0"
+ executor_health_check_interval:
+ description: |
+ How often, in seconds, the scheduler checks each configured executor's
health via
+ ``get_health()``, reported on ``/monitor/health``. Each check runs in
its own background
+ thread so a slow or unresponsive executor backend cannot delay task
scheduling; this
+ setting controls how often a check is attempted, not how long a check
is allowed to take.
Review Comment:
Since the timeout became half the interval, the end of this sentence isn't
true anymore; this setting now **does** control how long a check is allowed to
take
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -425,6 +471,141 @@ def __init__(
def heartbeat_callback(self, *, session: Session = NEW_SESSION) -> None:
stats.incr("scheduler_heartbeat", 1, 1)
+ def _check_executor_health(self) -> None:
+ """
+ Kick off a health check for each reporting executor, in the background.
+
+ Called on its own timer (``[scheduler]
executor_health_check_interval``), independent of the
+ scheduling loop and of each executor's own
:meth:`~airflow.executors.base_executor.BaseExecutor.heartbeat`.
+ A new check for a given executor is skipped while its previous check
is still running, so a
+ slow or hung dependency can only ever delay that executor's own health
being updated, never
+ pile up work or block this (or any other) scheduler tick.
+
+ ``get_health()`` is passed a timeout, but an implementation that
ignores it can still leave its
+ future pending forever. That future's own worker slot is never
reclaimed -- the pool is sized
+ one worker per executor, so a stuck probe can only ever cost that
executor's own slot -- but
+ without this check it would silently stop being checked again, with no
sign anything was
+ wrong. Once a pending future is a few intervals old, it's reported as
stalled instead.
+ """
+ for executor in self.executors:
+ key = _executor_health_key(executor)
+ if key not in self._executor_health_keys:
+ # Doesn't override get_health(); there's nothing to check or
record.
+ continue
+ future, started = self._executor_health_check_futures.get(key,
(None, 0.0))
+ if future is not None and not future.done():
+ stall_after = (
+ self._executor_health_check_interval *
self._executor_health_check_stall_after_intervals
+ )
+ if time.monotonic() - started > stall_after and key not in
self._stalled_health_checks:
+ self._stalled_health_checks.add(key)
+ self.log.warning("Health check for executor %s has stopped
responding", key)
+ stats.incr("scheduler.executor_health_check.stalled",
tags={"executor": key})
Review Comment:
Nit: you are tagging with the repr key here, but
`scheduler.executor_heartbeat_duration` below tags with the class name and
team_name, and uses `prune_dict`. They should be the same; I'd recommend they
both use the other version.
##########
airflow-core/tests/unit/jobs/test_scheduler_job_executor_health.py:
##########
@@ -0,0 +1,196 @@
+from __future__ import annotations
+
+import pytest
+from sqlalchemy import select
+
+from airflow.executors.base_executor import BaseExecutor
+from airflow.executors.executor_utils import ExecutorName
+from airflow.jobs.job import EXECUTOR_JOB_TYPE, HealthCheckResult, Job
+from airflow.jobs.scheduler_job_runner import SchedulerJobRunner,
_executor_health_key
+
+from tests_common.test_utils.config import conf_vars
+from tests_common.test_utils.db import clear_db_jobs
+
+pytestmark = pytest.mark.db_test
+
+
+class _StubExecutor(BaseExecutor):
+ mock_module_path = "stub.executor.path"
+ mock_alias = "stub_executor"
+
+ def __init__(self, result=True, team_name=None, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.name = ExecutorName(module_path=self.mock_module_path,
alias=self.mock_alias)
+ self.team_name = team_name
+ self.result = result
+ self.calls: list[float] = []
+
+ def get_health(self, timeout: float) -> bool | None:
+ self.calls.append(timeout)
+ if isinstance(self.result, Exception):
+ raise self.result
+ return self.result
+
+
[email protected](autouse=True)
+def clean_jobs():
+ clear_db_jobs()
+ yield
+ clear_db_jobs()
+
+
+def _make_runner(executor, session):
+ scheduler_job = Job()
+ session.add(scheduler_job)
+ session.commit()
+ runner = SchedulerJobRunner(job=scheduler_job, executors=[executor])
+ return runner
+
+
+def _get_executor_job(session, owner_job_id, executor_key):
+ return session.scalar(
+ select(Job).where(
+ Job.job_type == EXECUTOR_JOB_TYPE,
+ Job.owner_job_id == owner_job_id,
+ Job.executor_class == executor_key,
+ )
+ )
+
+
+def
test_touch_executor_heartbeats_creates_a_row_for_a_reporting_executor(session):
+ executor = _StubExecutor()
+ runner = _make_runner(executor, session)
+ key = _executor_health_key(executor)
+
+ runner._touch_executor_heartbeats(session=session)
+
+ job = _get_executor_job(session, runner.job.id, key)
+ assert job is not None
+ assert job.health_check_result == HealthCheckResult.UNKNOWN
+ assert job.latest_heartbeat is not None
+ assert job.hostname != key
+
+
+def
test_touch_executor_heartbeats_skips_executors_that_do_not_override_get_health(session):
+ executor = BaseExecutor()
+ executor.name =
ExecutorName(module_path="airflow.executors.local_executor.LocalExecutor",
alias="Local")
+ runner = _make_runner(executor, session)
+
+ runner._touch_executor_heartbeats(session=session)
+
+ assert session.scalar(select(Job).where(Job.job_type ==
EXECUTOR_JOB_TYPE)) is None
+
+
+def
test_touch_executor_heartbeats_moves_the_heartbeat_without_changing_the_result(session):
+ executor = _StubExecutor()
+ runner = _make_runner(executor, session)
+ key = _executor_health_key(executor)
+
+ runner._touch_executor_heartbeats(session=session)
+ first = _get_executor_job(session, runner.job.id, key)
+ first.health_check_result = HealthCheckResult.HEALTHY
+ first_heartbeat = first.latest_heartbeat
+ session.commit()
+
+ runner._touch_executor_heartbeats(session=session)
+ second = _get_executor_job(session, runner.job.id, key)
+
+ assert second.id == first.id
+ assert second.health_check_result == HealthCheckResult.HEALTHY
+ assert second.latest_heartbeat >= first_heartbeat
+
+
+def test_run_executor_health_check_records_healthy(session):
+ executor = _StubExecutor(result=True)
+ runner = _make_runner(executor, session)
+ key = _executor_health_key(executor)
+
+ runner._run_executor_health_check(executor, key)
+
+ job = _get_executor_job(session, runner.job.id, key)
+ assert job.health_check_result == HealthCheckResult.HEALTHY
+
+
+def test_run_executor_health_check_records_unhealthy(session):
+ executor = _StubExecutor(result=False)
+ runner = _make_runner(executor, session)
+ key = _executor_health_key(executor)
+
+ runner._run_executor_health_check(executor, key)
+
+ job = _get_executor_job(session, runner.job.id, key)
+ assert job.health_check_result == HealthCheckResult.UNHEALTHY
+
+
+def test_run_executor_health_check_records_unknown_on_none(session):
+ executor = _StubExecutor(result=None)
+ runner = _make_runner(executor, session)
+ key = _executor_health_key(executor)
+
+ runner._run_executor_health_check(executor, key)
+
+ job = _get_executor_job(session, runner.job.id, key)
+ assert job.health_check_result == HealthCheckResult.UNKNOWN
+
+
+def
test_run_executor_health_check_records_unknown_and_does_not_raise_when_get_health_raises(session):
+ executor = _StubExecutor(result=RuntimeError("boom"))
+ runner = _make_runner(executor, session)
+ key = _executor_health_key(executor)
+
+ runner._run_executor_health_check(executor, key)
+
+ job = _get_executor_job(session, runner.job.id, key)
+ assert job.health_check_result == HealthCheckResult.UNKNOWN
+
+
+def
test_run_executor_health_check_passes_half_the_configured_interval_as_timeout(session):
+ with conf_vars({("scheduler", "executor_health_check_interval"): "10.0"}):
Review Comment:
Nit: Can you move this to a decorator, please. Functionally the same, but
a style consistency thing:
```python
@conf_vars({("scheduler", "executor_health_check_interval"): "10.0"})
def
test_run_executor_health_check_passes_half_the_configured_interval_as_timeout(session):
...
```
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -340,6 +341,19 @@ def _get_current_dr_task_concurrency(states:
Iterable[TaskInstanceState]) -> Sub
)
+def _executor_health_key(executor: BaseExecutor) -> str:
+ """
+ Return the key an executor's health is tracked and recorded under.
+
+ ``ExecutorName.__repr__()`` is the identity ``ExecutorName`` itself
already uses for equality and
+ hashing, so using it here too keeps this side and
``_configured_executor_keys()`` in
+ ``airflow_health.py`` from drifting apart: an executor configured by a
bare module path has no
+ alias, and comparing ``alias or module_path`` against ``alias or class
name`` doesn't agree on
+ one for that case.
+ """
+ return repr(executor.name) if executor.name is not None else
type(executor).__name__
Review Comment:
`load_executor()` always sets `executor.name`, so this fallback only runs
for an executor passed in without a name. When it does, the key it produces
(CeleryExecutor) can never match `_configured_executor_keys()`
(:CeleryExecutor:), so that executor would read DOWN forever with nothing to
say why. I'd drop the fallback and skip unnamed executors where
`_executor_health_keys` is built, so they just don't get a row:
```python
return repr(executor.name) if executor.name is not None and
type(executor).get_health is not _BaseExecutor.get_health
```
##########
airflow-core/src/airflow/api/common/airflow_health.py:
##########
@@ -106,6 +107,42 @@ def _dag_processor_instance_health(job: Job) -> dict[str,
Any]:
}
+def _executor_instance_health(job: Job) -> dict[str, Any]:
+ return {
+ **_job_instance_health(job, "latest_executor_heartbeat"),
+ "executor": job.executor_class,
Review Comment:
`job.executor_class` is going t resolve to something like `:CeleryExecutor:`
or `team_a:my_exec:my_company.executors.MyExecutor`. Since the team_name is
right there, and that `executor_class` is kind of leaking an internal
implementation detail, I'd suggest that `name.alias` or `name.module_path`
might be a better fit here since that should be the same name the user provided
in the `[core] executor:` config entry. If we do that, then we're reporting it
as the name they know it as instead of the name we know it as, and that feels
like a better user experience. maybe:
```
"executor": (name.alias or name.module_path) if name else job.executor_class
```
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -1912,6 +2093,10 @@ def _execute(self) -> int | None:
except Exception:
self.log.exception("Exception when executing Executor.end
on %s", executor)
+ # Don't block scheduler shutdown on a health check that's still in
flight; it has no
+ # effect once nothing is left to record its result.
+ self._executor_health_check_pool.shutdown(wait=False,
cancel_futures=True)
Review Comment:
I might be missing something on this one so feel free to push back, but I
don't think anything ever closes an `ExecutorJob` row. The scheduler's row
gets an `end_date` and a final state when it stops (handled by `run_job()` and
`execute_job()` in `job.py`), but these executor rows stay running with no
`end_date` after their scheduler exits. That means `/api/v2/jobs` and the
`Jobs` page will keep showing them as running, and every scheduler restart adds
one more row per reporting executor.
I can think of two ways that a scheduler can stop, and each would need its
own fix:
1. Clean exit (including SIGTERM). This `finally` runs no matter how the
loop exits (`_exit_gracefully` calls `sys.exit()`), so it's the right place to
close this scheduler's rows:
```python
@provide_session
def _end_executor_jobs(self, state: JobState, *, session: Session =
NEW_SESSION) -> None:
session.execute(
update(Job)
.where(
Job.job_type == EXECUTOR_JOB_TYPE,
Job.owner_job_id == self.job.id,
Job.end_date.is_(None),
)
.values(state=state, end_date=timezone.utcnow())
)
```
Use the same state `execute_job()` gives the scheduler's row (SUCCESS for a
normal exit or SIGTERM, FAILED if the loop raised) and the `except Exception`
block just above is where you find out which. Wrap the call in a `try/except`
that logs, like the `executor.end()` calls above it, tha way a database error
during shutdown can't hide the original one.
2. Crash (SIGKILL, OOM, a lost node). `finally` never runs, so nothing here
helps. `adopt_or_reset_orphaned_tasks` already marks stale `SchedulerJob` rows
FAILED. A second `update(Job)` right after it for `EXECUTOR_JOB_TYPE`, with
the same heartbeat cutoff would treat these rows the same way. It also catches
a row that a health check still running at shutdown recreates after this
`finally` has closed the others. Keep it as a separate statement so the
"Marked %d SchedulerJob instances as failed" log line and the `_end` metric
still only count schedulers.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -404,6 +418,38 @@ def __init__(
self.executors: list[BaseExecutor] = executors if executors else
ExecutorLoader.init_executors()
self.executor: BaseExecutor = self.executors[0]
+ # Each executor's get_health() runs here, never on the scheduler
loop's own thread: a check
+ # that talks to a broker or a cluster control plane can be slow or
hang, and must not be able
+ # to delay task scheduling. One worker per configured executor is
enough since each executor
+ # is only ever checked by one in-flight call at a time (see
_check_executor_health).
+ self._executor_health_check_pool = ThreadPoolExecutor(
Review Comment:
Thread Pools aren't something I'm great with, but I think it's the wrong
tool here. My understanding is that Python waits for a ThreadPoolExecutor
worker thread when the interpreter exits, so even though `shutdown()` returns
right away, the process itself won't exit until `get_health()` returns.
But since we know that each executor can only ever have one check in flight,
you don't actually need a pool. A daemon thread per check can still hand back
the Future you already track, so everything you built around Future() would
stay the same:
```python
future: Future = Future()
threading.Thread(
target=self._run_executor_health_check_and_resolve,
args=(executor, key, future),
name=f"executor-health-check-{key}",
daemon=True,
).start()
self._executor_health_check_futures[key] = (future,
time.monotonic())
def _run_executor_health_check_and_resolve(self, executor: BaseExecutor,
key: str, future: Future) -> None:
try:
self._run_executor_health_check(executor, key)
finally:
future.set_result(None)
```
Then the pool, its `shutdown()` call, and the `ThreadPoolExecutor` import
can all go.
`LocalExecutor` does the same thing for a call that might never return
(`local_executor.py:387`, for example). The trade-off is that Python stops a
daemon thread abruptly at exit, so a check that's partway through just gets cut
off. I'd say that's fine in this case since the scheduler is shutting down
anyway.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -425,6 +471,141 @@ def __init__(
def heartbeat_callback(self, *, session: Session = NEW_SESSION) -> None:
stats.incr("scheduler_heartbeat", 1, 1)
+ def _check_executor_health(self) -> None:
+ """
+ Kick off a health check for each reporting executor, in the background.
+
+ Called on its own timer (``[scheduler]
executor_health_check_interval``), independent of the
+ scheduling loop and of each executor's own
:meth:`~airflow.executors.base_executor.BaseExecutor.heartbeat`.
+ A new check for a given executor is skipped while its previous check
is still running, so a
+ slow or hung dependency can only ever delay that executor's own health
being updated, never
+ pile up work or block this (or any other) scheduler tick.
+
+ ``get_health()`` is passed a timeout, but an implementation that
ignores it can still leave its
+ future pending forever. That future's own worker slot is never
reclaimed -- the pool is sized
+ one worker per executor, so a stuck probe can only ever cost that
executor's own slot -- but
+ without this check it would silently stop being checked again, with no
sign anything was
+ wrong. Once a pending future is a few intervals old, it's reported as
stalled instead.
+ """
+ for executor in self.executors:
+ key = _executor_health_key(executor)
+ if key not in self._executor_health_keys:
+ # Doesn't override get_health(); there's nothing to check or
record.
+ continue
+ future, started = self._executor_health_check_futures.get(key,
(None, 0.0))
+ if future is not None and not future.done():
+ stall_after = (
+ self._executor_health_check_interval *
self._executor_health_check_stall_after_intervals
+ )
+ if time.monotonic() - started > stall_after and key not in
self._stalled_health_checks:
+ self._stalled_health_checks.add(key)
+ self.log.warning("Health check for executor %s has stopped
responding", key)
+ stats.incr("scheduler.executor_health_check.stalled",
tags={"executor": key})
+ else:
+ self.log.debug(
+ "Skipping health check for executor %s, the previous
check hasn't finished", key
+ )
+ continue
+ self._stalled_health_checks.discard(key)
+ future =
self._executor_health_check_pool.submit(self._run_executor_health_check,
executor, key)
+ self._executor_health_check_futures[key] = (future,
time.monotonic())
+
+ def _run_executor_health_check(self, executor: BaseExecutor, key: str) ->
None:
+ """Run one executor's get_health() off the scheduler thread and record
an explicit result."""
+ try:
+ healthy =
executor.get_health(timeout=self._executor_health_check_timeout)
+ except Exception:
+ self.log.exception("Health check for executor %s raised an
exception", key)
+ healthy = None
+ if healthy is None:
+ result = HealthCheckResult.UNKNOWN
+ elif healthy:
+ result = HealthCheckResult.HEALTHY
+ else:
+ self.log.warning("Executor %s reported itself unhealthy", key)
+ result = HealthCheckResult.UNHEALTHY
+ try:
+ self._record_executor_health_result(key, executor.team_name,
result)
+ except Exception:
+ self.log.exception("Failed to record health check result for
executor %s", key)
+
+ @provide_session
+ def _touch_executor_heartbeats(self, *, session: Session = NEW_SESSION) ->
None:
+ """
+ Keep every reporting executor's Job row heartbeat moving on the
scheduler's own cadence.
+
+ ``get_health()`` itself only runs every
``executor_health_check_interval`` (default 30s), far
+ slower than a Job row can go quiet for before
``health_check_threshold`` calls it dead.
+ Updating the heartbeat only when a check finishes would make a
perfectly healthy executor
+ flicker to UNHEALTHY between checks, every cycle. Keeping the
heartbeat on the scheduler's own
+ short cadence instead, the same way it already does for its own Job
row, means the row's
+ liveness reflects "the scheduler is still running and still intends to
check this executor",
+ while ``health_check_result`` -- not this heartbeat -- carries
whatever the last actual check
+ found.
+ """
+ if not self._executor_health_keys:
+ return
+ now = timezone.utcnow()
+ for key, team_name in self._executor_health_keys.items():
+ job = self._get_or_create_executor_job(key, team_name,
session=session)
+ job.latest_heartbeat = now
+ session.commit()
+
+ def _get_or_create_executor_job(
+ self, executor_key: str, team_name: str | None, *, session: Session
+ ) -> Job:
+ """
+ Return this scheduler's own Job row for ``executor_key``, creating it
if needed.
+
+ Keyed by ``owner_job_id`` (this scheduler's own Job id), not just
``hostname``: two schedulers
+ running the same executor against shared infrastructure (e.g. the same
Celery broker) each get
+ their own row, so one scheduler's healthy report can never paper over
another's failing one.
+ A restarted scheduler gets a fresh row for the same reason its own Job
row is fresh -- a new
+ id -- instead of quietly taking over a row left behind by its previous
instance.
+ """
+ job = session.scalar(
+ select(Job).where(
+ Job.job_type == EXECUTOR_JOB_TYPE,
+ Job.executor_class == executor_key,
+ Job.owner_job_id == self.job.id,
+ Job.end_date.is_(None),
+ )
+ )
+ if job is None:
+ job = Job(
Review Comment:
Creating the row with `Job(...)` runs `Job.__init__`, which fires the
`on_starting` listener hook (`job.py:211`). That hook means "a component is
starting", but we are using it here every time an executor row is created; the
scheduler is already running. `before_stopping` never fires for these rows, so
a listener is going to see starts with no matching stops.
Core already has a pattern for creating a `Job` row on another component's
behalf. The Execution API's registration endpoint does a plain insert, with a
comment saying why (`api_fastapi/execution_api/routes/jobs.py:206`). The same
should work here. You do want the scheduler's host and user on these rows, so
set them explicitly:
```python
if job is None:
now = timezone.utcnow()
inserted = session.execute(
insert(Job).values(
job_type=EXECUTOR_JOB_TYPE,
executor_class=executor_key,
owner_job_id=self.job.id,
health_check_result=HealthCheckResult.UNKNOWN,
state=JobState.RUNNING,
start_date=now,
latest_heartbeat=now,
hostname=get_hostname(),
unixname=getuser(),
)
)
job_id = cast("CursorResult", inserted).inserted_primary_key[0] #
type: ignore[index]
if team_name:
session.add(JobTeam(job_id=job_id, team_name=team_name))
session.flush()
job = session.get(Job, job_id)
```
Since the team is set when the row is created, the `elif team_name:` branch
below can go, too since it currently rebuilds `team_names` on every heartbeat.
##########
airflow-core/src/airflow/jobs/scheduler_job_runner.py:
##########
@@ -425,6 +471,136 @@ def __init__(
def heartbeat_callback(self, *, session: Session = NEW_SESSION) -> None:
stats.incr("scheduler_heartbeat", 1, 1)
+ def _check_executor_health(self) -> None:
Review Comment:
Thanks for adding the new test file. `_run_executor_health_check`,
`_touch_executor_heartbeats` and `_get_or_create_executor_job` are covered now.
The one still missing is `_check_executor_health`, the method this thread is
on.
Two smaller ones while you're in here:
-` _run_executor_health_check` when `_record_executor_health_result` raises:
it should log the error and not raise.
- On the API side, the legacy status mapping: DEGRADED and all-UNKNOWN
should both give HEALTHY. There are tests for healthy, unhealthy and
never-reported, but not those two.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]