Nonetheless I support this idea :) I think it would be great to add executor health status on the dashboard. That would give more visibility to Airflow users.
On 2026/10/22 19:24:50 Joel Emmanuel wrote: > Thanks Vincent, that makes sense. Dropping the 72-hour framing, let's just > treat this as an open discussion. Appreciate the correction. > > Joel > > On Thu, 22 Oct 2026 at 15:17, Vincent Beck <[email protected]> wrote: > > > It is either a discuss thread or a lazy consensus but not both :) If you > > want to start a lazy consensus, then the thread subject should call it out > > with [LAZY CONSENSUS] instead of [DISCUSS]. Though, in this case I think a > > discuss thread makes more sense since many people do not have the context > > and 72 hours is a bit rough to make an opinion :) > > > > On 2026-10-22 18:56 UTC Joel Emmanuel wrote: > > > Hi all, > > > > > > I'm an MLH Fellowship participant working on this with Dennis Ferruzzi > > > (mentor), posting here per his guidance before any implementation starts. > > > > > > Problem > > > > > > /api/v2/monitor/health already reports per-instance liveness for > > Scheduler, > > > Triggerer, and DagProcessor. Each registers itself as a row in the Jobs > > > table (airflow.jobs.job.Job) with a job_type, hostname, and a > > > latest_heartbeat that updates on a timer; the health check queries > > > unfinished rows of that job_type and reports which ones are still > > > heartbeating. This is the per-instance shape PR #70416 (Jung-Hyun Andrew > > > Kim) added, with Vincent Beck's #73222 building on top of that work. > > > > > > Executors don't participate in this at all. There's no ExecutorJobRunner > > > and no executor job_type anywhere in airflow-core/src/airflow/jobs/. > > > BaseExecutor has a heartbeat() method, but it's purely internal: it > > drives > > > the executor's own sync() loop, and none of that is written anywhere the > > > health endpoint can read. > > > > > > In practice, if an executor crashes, gets wedged, or loses its connection > > > to its backend, there's no dedicated signal for that. Metrics like > > > scheduler.executor_heartbeat_duration and executor.open_slots exist if > > > enabled (they're off by default) and could in theory be pieced together > > by > > > someone watching them, but none of that surfaces a dead executor as > > clearly > > > as heartbeat staleness already does for Scheduler, Triggerer, and > > > DagProcessor. What actually reaches the health endpoint today can even be > > > misleading: if an executor's heartbeat() call hangs, it shows up as "the > > > scheduler is hung," which might be true, but isn't necessarily the real > > > cause. > > > > > > Proposed approach > > > > > > The goal is to make a stuck or dead executor show up the same way a stuck > > > Scheduler or Triggerer already does: through the existing health > > endpoint, > > > using the existing status / detailed_status / instances shape, so > > operators > > > have one place to check instead of a special case just for executors. > > This > > > is meant as a first step, not the whole fix. The longer-term goal is > > > letting executors eventually run as independent processes, and this > > health > > > check is the visibility work that needs to exist before that > > restructuring > > > can happen safely. > > > > > > Concretely: > > > > > > - Dashboard placement: executor health is a new top-level entry alongside > > > Scheduler/Triggerer/DagProcessor, not nested under Scheduler. > > > - Storage: reuse the Jobs table instead of adding a new one. A new > > > ExecutorJob row reports state=RUNNING for liveness, the same bookkeeping > > > pattern the other three components already use, plus a new column on the > > > Jobs table holding the actual health verdict (ExecutorHealth: > > > healthy/unhealthy (reason) / unknown, default unknown). That avoids > > forcing > > > JobState (RUNNING/SUCCESS/RESTARTING/FAILED) to carry meaning it was > > never > > > designed for, since none of those values maps cleanly onto executor > > status. > > > - Mechanism: the scheduler polls each executor on a timer via > > > get_executor_health(), a timed call to executor.get_health() that runs > > in a > > > thread with a timeout and skips a poll if the previous one is still > > > pending. It's registered next to the existing > > _handle_tasks_stuck_in_queued > > > timer. Executors reply healthy, unhealthy (with a reason), or unknown, > > > where unknown is the default so an executor that hasn't implemented a > > > health check doesn't show up as dead. No reply at all, though, counts as > > > unhealthy. Three things here protect the main scheduler loop, which must > > > never be delayed: the worker thread, the timeout, and the in-flight > > guard. > > > The scheduler's other periodic actions, like > > _handle_tasks_stuck_in_queued, > > > run inline on the main loop thread because they only touch the local > > > database and return fast. get_executor_health() can't assume that: it > > calls > > > out to the executor itself, a broker or a cluster API, and that call can > > > hang for reasons the scheduler doesn't control, so it gets its own worker > > > thread rather than running inline. The timeout also lives on the > > > scheduler's side rather than inside each executor's get_health(). The > > > scheduler can't guarantee every executor implementation bounds its own > > call > > > correctly, so a slow or hung executor should never be the one deciding > > how > > > long the main loop waits. Finally, the in-flight guard skips a new poll > > if > > > the previous one hasn't returned yet, so one stuck executor can't pile up > > > threads by never responding. > > > - Executor-instance identity: for Celery, the broker counts as the > > > instance. Celery workers are ephemeral and scale dynamically, so there's > > no > > > stable per-worker identity to key off (worker concurrency settings cap > > how > > > many processes run; they don't define an expected worker count). For > > > Kubernetes, the pod manager/cluster counts as the instance for the same > > > reason: pod counts aren't fixed in config either. LocalExecutor is out of > > > scope for now, since its liveness is just the scheduler's own liveness, > > so > > > a dedicated check wouldn't show anything new. > > > - Scope split: PR1 is core scaffolding only, covering the Alembic > > migration > > > for the new column, the storage and endpoint changes, the ExecutorHealth > > > enum aggregating into detailed_status, and tests. PR2 adds the > > > CeleryExecutor-specific get_health() override, which is the PoC target > > for > > > this round. LocalExecutor and Kubernetes each get a named follow-up issue > > > instead of a design we'd just be guessing at right now. > > > > > > What this deliberately doesn't do > > > > > > - Doesn't design LocalExecutor's or KubernetesExecutor's checks yet. > > That's > > > a deliberate follow-up, not an oversight. > > > - Doesn't pretend the Job-row pattern is a perfect fit just because it's > > > precedent. It's reused because it avoids duplicating liveness bookkeeping > > > that Scheduler, Triggerer, and DagProcessor already do, not because > > > JobState was ever designed with executors in mind. > > > - Doesn't try to solve process separation. This is the visibility layer a > > > later restructuring would depend on, and it's scoped narrowly on purpose. > > > > > > Posting this under lazy consensus. Will plan to open the first PR (core > > > scaffolding, no behavior change for existing executors) if there's no > > > objection in 72 hours. > > > > > > Joel > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
