GitHub user wolvery created a discussion: DISCUSS Durable executor launch records + typed 409 for stale /run attempts (PR #70931, refs #69760)
Hi all, I'd like the list's input on which of two approaches to pursue for #69760 before either is hardened for review. I've prototyped both: Minimal (KubernetesExecutor, equality check): https://github.com/apache/airflow/pull/69782 Durable redesign (all executors, RFC/Draft): https://github.com/apache/airflow/pull/70931 Issue: #69760 ## Problem When a task instance transitions out of its original launch (scheduler failover, pod adoption, stuck-queued reschedule, orphan reset, or a manual clear), the executor originally assigned may still call the Execution API `/run` for that task. Today the launch identity lives only on the mutable `TaskInstance` row, so a late/stale launch surfaces as an opaque `404 not_found` — indistinguishable from a genuinely unknown task. That consumes a retry and/or blocks clean requeueing. ## Two candidate designs Both share the same goal — reject a stale/superseded launch at `/run` — but differ in where the launch identity lives and how far the change reaches. ### Option A — reuse `external_executor_id`, equality check (PR #69782) - No schema change. Reuses the existing scheduler capability for executors that set `pre_assigns_external_executor_id = True`: at QUEUED the scheduler writes a fresh `external_executor_id` UUID to the `task_instance` row. - Opts KubernetesExecutor into that path, threads the token through the K8s workload lifecycle (preserves it across queued events so the scheduler doesn't overwrite it with the job id, annotates the pod, revalidates the DB row just before pod creation). - `/run` rejects a worker whose token no longer equals the current TI token. - Scope: KubernetesExecutor. Identity lives on the mutable/deletable TI row; it's an equality check, not a lifecycle. ### Option B — durable launch-record table, state machine (PR #70931, Draft/RFC) - New durable `task_instance_launch` table (with migration) storing an immutable token independent of the mutable/deleted `TaskInstance` row, with an explicit `active -> consumed | superseded` lifecycle and guarded transitions. - Executor-agnostic (Celery / K8s / Edge). The launch record is written atomically at `SCHEDULED -> QUEUED`. - Supersession is first-class (stuck-queued reschedule, failed adoption, orphan reset, clear/next-try); successful adoption preserves the token. - `/run` returns a typed `409 stale_executor_launch` for a known *terminal* token; unknown tokens still `404`. Gated behind Execution API version `v2026_06_30` (Cadwyn) so older Task SDK clients keep the legacy `404`. - Task SDK maps `409` to a new `TaskInstanceSupersededError`; a superseded worker logs once and exits 0 rather than failing. ## Open questions for the list 1. Which direction fits Airflow best — the minimal equality check on the existing field (A), or the durable, executor-agnostic launch record (B)? Is the extra table + migration in B justified, or is A's "no schema change, reuse `external_executor_id`" preferable and B a later follow-up? 2. Is a typed `409 stale_executor_launch` on `/run` the contract you'd want, and is gating it behind a new Execution API version the accepted compatibility story here? Or is A's simpler reject sufficient? 3. Should this stay KubernetesExecutor-scoped (A) or land executor-agnostic from the first cut (B)? 4. Does either warrant an AIP, or is a PR + this thread sufficient given it's an additive change (schema/table in B) plus a versioned API behaviour? Both PRs have Testing sections (unit + a live KubernetesExecutor kind-cluster validation on #70931) and #70931 carries a `significant` newsfragment. I'll consolidate onto whichever approach the list favours and close the other. Development was AI-assisted (disclosed in the PRs). Extra: task_instance_history is per-attempt, while the proposed record is per-launch, so I don’t believe they are equivalent. A single task attempt can be launched more than once—for example, after a stuck-queued reschedule. Each launch has a different token, but task_instance_history stores only one row per attempt and may be deleted with the TI. The launch table stores each token and atomically marks it consumed or superseded. DAG/task/run/try fields identify the attempt, but only the token identifies which launch remains valid. The distinction is: - task_instance_history: audit snapshot of a completed or replaced attempt; - task_instance_launch: durable identity and atomic state for each executor launch. Just to be clear: saying the current 404 necessarily consumes a retry was too broad. The main issue is that it conflates an unknown task with a known obsolete launch; the typed 409 makes that distinction explicit. For context, KubernetesExecutor is spawning duplicate, stale task pods that later fail at /run with 404. These pods may trigger unnecessary node provisioning, consume cluster capacity, and delay legitimate tasks from starting. The launch record lets Airflow identify the obsolete launch explicitly, while scheduler-side token validation can prevent stale pods from being created in the first place. P.s.: I used fable 5 for developing this. Thanks, Guilherme GitHub link: https://github.com/apache/airflow/discussions/72473 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
