Hi Ash, I checked this in detail: task_instance_history is per-attempt, while the proposed record is per-launch, so I don’t believe they are equivalent.
A single task attempt can be launched more than once—for example, after a stuck-queued reschedule. Each launch has a different token, but task_instance_history stores only one row per attempt and may be deleted with the TI. The launch table stores each token and atomically marks it consumed or superseded. DAG/task/run/try fields identify the attempt, but only the token identifies which launch remains valid. The distinction is: - task_instance_history: audit snapshot of a completed or replaced attempt; - task_instance_launch: durable identity and atomic state for each executor launch. One correction to my original message: saying the current 404 necessarily consumes a retry was too broad. The main issue is that it conflates an unknown task with a known obsolete launch; the typed 409 makes that distinction explicit. For context, KubernetesExecutor is spawning duplicate, stale task pods that later fail at /run with 404. These pods may trigger unnecessary node provisioning, consume cluster capacity, and delay legitimate tasks from starting. The launch record lets Airflow identify the obsolete launch explicitly, while scheduler-side token validation can prevent stale pods from being created in the first place. On Mon, Aug 3, 2026 at 6:13 PM Ash Berlin-Taylor <[email protected]> wrote: > We already have the task_instance_history row — what is the difference > between that/what we store in there+when, and your option B? > > > TaskInstance` row, so a late/stale launch surfaces as an opaque > > `404 not_found` — indistinguishable from a genuinely unknown task. That > > consumes a retry and/or blocks clean requeueing > > I don’t understand what you mean by this. Can you expand on this? We could > tell no such task vs task already in an invalid state without DB changes by > looking in the task_instance_history table if we have the task/run/dag ids > (which if they aren’t in the JWT token today could very easily be added) > > -ash > > > On 3 Aug 2026, at 16:00, Guilherme Silva Gonçalves <[email protected]> > wrote: > > > > Hi all, > > > > I'd like the list's input on which of two approaches to pursue for #69760 > > before either is hardened for review. I've prototyped both: > > > > Minimal (KubernetesExecutor, equality check): > > https://github.com/apache/airflow/pull/69782 > > Durable redesign (all executors, RFC/Draft): > > https://github.com/apache/airflow/pull/70931 > > Issue: #69760 > > > > ## Problem > > > > When a task instance transitions out of its original launch (scheduler > > failover, pod adoption, stuck-queued reschedule, orphan reset, or a > manual > > clear), the executor originally assigned may still call the Execution API > > `/run` for that task. Today the launch identity lives only on the mutable > > `TaskInstance` row, so a late/stale launch surfaces as an opaque > > `404 not_found` — indistinguishable from a genuinely unknown task. That > > consumes a retry and/or blocks clean requeueing. > > > > ## Two candidate designs > > > > Both share the same goal — reject a stale/superseded launch at `/run` — > but > > differ in where the launch identity lives and how far the change reaches. > > > > ### Option A — reuse `external_executor_id`, equality check (PR #69782) > > > > - No schema change. Reuses the existing scheduler capability for > executors > > that > > set `pre_assigns_external_executor_id = True`: at QUEUED the scheduler > > writes a > > fresh `external_executor_id` UUID to the `task_instance` row. > > - Opts KubernetesExecutor into that path, threads the token through the > K8s > > workload lifecycle (preserves it across queued events so the scheduler > > doesn't > > overwrite it with the job id, annotates the pod, revalidates the DB row > > just > > before pod creation). > > - `/run` rejects a worker whose token no longer equals the current TI > token. > > - Scope: KubernetesExecutor. Identity lives on the mutable/deletable TI > row; > > it's an equality check, not a lifecycle. > > > > ### Option B — durable launch-record table, state machine (PR #70931, > > Draft/RFC) > > > > - New durable `task_instance_launch` table (with migration) storing an > > immutable > > token independent of the mutable/deleted `TaskInstance` row, with an > > explicit > > `active -> consumed | superseded` lifecycle and guarded transitions. > > - Executor-agnostic (Celery / K8s / Edge). The launch record is written > > atomically at `SCHEDULED -> QUEUED`. > > - Supersession is first-class (stuck-queued reschedule, failed adoption, > > orphan > > reset, clear/next-try); successful adoption preserves the token. > > - `/run` returns a typed `409 stale_executor_launch` for a known > *terminal* > > token; unknown tokens still `404`. Gated behind Execution API version > > `v2026_06_30` (Cadwyn) so older Task SDK clients keep the legacy `404`. > > - Task SDK maps `409` to a new `TaskInstanceSupersededError`; a > superseded > > worker logs once and exits 0 rather than failing. > > > > ## Open questions for the list > > > > 1. Which direction fits Airflow best — the minimal equality check on the > > existing field (A), or the durable, executor-agnostic launch record > (B)? > > Is the extra table + migration in B justified, or is A's "no schema > > change, > > reuse `external_executor_id`" preferable and B a later follow-up? > > 2. Is a typed `409 stale_executor_launch` on `/run` the contract you'd > > want, and > > is gating it behind a new Execution API version the accepted > > compatibility > > story here? Or is A's simpler reject sufficient? > > 3. Should this stay KubernetesExecutor-scoped (A) or land > executor-agnostic > > from the first cut (B)? > > 4. Does either warrant an AIP, or is a PR + this thread sufficient given > > it's an > > additive change (schema/table in B) plus a versioned API behaviour? > > > > Both PRs have Testing sections (unit + a live KubernetesExecutor > > kind-cluster > > validation on #70931) and #70931 carries a `significant` newsfragment. > I'll > > consolidate onto whichever approach the list favours and close the other. > > Development was AI-assisted (disclosed in the PRs). > > > > P.s.: I used fable 5 for developing this. > > Thanks, > > Guilherme > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
