We already have the task_instance_history row — what is the difference between that/what we store in there+when, and your option B?
> TaskInstance` row, so a late/stale launch surfaces as an opaque > `404 not_found` — indistinguishable from a genuinely unknown task. That > consumes a retry and/or blocks clean requeueing I don’t understand what you mean by this. Can you expand on this? We could tell no such task vs task already in an invalid state without DB changes by looking in the task_instance_history table if we have the task/run/dag ids (which if they aren’t in the JWT token today could very easily be added) -ash > On 3 Aug 2026, at 16:00, Guilherme Silva Gonçalves <[email protected]> wrote: > > Hi all, > > I'd like the list's input on which of two approaches to pursue for #69760 > before either is hardened for review. I've prototyped both: > > Minimal (KubernetesExecutor, equality check): > https://github.com/apache/airflow/pull/69782 > Durable redesign (all executors, RFC/Draft): > https://github.com/apache/airflow/pull/70931 > Issue: #69760 > > ## Problem > > When a task instance transitions out of its original launch (scheduler > failover, pod adoption, stuck-queued reschedule, orphan reset, or a manual > clear), the executor originally assigned may still call the Execution API > `/run` for that task. Today the launch identity lives only on the mutable > `TaskInstance` row, so a late/stale launch surfaces as an opaque > `404 not_found` — indistinguishable from a genuinely unknown task. That > consumes a retry and/or blocks clean requeueing. > > ## Two candidate designs > > Both share the same goal — reject a stale/superseded launch at `/run` — but > differ in where the launch identity lives and how far the change reaches. > > ### Option A — reuse `external_executor_id`, equality check (PR #69782) > > - No schema change. Reuses the existing scheduler capability for executors > that > set `pre_assigns_external_executor_id = True`: at QUEUED the scheduler > writes a > fresh `external_executor_id` UUID to the `task_instance` row. > - Opts KubernetesExecutor into that path, threads the token through the K8s > workload lifecycle (preserves it across queued events so the scheduler > doesn't > overwrite it with the job id, annotates the pod, revalidates the DB row > just > before pod creation). > - `/run` rejects a worker whose token no longer equals the current TI token. > - Scope: KubernetesExecutor. Identity lives on the mutable/deletable TI row; > it's an equality check, not a lifecycle. > > ### Option B — durable launch-record table, state machine (PR #70931, > Draft/RFC) > > - New durable `task_instance_launch` table (with migration) storing an > immutable > token independent of the mutable/deleted `TaskInstance` row, with an > explicit > `active -> consumed | superseded` lifecycle and guarded transitions. > - Executor-agnostic (Celery / K8s / Edge). The launch record is written > atomically at `SCHEDULED -> QUEUED`. > - Supersession is first-class (stuck-queued reschedule, failed adoption, > orphan > reset, clear/next-try); successful adoption preserves the token. > - `/run` returns a typed `409 stale_executor_launch` for a known *terminal* > token; unknown tokens still `404`. Gated behind Execution API version > `v2026_06_30` (Cadwyn) so older Task SDK clients keep the legacy `404`. > - Task SDK maps `409` to a new `TaskInstanceSupersededError`; a superseded > worker logs once and exits 0 rather than failing. > > ## Open questions for the list > > 1. Which direction fits Airflow best — the minimal equality check on the > existing field (A), or the durable, executor-agnostic launch record (B)? > Is the extra table + migration in B justified, or is A's "no schema > change, > reuse `external_executor_id`" preferable and B a later follow-up? > 2. Is a typed `409 stale_executor_launch` on `/run` the contract you'd > want, and > is gating it behind a new Execution API version the accepted > compatibility > story here? Or is A's simpler reject sufficient? > 3. Should this stay KubernetesExecutor-scoped (A) or land executor-agnostic > from the first cut (B)? > 4. Does either warrant an AIP, or is a PR + this thread sufficient given > it's an > additive change (schema/table in B) plus a versioned API behaviour? > > Both PRs have Testing sections (unit + a live KubernetesExecutor > kind-cluster > validation on #70931) and #70931 carries a `significant` newsfragment. I'll > consolidate onto whichever approach the list favours and close the other. > Development was AI-assisted (disclosed in the PRs). > > P.s.: I used fable 5 for developing this. > Thanks, > Guilherme --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
