Hi all, I'd like the list's input on which of two approaches to pursue for #69760 before either is hardened for review. I've prototyped both:
Minimal (KubernetesExecutor, equality check): https://github.com/apache/airflow/pull/69782 Durable redesign (all executors, RFC/Draft): https://github.com/apache/airflow/pull/70931 Issue: #69760 ## Problem When a task instance transitions out of its original launch (scheduler failover, pod adoption, stuck-queued reschedule, orphan reset, or a manual clear), the executor originally assigned may still call the Execution API `/run` for that task. Today the launch identity lives only on the mutable `TaskInstance` row, so a late/stale launch surfaces as an opaque `404 not_found` — indistinguishable from a genuinely unknown task. That consumes a retry and/or blocks clean requeueing. ## Two candidate designs Both share the same goal — reject a stale/superseded launch at `/run` — but differ in where the launch identity lives and how far the change reaches. ### Option A — reuse `external_executor_id`, equality check (PR #69782) - No schema change. Reuses the existing scheduler capability for executors that set `pre_assigns_external_executor_id = True`: at QUEUED the scheduler writes a fresh `external_executor_id` UUID to the `task_instance` row. - Opts KubernetesExecutor into that path, threads the token through the K8s workload lifecycle (preserves it across queued events so the scheduler doesn't overwrite it with the job id, annotates the pod, revalidates the DB row just before pod creation). - `/run` rejects a worker whose token no longer equals the current TI token. - Scope: KubernetesExecutor. Identity lives on the mutable/deletable TI row; it's an equality check, not a lifecycle. ### Option B — durable launch-record table, state machine (PR #70931, Draft/RFC) - New durable `task_instance_launch` table (with migration) storing an immutable token independent of the mutable/deleted `TaskInstance` row, with an explicit `active -> consumed | superseded` lifecycle and guarded transitions. - Executor-agnostic (Celery / K8s / Edge). The launch record is written atomically at `SCHEDULED -> QUEUED`. - Supersession is first-class (stuck-queued reschedule, failed adoption, orphan reset, clear/next-try); successful adoption preserves the token. - `/run` returns a typed `409 stale_executor_launch` for a known *terminal* token; unknown tokens still `404`. Gated behind Execution API version `v2026_06_30` (Cadwyn) so older Task SDK clients keep the legacy `404`. - Task SDK maps `409` to a new `TaskInstanceSupersededError`; a superseded worker logs once and exits 0 rather than failing. ## Open questions for the list 1. Which direction fits Airflow best — the minimal equality check on the existing field (A), or the durable, executor-agnostic launch record (B)? Is the extra table + migration in B justified, or is A's "no schema change, reuse `external_executor_id`" preferable and B a later follow-up? 2. Is a typed `409 stale_executor_launch` on `/run` the contract you'd want, and is gating it behind a new Execution API version the accepted compatibility story here? Or is A's simpler reject sufficient? 3. Should this stay KubernetesExecutor-scoped (A) or land executor-agnostic from the first cut (B)? 4. Does either warrant an AIP, or is a PR + this thread sufficient given it's an additive change (schema/table in B) plus a versioned API behaviour? Both PRs have Testing sections (unit + a live KubernetesExecutor kind-cluster validation on #70931) and #70931 carries a `significant` newsfragment. I'll consolidate onto whichever approach the list favours and close the other. Development was AI-assisted (disclosed in the PRs). P.s.: I used fable 5 for developing this. Thanks, Guilherme
