dstandish opened a new issue, #71453:
URL: https://github.com/apache/airflow/issues/71453
## Background
`created_dag_version_id`
(`airflow-core/src/airflow/models/dagrun.py:301-304`) was added in #49097 with
this documented contract:
> "The id of the dag version column that was in effect at dag run creation
time."
and the paired relationship docstring:
> "The dag version that was active when the dag run was created, if
available."
At the time this was accurate: for a bundle-versioned (pinned) run, the
version recorded at creation was — by design — the version that run would run
forever. Per a conversation with the original author of AIP-65 DAG versioning:
bundle-versioned runs were always meant to execute the version pinned at
creation; the way to get "run with the latest code" behavior was to *not* use a
versioned bundle at all (unpinned/local bundles always resolve to the latest
`DagVersion` — see `DBDagBag._version_from_dag_run`,
`airflow-core/src/airflow/models/dagbag.py:210-216`). There was never supposed
to be a way to move a pinned run off the version it was created with.
## What changed
Starting with #52177 (introduces the `run_on_latest_version` flag) and
#54984 ("Run verify_integrity when cleared to run on latest"), and extended by
#59764 and #65835/#66901, `created_dag_version_id` began being **mutated**
after creation:
- `_update_dagrun_to_latest_version` —
`airflow-core/src/airflow/models/taskinstance.py:344`
- `clear_task_instances(..., run_on_latest_version=True)` —
`airflow-core/src/airflow/models/taskinstance.py:474` and `:497`
An open PR, #71425, is currently extending this same mutation to
running/queued DagRuns.
None of these renamed or re-documented the field. It still says "recorded at
creation" while several code paths now treat it as "the version this run should
currently be considered pinned to" — two different, and now conflated, meanings
living in one column.
## Why this matters
The mismatch between the documented contract and actual behavior has already
produced concrete defects downstream, because other code correctly relied on
the *original* contract and nothing flagged that the contract had changed
underneath it:
- `DagRun.dag_versions` silently drops versions still in use by uncleared
task instances after a partial "run on latest version" clear.
- `only_new` clear (`_get_new_task_ids`) can report zero new tasks when
there genuinely are some, for the same reason.
(Filed as separate issues — linked below once created.)
Both are symptoms, not the root cause. The root cause is that one field is
now asked to represent two different things — an immutable historical fact, and
a mutable "current pinned version" pointer — with no way for a reader to tell
which one they're getting, or whether the run has been through a partial
version bump that leaves it internally inconsistent.
## What needs deciding
This is a design question as much as a naming one:
1. **Does "run on latest version" belong on bundle-versioned/pinned runs at
all?** The original design intent was that pinning is absolute, and "latest" is
achieved by not pinning. If the answer is no, the mutation added since #54984
should be reconsidered/reverted for the pinned case.
2. **If the capability is intentionally kept**, `created_dag_version_id`
should stop being overloaded: introduce a genuinely separate, clearly-named
mutable field for "the version this pinned run is currently expected to run at"
(e.g. `pinned_dag_version_id` or `current_dag_version_id`), keep
`created_dag_version_id` strictly immutable as documented, and audit every
consumer (`_version_from_dag_run`, `get_dag_for_run_or_latest_version`,
`DagRun.dag_versions`, `_get_new_task_ids`, and anything else reading this
field) to use the correct one.
Either way, the current state — one field, two incompatible meanings, no
rename, no updated docs — should be resolved before more functionality (e.g.
#71425) is built on top of it.
---
Drafted-by: Claude Code (Sonnet 5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
