Hi everyone,
I want to start another discussion about a common issue with how Airflow
runs historical data: the lack of code and environment freezing for past
runs.
Right now, Airflow's *workflow logic is separated from its runtime
environment*. When a past workflow fails or we need to run a historical
backfill (for example, re-running a task from 3 months ago), Airflow
executes that past task using today’s live code, current environment
variables, and the latest software dependencies.
This causes a few major problems:
- Broken Backfills: If we renamed a database column or refactored code
today, running a backfill for last year will either break completely or
silently corrupt old data.
- Hard to Debug: It is very difficult to debug past task failures
because the code and Python packages on the worker have changed since the
error happened.
- Compliance & Audits: In regulated industries, proving the exact code
version that generated a financial or user report 6 months ago is
incredibly painful. It requires manually matching Git history, CI logs, and
Airflow logs.
Proposed Solution: "Time-Travel Pipeline Runs"
I propose saving a snapshot of the exact code and environment every time a
pipeline starts. Airflow would save two unique identifiers directly to the
pipeline run's metadata:
1. The Git commit hash (the exact code logic).
2. The container image ID (the exact software versions and dependencies).
This turns pipeline re-runs into true time capsules. If you retry a failed
task from last month, Airflow will instruct the executor to pull and run
the exact container image and code version that was active last month.
Example metadata structure:
*{ "execution_snapshot": { "git_commit": "4f82d1c67a",
"docker_digest": "registry.example.com/airflow-dags@sha256:a3f128c..
<http://registry.example.com/airflow-dags@sha256:a3f128c..>.",
"pinned_at": "2026-09-23T20:00:00Z" }}*
I would love to get your initial feedback here on the mailing list, and I
look forward to introducing this concept on one of future dev calls!
Best regards, Daniyar
