GitHub user daniyarndh edited a discussion: Native DAG-Level Time Travel for
Reproducible Backfills
Currently, Airflow DAG execution logic is decoupled from its runtime execution
state. When a DAG run fails or requires a historical backfill (e.g., re-running
a task from 3 months ago), Airflow executes the task using **today's live code
base**, current environment variables, and latest provider dependencies.
This leads to some significant operational challenges:
1. **Unintended Side-Effects during Backfills:** Code refactors or column
renames deployed today will break or silently corrupt data when re-running
historical DAG runs.
2. **Reproducibility Friction:** Debugging past task failures is difficult
because the worker environment (Python package versions, DAG logic) has shifted
since the failure occurred.
3. **Auditability & Compliance:** In regulated industries, proving the exact
code version and transformation logic executed at a specific point in time
requires manual correlation across Git history, CI logs, and Airflow task logs.
## Proposed Solution
We want to introduce a `Time-Travel Pipeline runs` feature that takes a
snapshot of the exact code and environment every time a pipeline starts.
Airflow saves two unique identifiers directly to the run's metadata:
- The exact Git code commit (the logic)
- The exact container image ID (the software and dependencies)
In short, it turns pipeline re-runs into true time capsules, ensuring backfills
are safe, bugs are reproducible, and past reports are fully auditable. Example:
```json
{
"execution_snapshot": {
"git_commit": "4f82d1c67a",
"docker_digest":
"[registry.example.com/airflow-dags@sha256:a3f128c98e](https://registry.example.com/airflow-dags@sha256:a3f128c98e)...",
"pinned_at": "2026-09-23T20:00:00Z"
}
}
GitHub link: https://github.com/apache/airflow/discussions/73636
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
