GitHub user daniyarndh created a discussion: Native DAG-Level Time Travel for Reproducible Backfills
Currently, Airflow DAG execution logic is decoupled from its runtime execution state. When a DAG run fails or requires a historical backfill (e.g., re-running a task from 3 months ago), Airflow executes the task using **today's live code base**, current environment variables, and latest provider dependencies. This leads to some significant operational challenges: 1. **Unintended Side-Effects during Backfills:** Code refactors or column renames deployed today will break or silently corrupt data when re-running historical DAG runs. 2. **Reproducibility Friction:** Debugging past task failures is difficult because the worker environment (Python package versions, DAG logic) has shifted since the failure occurred. 3. **Auditability & Compliance:** In regulated industries, proving the exact code version and transformation logic executed at a specific point in time requires manual correlation across Git history, CI logs, and Airflow task logs. We want to introduce a `Time-Travel Pipeline runs` feature that takes a snapshot of the exact code and environment every time a pipeline starts. Airflow saves two unique identifiers directly to the run's metadata: - The exact Git code commit (the logic) - The exact container image ID (the software and dependencies) In short, it turns pipeline re-runs into true time capsules, ensuring backfills are safe, bugs are reproducible, and past reports are fully auditable. GitHub link: https://github.com/apache/airflow/discussions/73636 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
