subhash-0000 commented on code in PR #60818: URL: https://github.com/apache/airflow/pull/60818#discussion_r2777171784
########## airflow-core/docs/core-concepts/backfill.rst: ########## @@ -47,6 +47,24 @@ Run ordering You can run your backfill in reverse, i.e. latest runs first. The CLI option is ``--run-backwards``. +Running backfills on paused DAGs +--------------------------------- + +Airflow allows backfills to run on paused DAGs. This is useful when: + +* You need to reprocess historical data without activating the regular DAG schedule +* The data source is not available for regular runs +* You want to avoid unnecessary computation and logs from scheduled triggers + +When you create a backfill on a paused DAG, only the backfill runs will execute—the DAG remains paused and will not be triggered by its regular schedule. + +**Via UI**: When creating a backfill for a paused DAG, you can: + +- **Check "Unpause on trigger"**: This will unpause the DAG before running the backfill (and regular schedules will resume) +- **Uncheck "Unpause on trigger"** (default): The backfill will run but the DAG stays paused Review Comment: Hi @pierrejeambrun, you're absolutely right—I apologize for not testing this manually earlier. My unit tests passed, but they didn't capture the full scheduler integration. Root Cause: While the DagRuns were created, the Task Instances were being blocked at three checkpoints in the scheduler: Scheduler Task Query: The query wasn't fetching TIs from paused DAGs even if they were part of a backfill. Task Dependency Check: Tasks were failing the DagUnpausedDep validation. State Update: The scheduler was actively trying to set backfill runs to a finished state because the DAG was paused. Fixes Applied: I've added backfill_id exceptions to: scheduler_job_runner.py (Task query logic and state update loop) dag_unpaused_dep.py (Bypassing the unpaused dependency for backfills) The changes are pushed and CI is running. Could you please take another look? The tasks should now transition from Queued to Running as expected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
