Abdulrehman-PIAIC80387 opened a new pull request, #68705: URL: https://github.com/apache/airflow/pull/68705
When a backfill is created via `POST /api/v2/backfills`, `_create_backfill` commits the `Backfill` row **before** creating its dag runs. If run creation then fails — the reported case is `sqlite3.OperationalError: database is locked` under concurrent requests, but any error would do — the already-committed `Backfill` row survives with no/partial runs. The `num_active > 0` check then treats it as an in-progress backfill and blocks **all** future backfills for that dag with "already running backfill". So this is an atomicity problem, not really SQLite-specific: any failure mid-creation leaves an orphaned, un-removable backfill. ## Fix Make creation self-healing: wrap run creation in a `try/except` and, on any failure, roll back the in-flight work and delete the orphaned `Backfill` row (its `BackfillDagRun` rows cascade) before re-raising. A failed creation now leaves no rows behind, so the dag is not blocked. This keeps the existing early commit (which also acts as the concurrency guard for the "one active backfill per dag" check), so it does not change concurrency behaviour — it only ensures a failed attempt cleans up after itself. ## Notes There is a separate, pre-existing concurrency question (two simultaneous requests both passing the `num_active` check) that would need a DB-level guard to fully close; I raised both options on the issue. This PR deliberately scopes to the orphaned-backfill bug that's actually reported. ## Tests Added `test_create_backfill_no_orphan_on_run_creation_failure`: forces a failure during run creation and asserts no `Backfill` row remains and a subsequent backfill succeeds. closes: #68699 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Mirza Generated-by: Mirza following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
