luc-pimentel commented on issue #68721: URL: https://github.com/apache/airflow/issues/68721#issuecomment-6256270078
I reproduced this on 3.3.2 with Postgres 16. A backfill of 35,137 runs took 4 min 56 s to create. The scheduler marked it complete 2 min 14 s after `created_at`, while the runs were still inside the creation transaction. When creation finished, every run was queued under a completed backfill and cancel answered 409. A 289-run backfill, created in 2 s, works fine. The cause is the early commit in `_create_backfill`: the Backfill row is committed, then the runs are written in a transaction that commits at the end. Until then the scheduler sees a backfill with no runs, and after two minutes it marks it complete. I'd like to fix it the way #62561, #68125 and #68705 asked for: one transaction, with `session.flush()` instead of the early commit. That leaves one question. The one-active-backfill check relies on that early commit, and with one transaction a second create for the same Dag can slip through while the first is still writing. I see three ways to handle that, and I'd like you to pick one. - keep the check as is, with a wider window (what #68705 does) - lock the `DagModel` row for the creation (#68125; a review in #67900 asked for the same lock to be reverted, since a lock held for minutes blocks other updates of that row) - a partial unique index on `backfill(dag_id) WHERE completed_at IS NULL`, which needs a migration and a MySQL workaround I'll open the PR against whichever you choose, crediting #68125 and #68705. --- Drafted-by: Claude Code 2.1.303; reviewed by @luc-pimentel before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
