matthieuauger opened a new pull request, #72328: URL: https://github.com/apache/airflow/pull/72328
related: #71488 (its follow-up — “`clear_task_instances` has the same per-TI shape”) `clear_task_instances` did, for **every** task instance: - fetch the serialized dag row (data column included) and deserialize it — `DBDagBag.get_latest_version_of_dag` reads and deserializes on every call; - look the latest `DagVersion` up; - count its archived tries in `task_instance_history`, and look its `HITLDetail` up (`record_ti`); - delete its `task_reschedule` rows. Five round-trips and one full dag deserialization per task instance, inside one transaction that holds the locks of everything already touched. Measured on a run with 3,000 mapped task instances (Airflow 3.3.1, PostgreSQL 17 behind pgbouncer, 2.5 ms RTT, a 2-task dag): `POST clearTaskInstances` took **81 s** (finished run) to **123 s** (failed run) server-side. On a real dag the deserialization alone costs 5–30 ms per call (hundreds of ms cold), so it scales with the dag, not only with the run. This change resolves once per call what is shared by every task instance of a dag or of a run (the serialized dag, the latest `DagVersion`, the dag of a run), and looks up in bulk what varies per task instance: `TaskInstanceHistory.record_tis` finds the archived tries with one query per dag run and the HITL details with one query, and the reschedules are deleted with one statement (chunked). The outcome per task instance is unchanged — `record_ti` keeps its shape for the single-TI callers and shares the recording code. Same run, same machine: **81 s → 23 s** and **123 s → 21 s**. The new test locks the query count in: **15 → 123 queries from 3 to 30 task instances before** (19 → 154 with `run_on_latest_version`), bounded now. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
