dheerajturaga opened a new pull request, #75344: URL: https://github.com/apache/airflow/pull/75344
Part of [AIP-125 (Inbound Webhooks)](https://lists.apache.org/thread/p3dpnor4zwko8tldpncj6ftj1bpp18q7), which lets external services such as GitHub trigger Dags by sending webhooks to Airflow. A task that waits for a webhook delivery parks in `awaiting_input` with no trigger and no Human-in-the-loop form, so nothing in the UI or the REST API said what it was waiting for, and there was no number for how many tasks wait on a webhook. This shows a task's webhook wait and catches the deliveries page up with the waits unit (#75340 to #75343). - `GET /api/v2/dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/webhookWaits` (experimental) lists the waits of a task instance's current try, or of the try that `try_number` names, newest first and never with a payload. It takes read access to the task instance, as the task instance's other details do, not to the webhook's connection. It is a route of its own rather than a field of the task instance response, which is also listed in bulk, and its router is included before the task instances router, whose `/taskInstances/{task_id}/{map_index}` would otherwise take the path. - The task instance's Details tab shows the selected try's newest wait: its webhook, status, events, match, deadline and, once a delivery matched it, a link to that delivery. Nothing closes a wait when its try stops, so for a try that is no longer running or parked while its wait is still `registered` or `received`, the panel says that no delivery will resume it. - The deliveries page offers the `early`, `expired` and `resumed_wait` outcomes, and its details panel lists the waits a delivery went to, each linked to its task instance. For those links, a delivery's `waits` actions now record the task instance's public map index, plus `region_id` and `region_index` inside a dynamic region: the stored index alone pointed a task in a loop at a mapped task instance that does not exist. - The scheduler emits `webhooks.open_waits`, a gauge tagged by webhook, from one grouped query each `[scheduler] ti_metrics_interval`. It counts only the open waits whose task is running or parked, since a wait stays open after its task stops, and reports 0 once for a webhook left with none. This merges after the four PRs of the waits unit. ### Performance Measured on SQLite on the host, Python 3.12. The data has 50 Dags, 1,000 Dag runs and 20,000 task instances. 1,400 of them are in the states that `_emit_ti_metrics` reports, and 700 are running or awaiting_input. The "with waits" copy of the database adds 2,350 `inbound_wait` rows across 5 webhooks: 300 open waits on live task instances, 50 open waits on failed ones, and 2,000 closed waits. | Path | Queries base → head | Base median | Head median | Change | Control (`_emit_dag_runs_metric`) base / head | |---|---|---|---|---|---| | `SchedulerJobRunner._emit_ti_metrics`, no webhook waits | 1 → 2 | 4.008 ms | 4.161 ms | +0.152 ms (+3.8%) | 0.225 ms / 0.221 ms (-2.1%) | | `SchedulerJobRunner._emit_ti_metrics`, 2,350 waits (350 open, 300 on live task instances), 5 webhooks | 1 → 2 | 3.946 ms | 4.334 ms | +0.388 ms (+9.8%) | 0.225 ms / 0.221 ms (-2.1%) | The layer makes `_emit_ti_metrics` slower: +0.15 ms per call (+3.8%) with no waits and +0.39 ms (+9.8%) with 350 open waits. Every head run was slower than every base run, and the control moved only -2%. The cost comes from the one extra grouped query for the open waits. That is acceptable because the method runs only when metrics are enabled, and then once per `ti_metrics_interval` (30 s by default), so the extra cost is under 0.4 ms every 30 s. The query starts from the `ti_state` index and looks up waits through `idx_inbound_wait_ti_id`, so its cost grows with the number of live task instances, not with the history of closed waits. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
