dheerajturaga opened a new pull request, #75341: URL: https://github.com/apache/airflow/pull/75341
Part of [AIP-125 (Inbound Webhooks)](https://lists.apache.org/thread/p3dpnor4zwko8tldpncj6ftj1bpp18q7), which lets external services such as GitHub trigger Dags by sending webhooks to Airflow. A task that waits for a webhook delivery registers its wait (#75340) and parks in `awaiting_input`, the state Human-in-the-loop tasks park in since #68028. The wait can receive its delivery before the task parks: an early delivery claimed at registration, or one that matched while the task was still running ({{PR-27}}). This makes such a task resume when it parks, and makes a wait that nothing matches end at its deadline. - At park time the Execution API locks the task instance's open wait after the task instance and `hitl_detail` rows, so the task instance comes before its wait, as in registrations and deliveries. A received wait resumes the task at once with its delivery. Any other open wait parks the task until the wait's deadline, which the server holds, instead of the timeout the worker sent. - The scheduler's `awaiting_input` timeout sweep loads and locks the open waits of each batch in one query, after the task instances. A received wait resumes its task with its delivery; any other resumes it with the timeout event `{"error": "No matching delivery arrived before <deadline>", "error_type": "timeout"}`. The sweep deletes nothing; `airflow db clean` removes waits. An open wait decides a park before a Human-in-the-loop response does, because a response stays recorded after it resumed its task and would otherwise resume a later webhook park of the same try. A park without an open wait, as every Human-in-the-loop park is, behaves as before. The first tests in this PR pin that behaviour and were written against the unchanged park code. Any exception at park time fails the task, so the lookups never raise: a received wait whose delivery record `airflow db clean` removed parks until its deadline and then times out. This is one of the four PRs of the webhook waits unit, with #75340, {{PR-27}} and {{PR-28}}. They merge together once all four are approved, so that no release can register a wait that nothing resumes. ### Performance Timed on SQLite with the Execution API test client (park) and `SchedulerJobRunner` on the test DB (sweep). Base and head ran alternately, 5 runs each, in detached temporary worktrees: base is this PR's parent and head is this PR. Each park run is the median of 300 calls after 20 warm-up calls, with the four park and control paths shuffled every round. Each sweep run is the median of 40 sweeps of a 100-TI batch after 3 warm-ups. Every timed park and sweep includes its commit. Timing used `perf_kit.repeat_and_time.timing`. The table shows the median of the five run medians. | Path | Base median | Head median | Change | Queries base -> head | Control base / head | |---|---|---|---|---|---| | Park, `PATCH /execution/task-instances/{id}/state` awaiting_input, Human-in-the-loop request (unanswered), no wait | 10.77 ms | 11.09 ms | +0.33 ms (+3.0%) | 6 -> 7 | heartbeat 8.02 / 8.02 ms (-0.1%) | | Park, no Human-in-the-loop request, no wait | 10.63 ms | 10.89 ms | +0.26 ms (+2.4%) | 6 -> 7 | heartbeat 8.02 / 8.02 ms (-0.1%) | | Park, registered open webhook wait (base ignores the wait and parks it as a plain park) | 10.73 ms | 10.79 ms | +0.06 ms (+0.6%) | 6 -> 7 | heartbeat 8.02 / 8.02 ms (-0.1%) | | `check_awaiting_input_timeouts`, 100 timed-out Human-in-the-loop TIs, no waits, no defaults | 32.14 ms | 32.76 ms | +0.62 ms (+1.9%) | 101 -> 102 | `check_trigger_timeouts` on 100 deferred TIs 5.67 / 5.49 ms (-3.1%) | Slightly slower, by one indexed SELECT on inbound_wait: about +0.3 ms per awaiting_input park (+2–3%, 6 -> 7 queries) and about +0.6 ms per 100-TI timeout sweep (+1.9%, 101 -> 102 queries, one query per batch whatever its size). Both controls stayed within 3.1%, and the cost is acceptable: the park and the sweep can only resume or time out a webhook wait if they take its lock, and that lookup is a single indexed query that does not grow with the batch. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
