Hi all,
I want to gauge the interest in supporting inbound webhooks for Airflow.
Why?
Webhooks are how most services announce events: source control and CI
(GitHub, GitLab, Bitbucket), data tools (dbt Cloud, Fivetran, Airbyte),
ML platforms, payment and commerce systems, incident tools (PagerDuty,
Datadog) and issue trackers (Jira, Linear). Those events are often what
a Dag is waiting for, but Airflow can't receive them: senders can't
attach an Airflow API token, they sign deliveries with a shared secret,
and each has its own payload format. So we poll with sensors, which hold
capacity and react late, or run a relay service in front of Airflow.
AssetWatcher covers pull-based queues; there's no push equivalent.
Webhooks would let a Dag start the moment a dbt Cloud job finishes, a
service is deployed, a PagerDuty incident opens or a vendor's export is
ready.
The idea:
Turn incoming webhooks into asset events that trigger your Airflow Dags!
Each verified delivery is recorded as an asset event named after what
happened, and Dags subscribe by scheduling on that name. When a pull
request in my-org/web gets the deploy-preview label, this Dag runs, with
the payload in its triggering_asset_events:
@dag(schedule=Asset("github/my-org/web/pull_request/labeled/deploy-preview"))
Broader names work too: github/my-org/*/push/main fires on a push to
main in any repository of the org. Senders don't need to know Dag IDs,
one delivery can start several Dags, and the scheduler creates the runs
as for any asset event.
Try it out:
I've built this as a provider for Airflow 3.1.1+, with GitHub, GitLab
and Jira built in, plus a generic option for any service that signs its
body with HMAC-SHA256 or sends a secret header:
https://github.com/dheerajturaga/airflow-provider-inbound-webhooks
https://pypi.org/project/airflow-provider-inbound-webhooks/
The README has a quick start that needs no third-party account, and
docs/design.md covers what could move to core.
If there's interest, I'm happy to draft an AIP. Thoughts welcome!
Thanks,
Dheeraj
