mlevkov opened a new pull request, #3798:
URL: https://github.com/apache/iggy/pull/3798
Implements the webhook gateway accepted in #3039. Iggy currently has no way
to receive a webhook; every provider that pushes events over HTTP needs a
separate service in front whose only job is to accept a POST and republish it.
This connector removes that hop.
## Shape
One plugin `.so` is loaded once regardless of how many `[[source]]` entries
reference it, so the listener lives in a process-global registry keyed by
listen address rather than on any single instance. The first `open()` binds the
public and admin ports; later opens validate their body limit, admin address,
management token and instance name against the running listener before joining;
the last `close()` releases both ports, which the runtime's stop-then-start
restart flow depends on. A single port can therefore serve many providers, each
routed to its own topic.
- `POST /topics/{topic_path}` — named path, one per instance, guarded by an
optional bearer token
- `POST /e/{endpoint_id}` — secret path, 128 bits in the URL itself, with
optional per-endpoint bearer or HMAC on top
Requests resolve against an `ArcSwap` route table rebuilt whole on every
control-plane change, so one atomic load yields both the endpoint's auth rules
and the destination bridge. HMAC is verified over the raw body in constant
time. Revoked endpoints answer 404 alongside paths that never existed, so a
leaked URL cannot be used to confirm it was once live.
Endpoints can be registered, re-keyed and revoked at runtime through a
token-guarded API on the admin listener (absent entirely when no token is
configured). Those endpoints ride the SDK's `ConnectorState`, and revocation
writes a tombstone that outranks TOML on restore, so a stale config file cannot
resurrect an endpoint an operator revoked.
## Delivery semantics
Best-effort in both directions, and the README leads with it rather than
burying it. HTTP 200 means accepted into an in-memory buffer, not durably
stored; both the loss and duplicate windows are enumerated with what mitigates
each. A full bridge answers 429 with `Retry-After` rather than blocking.
**Depends on #3795 for the full backpressure story.** The bridge is bounded
today, so 429 fires on an arrival burst the poll loop cannot keep up with. What
is missing is the coupling: until the bounded runtime forwarding channel lands,
`poll()` drains into an unbounded channel, so a slow Iggy does not propagate
back into 429. The README documents this rather than implying the chain is
complete.
State is attached only to an empty batch. The runtime saves state solely on
the success branch of the Iggy send, and an empty send always succeeds, so a
management mutation cannot be lost to an unrelated send failure.
## Deviations from the design doc, all deliberate
- **`topic_path` and `instance_name` are explicit config fields.** Only
`plugin_config` crosses the FFI, so the plugin cannot see its connector key or
its `[[streams]]` entry. Same resolution the design already accepted for the
named path.
- **The 503 "instance closing" row collapses into 404.** Deregistration is
atomic under the registry lock, so no observable window exists. 503 survives on
`GET /health` when no instance is joined.
- **Stripe and Twilio HMAC are not supported.** This validator takes a hex
digest of the body behind a fixed prefix, which covers GitHub and most generic
partner webhooks. Stripe signs `{timestamp}.{body}` behind a compound header
and Twilio signs URL plus sorted params as base64. The design's example config
showed Stripe working; it would not have. Documented with the
forward-and-verify-downstream workaround, and the shipped example uses bearer
instead.
- **`schema = "raw"` is mandatory** and now stated as such — the connector
always produces raw bodies, and a JSON encoder rejects every message.
## Testing
110 unit tests and 5 integration tests. The integration suite needs no
containers, since the connector is itself the HTTP server and the test client
is the webhook sender: it covers a signed POST reaching Iggy byte-for-byte with
headers intact, two instances sharing one listener, the register/POST/revoke
flow, a dynamic endpoint with its secret surviving a connector restart, and a
revoked *static* endpoint staying dead across a restart that re-reads the TOML
still declaring it.
No integration test for 429 under saturation: with a healthy Iggy the poll
loop drains the bridge continuously, so provoking a full bridge from outside
races the drain. Two unit tests cover it deterministically instead.
## Verification
`cargo fmt`, `cargo sort --check --no-format --workspace`, `cargo clippy
--all-features --all-targets -- -D warnings` (connector and integration),
`cargo test`, `taplo fmt --check`, `hawkeye check`, markdownlint, and the
trailing-whitespace/newline scripts all pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
