i'm thinking of a pub-sub layer inside MM3 core using a outbox + worker design, core already emits internal Zope events whenever the state changes, we can subcribe to those existing events and shift them into normalized callback events, instead of adding newer trigger points.
each event will be written to an outbox table in the same transaction as the business logic changes therefore adding resiliency, if a process crashes. a new CallbackRunner in the core, that shall periodically sweep the outbox events, and resolve the matching webhook subscriptions(from subscribers) and POST the signed payloads(HMAC) to subscribers(Postorius/HyperKitty etc). Here, the message delivery will be at least once, with retry and exponential backoff and attempt tracking, dead state. Note: the existing polling APIs are unchanged for compatibility reasons 1. add the required database tables(subscriptions, outbox, delivery attempts etc) 2. start capturing small set of events like list.created, member.subscribed, list.deleted for initial phase, then do the same for all other events 3. add a new CallbackRunner(worker) that reads events from the outbox table and sends them to subscribed webhook endpoints, to make this work, we add REST APIs endpoints to create, manage, and test webhook subscriptions. 4. adding security enhancements, like HMAC signing, timestamp validations and SSRF protections 5. the webhook payloads pushed by the CallbackRunner will have the event type, a schema version, and a unique event_id to help subscribers handle different types of events which come from a single callback endpoint + perform the retries and events deduplication 6. testing the transaction integrity, retry behavior, idempotency, HMAC sig verification, worker restarts. Questions: i. is the GSoC project Core only or HyperKitty/Postorius integration necessary too? ii. which event types should i focus on first (list.created, list.deleted, member.subscribed, member.unsubscribed)? iii. callback subscription management be admin-only or scoped to list owners too? iii. for the event delivery, is atleast once accepted or the events should have ordering gurantees? _______________________________________________ Mailman-Developers mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
