khushal via Mailman-Developers writes: > i'm thinking of a pub-sub layer inside MM3 core using a outbox + > worker design,
My apologies, I should have removed that task. It was done last summer, but the contributor changed career paths midsummer and hasn't finished the integration. If you really, really want to do this, we can discuss if you can do more than the previous project, but it would be a lot easier if you do another task. Comments since you went to this much trouble: > core already emits internal Zope events whenever the state changes, > we can subcribe to those existing events and shift them into > normalized callback events, instead of adding newer trigger points. I don't understand which existing events you have in mind, nor am I sure all of the interesting events necessarily have corresponding Zope events. > each event will be written to an outbox table in the same > transaction as the business logic changes therefore adding > resiliency, if a process crashes. > > a new CallbackRunner in the core, that shall periodically sweep the > outbox events, and resolve the matching webhook subscriptions(from > subscribers) and POST the signed payloads(HMAC) to > subscribers(Postorius/HyperKitty etc). Here, the message delivery > will be at least once, with retry and exponential backoff and > attempt tracking, dead state. Are you reinventing the wheel? There are a number of generic pub-sub implementations already available that can be hooked into. > Note: the existing polling APIs are unchanged for compatibility > reasons > > 1. add the required database tables(subscriptions, outbox, delivery > attempts etc) > 2. start capturing small set of events like list.created, > member.subscribed, list.deleted for initial phase, then do the > same for all other events > 3. add a new CallbackRunner(worker) that reads events from the > outbox table and sends them to subscribed webhook endpoints, to > make this work, we add REST APIs endpoints to create, manage, > and test webhook subscriptions. > 4. adding security enhancements, like HMAC signing, timestamp > validations and SSRF protections > 5. the webhook payloads pushed by the CallbackRunner will have the > event type, a schema version, and a unique event_id to help > subscribers handle different types of events which come from a > single callback endpoint + perform the retries and events > deduplication > 6. testing the transaction integrity, retry behavior, idempotency, > HMAC sig verification, worker restarts. > > > Questions: > i. is the GSoC project Core only or HyperKitty/Postorius > integration necessary too? The obvious testbed for the sub side is one of the existing web UI applications. Otherwise you'd have to create a sub client specifically to demonstrate the integrated functionality. > ii. which event types should i focus on first (list.created, > list.deleted, member.subscribed, member.unsubscribed)? The point of pub-sub is to update displays so the user doesn't have to. List creation/deletion currently can only be done by the superuser, and the feedback is immediate. It's not clear to me when the subscription events would be of interest, although I can imagine an admin looking at the membership list and not realizing they need to refresh. The events of real interest are new messages in HyperKitty, and new moderation events in Postorius (both message arrival and resolution by other moderators). I would do the moderation events, as these are almost always simply "let's refresh the whole page", where realtime refreshing of HyperKitty is going to require a bit of thought (and possibly extension of the pages -- eg, suppose you're looking at a post, and a new message comes in: how should that be notified?) > iii. callback subscription management be admin-only or scoped to > list owners too? For HyperKitty, subscribers will want to know if new messages arrive to this thread, this list, or possibly any list (pretty wide scope here). For Postorius, the most interesting cases are arrival and displosition of held messages. > iv. for the event delivery, is at least once accepted or the events > should have ordering gurantees? That's going to depend on the event. -- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan _______________________________________________ Mailman-Developers mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/mailman-developers.python.org/ Mailman FAQ: https://wiki.list.org/x/AgA3 Security Policy: https://wiki.list.org/x/QIA9
