Some of StatusNet's awesomer features like the XMPP and Twitter bridges
require running background daemons to watch event queues or keep
connections to XMPP servers open.
Alas, this just isn't going to scale to the future StatusNet 1.0 world
where we're going to be running thousands of instances for our hosted
services. We see a lot of problems with memory leaks and instability in
those daemons with even just a few live sites now...
I've worked out what I think is a feasible architecture for a more
scalable queue/daemon system for big sites to use; details and some
diagrams I whipped up to help me keep my head straight are up on the wiki:
http://status.net/wiki/Daemon_redesign
I'm planning to use a single lightweight master daemon which will
maintain long-running connections to the ActiveMQ queue and XMPP
daemons. Actual event handling will still be done at the PHP/StatusNet
level, but now as short-running processes which will handle a single
event and exit.
This reduces the surface area for memory leaks and other oddities we
encounter in long-running PHP scripts, and most importantly means we
only have to run as many processes are there actually are events being
handled!
Please feel free to give some feedback before I jump into implementation. :)
-- brion vibber (brion @ status.net)
Senior Software Architect, StatusNet
San Francisco
_______________________________________________
StatusNet-dev mailing list
StatusNet-dev@lists.status.net
http://lists.status.net/mailman/listinfo/statusnet-dev
