Forgot to include the mailing list... 2010/1/19 Brion Vibber <br...@status.net>
> On 01/19/2010 09:04 AM, Federico Marani wrote: > > Hi Brion, > > You did some impressive work on statusnet back-end... well-done > > > Thanks! > > > I have a couple of questions i'd like to ask you > ex. you were thinking of writing the master daemon in python. What made you > change opinion? > Is the current stripped down php master process more kind with memory? > > > It's easier to work directly within the PHP framework to avoid duplicating > effort on things like configuration... also we ended up giving up the idea > of launching short-running processes because PHP engine startup time + PHP > script compilation for each event was too expensive. Staying within a PHP > process means we minimize the amount of startup we have to do for each event > to just reloading the StatusNet configuration. > Seems the best thing to do, spawning PHP interpreters for every message is really expensive in terms of CPU > > Some techniques i've seen used around involved using an "angel" process > which is activated every X minutes by cron and checks memory consumption of > the daemon process. If not within certain parameters, it restarts it. Is > this approach not viable for statusnet? > > > Currently when running with multiple child processes, the parent process > monitors and restarts them if they exit. Individual child processes will > exit gracefully if they pass a soft memory limit, though a sudden spike in > theory could bump things all the way to php.ini's hard memory_limit between > soft checks. (Because the parent process doesn't do much > The way we've done this is similar... Master process periodically sends a signal (SIGUSR2) to each child, this child writes to a stats file with all memory details in it, and the master reads the file. If memory consumption read from the file is too high or file is not present (means the child is stuck) then respawn it. > We don't yet have a good solution for _stuck_ threads, for instance if the > queue server stops delivering, but I'm considering some possibilities. > > Eh, this is imho difficult ground.. Messaging theory says you should use control messages. If you periodically inject control messages in the queues that trigger some logging at child level you would be able to close the loop and see if there's anything wrong with the process. http://www.eaipatterns.com/TestMessage.html Don't want to promote any book, but you may be interested in Enterprise integration patterns... it's the bible of messaging :) > > Also noticed that you mention ActiveMQ in your post.. i thought Evan > decided to use RabbitMQ as queue server for identi.ca. What are you > currently using? > > > We're currently using ActiveMQ, though in theory we should be able to work > with anything providing a STOMP interface. RabbitMQ is on the list of things > to try but it's falling down our list a bit after our negative experiences > with ejabberd, which is also erlang-based... > > -- brion vibber (brion @ status.net)de > > > > > Regards, > Federico > > 2010/1/13 Brion Vibber <br...@status.net> > >> Woohoo! After a couple months off and on adjusting the architecture to >> something that seems to meet our needs, I've merged my refactoring of >> StatusNet's background queue processing to 0.9.x. >> >> Some of my design notes are up on the wiki, with a couple updates based on >> tweaks I made from my original plans: >> http://status.net/wiki/Daemon_redesign >> >> Key items from the commit summary: >> >> Major refactoring of queue handlers to support running multiple sites in >> one daemon. >> >> Key changes: >> * Initialization code moved from common.php to StatusNet class; >> can now switch configurations during runtime. >> * As a consequence, configuration files must now be idempotent... >> Be careful with constant, function or class definitions. >> * Control structure for daemons/QueueManager/QueueHandler has been >> refactored; >> the run loop is now managed by IoMaster run via scripts/queuedaemon.php >> IoManager subclasses are woken to handle socket input or polling, and may >> cover multiple sites. >> * Plugins can implement notice queue handlers more easily by registering a >> QueueHandler class; no more need to add a daemon. >> >> The new QueueDaemon runs from scripts/queuedaemon.php: >> >> * This replaces most of the old *handler.php scripts; they've been >> refactored >> to the bare handler classes. >> * Spawns multiple child processes to spread load; defaults to CPU count on >> Linux and Mac OS X systems, or override with --threads=N >> * When multithreaded, child processes are automatically respawned on >> failure. >> * Threads gracefully shut down and restart when passing a soft memory >> limit >> (defaults to 90% of memory_limit), limiting damage from memory leaks. >> * Support for UDP-based monitoring: http://www.gitorious.org/snqmon >> >> Rough control flow diagram: >> QueueDaemon -> IoMaster -> IoManager >> QueueManager [listen or poll] -> QueueHandler >> XmppManager [ping& keepalive] >> XmppConfirmManager [poll updates] >> >> Todo: >> >> * Respawning features not currently available running single-threaded. >> * When running single-site, configuration changes aren't picked up. >> * New sites or config changes affecting queue subscriptions are not yet >> handled without a daemon restart. >> * SNMP monitoring output to integrate with general tools (nagios, ganglia) >> * Convert XMPP confirmation message sends to use stomp queue instead of >> polling >> * Convert xmppdaemon.php to IoManager? >> * Convert Twitter status, friends import polling daemons to IoManager >> * Clean up some error reporting and failure modes >> * May need to adjust queue priorities for best perf in backlog/flood cases >> >> Detailed code history available in my daemon-work branch: >> http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work<http://www.gitorious.org/%7Ebrion/statusnet/brion-fixes/commits/daemon-work> >> >> -- brion vibber (brion @ status.net) >> _______________________________________________ >> StatusNet-dev mailing list >> StatusNet-dev@lists.status.net >> http://lists.status.net/mailman/listinfo/statusnet-dev >> > > > > -- > --------------- FM > Flow with whatever is happening and let your mind be free. Stay centered by > accepting whatever you are doing. This is the ultimate. > - 庄子 (Chuang Tzu) > > > -- --------------- FM Flow with whatever is happening and let your mind be free. Stay centered by accepting whatever you are doing. This is the ultimate. - 庄子 (Chuang Tzu)
_______________________________________________ StatusNet-dev mailing list StatusNet-dev@lists.status.net http://lists.status.net/mailman/listinfo/statusnet-dev
