On Nov 27, 1:24 pm, Jason Judge <[email protected]> wrote: > Supposing I have a central server with a queue that handling logging > of events, and a number of satellite servers that generate events > which need logging, then an obvious way of handling those events will > be for the satellite servers to push the events to the central queue.
We used to use beanstalk for this (because it was "in our toolkit" anyway). Applications logged to a local beanstalk instance, each server had a program running that'd push them to the central one and then a program there would fetch the events and process them. When things worked, everything was fine, but it'd fall down pretty hard whenever there were problems with the central processing or something else. (Mostly just in the resource consumption when the queue gets large). We changed to using 'scribe' - https://github.com/facebook/scribe - which was made for exactly this use case and have very much been a "set it and forget it" solution for us. One nice thing is that it makes it easy to "route" the logs from various applications in different ways and to multiple final recipients. For example write them to a file system or HDFS, to a program that analyzes and maybe to a program that gives us HTTP access to recent log data. (Etc etc). Ask -- You received this message because you are subscribed to the Google Groups "beanstalk-talk" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
