Hey everyone, recently we have gotten some really great community contributions, one of which adds support for the AMQP messaging protocol to carbon. I have started migrating some of my systems to using this and I think it is a great fit for graphite. If you aren't familiar with it already I highly recommend reading http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ for a brief introduction.
One area I think AMQP can be especially useful is with a clustered graphite installation. Most of you are probably not familiar with Graphite's clustering capabilities because I have not documented it at all (sorry, hope to change that soon). But essentially, you just setup N different independent graphite servers and then configure the webapps to know about one another. They then share data in a manner transparent to the user, making each server look like one big graphite installation instead of several independent ones. The tricky part is partitioning your metrics across the servers. Thus far I've solved this problem with a daemon called carbon-relay.py, which basically acts as an application-level load balancer for your metrics. You are supposed to send all of your data to carbon-relay, who then looks up some rules you've given it to decide which carbon-cache(s) to relay each metric on to. With AMQP, there seems to be a much simpler way to solve this problem. Topic exchanges use a dot-delimited naming structure that supports pattern matching very similar to graphite's. Basically, you could just publish messages containing a value and a timestamp and use the metric name as the routing key. Then each carbon-cache can consume from the exchange with one or more binding patterns. For the simplest case of having only one server the binding pattern would simply be "#" (which is the same as using a fanout exchange). For more complex cases you could control what data goes to what server by means of configuring each carbon-cache's binding patterns. This would effectively replace carbon-relay, and I believe, solve the problem in a more robust way. This is a bit different than they way it currently works in trunk so I wanted to run it by everyone and see what your thoughts are, especially if you are already using AMQP or carbon-relay. I am currently in the process of testing this configuration and if it works well I will try it in my production system. If that goes well, I would like to include the new behavior in this month's release. So please send me any comments, questions, concerns, etc. If you don't plan on using AMQP, that's fine too, the old interface is not going away. -Chris
_______________________________________________ Mailing list: https://launchpad.net/~graphite-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~graphite-dev More help : https://help.launchpad.net/ListHelp
