--- On Fri, 2/12/10, Chris Davis <[email protected]> wrote:
> From: Chris Davis <[email protected]> > Subject: [Graphite-dev] request for comments - using amqp and replacing > carbon-relay > To: "graphite-dev" <[email protected]> > Date: Friday, February 12, 2010, 2:00 PM > Hey everyone, recently we have gotten some > really great community contributions, one of which adds > support for the AMQP messaging protocol to carbon. I have > started migrating some of my systems to using this and I > think it is a great fit for graphite. If you aren't > familiar with it already I highly recommend > reading http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ for > a brief introduction. > > One area I think AMQP can be especially useful > is with a clustered graphite installation. Most of you are > probably not familiar with Graphite's clustering > capabilities because I have not documented it at all (sorry, > hope to change that soon). But essentially, you just setup N > different independent graphite servers and then configure > the webapps to know about one another. They then share data > in a manner transparent to the user, making each server look > like one big graphite installation instead of several > independent ones. The tricky part is partitioning your > metrics across the servers. Thus far I've solved this > problem with a daemon called carbon-relay.py, which > basically acts as an application-level load balancer for > your metrics. You are supposed to send all of your data to > carbon-relay, who then looks up some rules you've given > it to decide which carbon-cache(s) to relay each metric on > to. > > With AMQP, there seems to be a much simpler way > to solve this problem. Topic exchanges use a dot-delimited > naming structure that supports pattern matching very similar > to graphite's. Basically, you could just publish > messages containing a value and a timestamp and use the > metric name as the routing key. Then each carbon-cache can > consume from the exchange with one or more binding > patterns. For the simplest case of having only one server > the binding pattern would simply be "#" (which is > the same as using a fanout exchange). For more complex cases > you could control what data goes to what server by means of > configuring each carbon-cache's binding patterns. This > would effectively replace carbon-relay, and I believe, solve > the problem in a more robust way. We use a simple fanout exchange for our metrics with 1 carbon-cache server reading from that exchange. Currently we're processing 75k+/minute of metrics using 1 pair of rabbitmq servers in 1 colo, receiving for all colos (4). This works fairly well for us. The only problem we had was that carbon-cache was writing so hard to the disk that it slowed things down. The solution was to set MAX_UPDATES_PER_SECOND from 1000 to 50. This reduced the disk utilization from 100% to just around 30%. Works great. -allan > This is a bit different than they way it > currently works in trunk so I wanted to run it by everyone > and see what your thoughts are, especially if you are > already using AMQP or carbon-relay. I am currently in the > process of testing this configuration and if it works well I > will try it in my production system. If that goes well, I > would like to include the new behavior in this month's > release. So please send me any comments, questions, > concerns, etc. > > If you don't plan on using AMQP, that's > fine too, the old interface is not going away. > -Chris > > -----Inline Attachment Follows----- > > _______________________________________________ > Mailing list: https://launchpad.net/~graphite-dev > Post to : [email protected] > Unsubscribe : https://launchpad.net/~graphite-dev > More help : https://help.launchpad.net/ListHelp > _______________________________________________ Mailing list: https://launchpad.net/~graphite-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~graphite-dev More help : https://help.launchpad.net/ListHelp
