On Wed, Nov 25, 2009 at 4:20 PM, Daniel Pocock <dan...@pocock.com.au> wrote: > One problem I've been wondering about recently is the scalability of > gmetad/rrdtool.
[cut] > In a particularly large organisation, moving around the RRD files as > clusters grow could become quite a chore. Is anyone putting their RRD > files on shared storage and/or making other arrangements to load balance > between multiple gmetad servers, either for efficiency or fault tolerance? We do. We run 8 gmetad servers, 2 in each colo x 3 colos + 2 centrals and rrds are stored in ram disk on each node. Nodes are setup with unicast and data is sent to both heads in the same colo for fault tolerance/redundancy. This is all good until you have a gmetad failure or need to perform maintenance on one of the nodes because at that point as data stops flowing in you will have to rsync back once you're done from the other head and it doesn't matter how you do it (live rsync or stop the other head during the sync process) you will lose data. That said it could be easily argued that you have no guarantee that both heads have the same data to start with because messages are udp and there's no guarantee either node will have not lost some data the other hasn't. Of course there is a noticeable difference between a random message loss and a say 15 window blackout during maintenance, but then if your partitions are small enough a live rsync could possibly incur in a small enough loss... it really depends. As to share storage we haven't tried but my personal experience is that given how a local filesystem can't manage that many small writes and seeks using any kind of remote FS isn't going to work. I see two possible solutions: 1. client caching 2. built-in sync feature In 1. gmond would cache data locally if it could not contact the remote end. This imho is the best solution because it helps not only with head failures and maintenance, but possibly addresses a whole bunch of other failure modes too. 2. instead would make gmetad aware of when it got data last and be able to ask another gmetad for its missing data and keep fetching until the delta (data loss) is small enough (user configured) that it can again receive data from clients. This is probably harder to implement and still would not guarantee no data loss, but I don't think that's a goal. The interesting property of this approach is that it'd open the door for realtime merge of data from multiple gmetads so that as long that at least one node has received a message a client wouldn't ever see a gap effectively providing no data loss. I'm toying with this solution in a personal non-ganglia related project as it's applicable to anything with data stored in rrd over multiple locations. thanks -- "Behind every great man there's a great backpack" - B. ------------------------------------------------------------------------------ Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev _______________________________________________ Ganglia-developers mailing list Ganglia-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ganglia-developers
