On Wed, Nov 25, 2009 at 4:20 PM, Daniel Pocock <dan...@pocock.com.au> wrote:
> One problem I've been wondering about recently is the scalability of
> gmetad/rrdtool.


> In a particularly large organisation, moving around the RRD files as
> clusters grow could become quite a chore.  Is anyone putting their RRD
> files on shared storage and/or making other arrangements to load balance
> between multiple gmetad servers, either for efficiency or fault tolerance?

We do. We run 8 gmetad servers, 2 in each colo x 3 colos + 2 centrals
and rrds are stored in ram disk on each node. Nodes are setup with
unicast and data is sent to both heads in the same colo for fault
tolerance/redundancy. This is all good until you have a gmetad failure
or need to perform maintenance on one of the nodes because at that
point as data stops flowing in you will have to rsync back once you're
done from the other head and it doesn't matter how you do it (live
rsync or stop the other head during the sync process) you will lose
data. That said it could be easily argued that you have no guarantee
that both heads have the same data to start with because messages are
udp and there's no guarantee either node will have not lost some data
the other hasn't. Of course there is a noticeable difference between a
random message loss and a say 15 window blackout during maintenance,
but then if your partitions are small enough a live rsync could
possibly incur in a small enough loss... it really depends.

As to share storage we haven't tried but my personal experience is
that given how a local filesystem can't manage that many small writes
and seeks using any kind of remote FS isn't going to work.

I see two possible solutions:
1. client caching
2. built-in sync feature

In 1. gmond would cache data locally if it could not contact the
remote end. This imho is the best solution because it helps not only
with head failures and maintenance, but possibly addresses a whole
bunch of other failure modes too.
2. instead would make gmetad aware of when it got data last and be
able to ask another gmetad for its missing data and keep fetching
until the delta (data loss) is small enough (user configured) that it
can again receive data from clients. This is probably harder to
implement and still would not guarantee no data loss, but I don't
think that's a goal. The interesting property of this approach is that
it'd open the door for realtime merge of data from multiple gmetads so
that as long that at least one node has received a message a client
wouldn't ever see a gap effectively providing no data loss. I'm toying
with this solution in a personal non-ganglia related project as it's
applicable to anything with data stored in rrd over multiple


"Behind every great man there's a great backpack" - B.

Join us December 9, 2009 for the Red Hat Virtual Experience,
a free event focused on virtualization and cloud computing. 
Attend in-depth sessions from your desk. Your couch. Anywhere.
Ganglia-developers mailing list

Reply via email to