I'll try to answer all of these.
On Thursday, December 5, 2002, at 09:23 PM, [EMAIL PROTECTED]
wrote:
Frederico & Steven, I really apprceciate your thoughts about the
Ganglia front-end architecture.
I have one more question. Is gmetad robust? If I've got this right,
gmond maintains only the lastest metric values received for the
cluster. If all of the gmetads go down, aren't all the values during
that time period lost forever?
If a gmetad goes down, it stops recording metric value history. When it
comes up, this will show as a gap in the graphs.
If at least one gmetad stays up, then when others come up and pull the
xml description from the gmetad that survived, will they merge all of
the values missing from their own rrd?
This does not happen. Gmetad's are not robust the way gmonds are. They
do not attempt to "bring newcomers up to date" as gmond does. This has
to do with security: how do we know you deserve the old data? With
gmond, the security is implicit in being part of the multicast channel.
Also, the rrds are very timestamp sensitive. Even if we did give a
recovering gmetad data for its gaps, small clock skews would make the
graphs look terrible. Not that this isn't something we could overcome
with careful engineering, however. Our assumption is that gmetads are
running on dedicated monitoring hardware that is hand administered and
possibly redundant. If a gmetad goes down, an operator can copy the rrd
files from a surviving gmetad to fill in the gaps. However in practice,
gaps are not that big of a deal, and don't degrade performance or
correctness like a gmond failure does.
If so, how do you know at any given time whether a particular gmetad
is up to date?
A gmetad always makes graphs based on fresh data. If it is drawing
anything on the left side of a graph, it is up to date. Otherwise it is
dead. If there are gaps in the graph, it means the gmetad was down for
that period of history. I may be misunderstanding your question here.
What advice would you give in terms of the gmetad to gmond ratio? For
maximum redunancy, should every node run both gmond and gmetad?
Since keeping metric history with RRD databases is computation and I/O
intensive, I would not suggest this. We keep a gmetad service running
on the frontend node of a cluster, that is one gmetad for the cluster.
Jonathan
Hope this helps,
Federico
Rocks Cluster Group, Camp X-Ray, SDSC, San Diego
GPG Fingerprint: 3C5E 47E7 BDF8 C14E ED92 92BB BA86 B2E6 0390 8845
