So, as Steven and others have mentioned, we have a problem with ganglia
metrics. Metrics currently lie in a flat namespace, with no hierarchical
groupings. I have talked with Matt and Mason (a ganglia developer and my
boss) about this problem, and would like to state and define some of our
ideas.
Through phone discussions with Matt, I've come to like the idea of a
<BRANCH name="CPU"> type tag. This would enable hierarchical namespaces.
An important question is why would we want these? There are several
reasons.
Elements in a branch need to share some common semantics. One example of
a shared semantic is all metrics under the CPU branch could get
displayed in the same section of a webpage. Therefore, if a new cpu
metric was introduced, say "L1 cache size", the web page would
automatically put it in the correct section.
Another advantage of hierarchies comes from object-oriented design.
Attributes in the Branch tag, such as DMAX (when metrics get deleted),
become the default for all metrics below it. These can be overrided by
the individual metrics, analogous to overriding baseclass methods in an
OO class tree. This gives an easy way to assign attribute values to a
group of metrics.
A third advantage is cleaner namespaces. You can call 'cpu_num' simply
'num'. Similar naming simplifications are possible for the other
metrics. The most significant advantage is that we only have to worry
about name collisions among siblings in the tree. There can be a 'num'
metric in another branch (for example, the 'num' of network interfaces).
So how do we name metrics in the XDR packet if we adopt a metric
hierarchy? This is a difficult problem, since we want to allow new
metrics to appear at any time. Imagine an XDR packet comes in. We need
to identify the metric, and update its value in our hash tables.
We know which host sent the packet from its source address, so we look
at our hash table for that host. Now how do we know which branch it goes
in? Mason suggested that each XDR metric packet contains the name of its
most immediate ancestor in the metric tree. The thinking is that since
order does not matter among siblings, we could search the tree until we
found the correct branch, and put the metric under it. This would
require the XDR packet to carry a minimal amount of information, and the
only restriction would be each branch must have a unique name.
To update the hash table effectively, each sibling must have a unique
ID. If we get a XDR packet for metric "memory->free", we find the
"memory" branch, and put a "free" metric under it. If at a later time we
receive another packet for the same metric, we look for the "free"
metric, and update its value. I suggest that like user-defined metrics,
we use a string as the hash key. In this case, the string "memory" would
key the branch hash table, and "free" would key the metric.
The problem is that the "memory" branch may itself lie along several
branches. When a host sends us a "memory->free" metric, do we even know
about the "memory" branch? Let's say we require a host to send a
"create-branch" message before sending metrics for that branch. Well
what if some nodes miss the create message, or worse yet, they are just
recovering from a crash? They would not know about the "memory" branch
at all.
I believe the answer is that new nodes get their branch hierarchy all at
once from the oldest gmond in the cluster (which I will call the eldest
node). Matt has been talking about this for some time, as it will solve
some other problems as well. If we get an XDR metric packet that
specifies an unknown branch, we discard it. However, we realize that we
must have missed something, so we query the eldest node for their metric
hierarchy. If we can't find the eldest node, we query the second eldest,
etc. We also query the second eldest if we didn't learn anything new
from the eldest himself. (This solves the problem of the eldest node
having incomplete information).
The assumption is that the eldest node has been listening to all the
"create-branch" messages, and has a complete metric tree.
This email message is getting too long, but I would go on about how we
could use the idea of database indexes to quickly locate any branch in
the tree.
I hope I have been relatively clear about these ideas. I realize this
problem is pretty dense, and this solution is in its infancy. But the
point I would like to drive home is that a naming hierarchy is helpful
for specific reasons, and that its efficient implementation is possible
in the ganglia framework.
Sincerely,
Federico
Rocks Cluster Group, Camp X-Ray, SDSC, San Diego
GPG Fingerprint: 3C5E 47E7 BDF8 C14E ED92 92BB BA86 B2E6 0390 8845
