Federico Sacerdoti wrote:
I like your comments. Here are some counter-questions :)
I like your counter-questions. Here are some counter-counter-questions,
interspersed with wisecracks like this one.
On Friday, August 30, 2002, at 02:34 PM, Steven Wagner wrote:
It seems to me this would also make the "DSO-ification" of the
monitoring core a smoother process, not to mention a cleaner one from
the standpoint of those developing the DSO's. :)
Good point.
I am particularly interested in laying the groundwork for that so that we
can upgrade the monitoring core here with a vanilla RPM and not worry about
hosing any future proprietary metrics.
So I am thinking that sending the fully-qualified metric name (as shown
above) is a better idea now - it handles failures more effectively. When
a node comes up it would receive metrics that look like
"host1/cpu/cache/size" (fully-qualified with all the metric's ancestors)
instead of "cache/size" (relative as I had suggested previously). This
fits in with Steven's idea of hosts being authoritative for branches
they created - each metric specifies its branches explicitly. It also
reduces reliance on an elder node for the branch hierarchy.
I side with optimizing the packet size if the choice is between having
small packet sizes and small/easy-to-follow data structures in the
monitoring core. My rationale for this is that the number of metrics
probably won't scale beyond fifty or a hundred values, if that. I am
somewhat more sensitive to inflating the packet size, which will cause a
geometric increase in network traffic as the number of metrics and/or nodes
grow.
The node hostnames here are 12 characters long (and the hostname metric
reports their FQDNs). I like the idea of identifying the hosts separately
using a metric transmitted sparingly instead of tacking it on to every
metric name the host transmits... although I definitely think it's the
right way to go as far as naming schemes go (debug/metadata display), I
believe the hostname should be decoupled from the transmitted metric.
And I'm making these points not necessarily because I adore the sublime
beauty of a 128-byte XDR packet versus a 192-byte XDR packet, but... well,
actually, I guess that's what I'm doing, actually. :) Part of the
elegance of the program is its compactness and simplicity on the network
level, after all.
Maintaining a separate hash (of (strlen(metricname) + 1 + sizeof(uint32_t))
* num_of_metrics) bytes in size) that has XDR metric names as keys and an
internal metric index as values doesn't strike me as being too icky from a
memory/performance footprint either. Plus you are only ever doing two hash
table lookups to reach any metric, as opposed to walking an n-branch metric
tree (which, if you get right down to it, is probably at least three levels
deep - hostname, metric-main-type, actual-metric-value (i.e. host1.cpu.num).
All of my comments of course focus on enhancing the flexibility of metrics
from the multicast side, not the formatting/output of the XML dump.
There's no explicit provision mentioned for handling the proposed metric
hierarchy.
This way a node can easily create branches as needed for any metric it
receives.
About the "hash for storing fully qualified metric names (FQMN :)". How
would we populate such a hash? At some level, the metric must specify
its fully-qualified name, so we know where to put it. A hash value is no
good if we don't already have the name stored. How would you handle new
metrics? I think we could runlength-encode the name strings to save
space if we need to, but having each metric carry its full name seems
clearer to me.
At some point a node is going to need each branch that contains a metric to
be described to it in detail. When this description is finished being
parsed and added to the internal metric hash, the monitoring core makes a
note of the XDR metric key and adds that to the XDR->internal hash along
with a pointer to the newly-created entry. This second hash doesn't even
handle hierarchy - but parent and child(ren) fields should be present in
the entry referenced by the pointer in the lookup hash.
This way, every metric-related message each monitoring core receives will
"resolve" to something, and from that you should be able to compute its FQMN.
I imagine a hash_find(node, "cpu", "cache") function that takes a
variable number of arguments to locate the hash table to insert a given
metric in (the metric here: host1/cpu/cache/size). The 'node' argument
specifies the root of the metric tree - the node hash table for host1.
Note each branch would get it's own hash table so that hash_foreach()
will work correctly and printing the XML will be easy.
To make this work, we simply add a 'hash_t *branch' member to the
metric_data_t structure. If branch==NULL then we are a leaf (actual
metric), else this is a branch that points to another hash table. I can
visualize the XML output code now...
Sounds like we're actually talking about the same thing. I'm just
advocating a different method on the XDR side.
But yeah, the XML output code will not be too tough. :)
Dense, yes, but the area of metrics is just about the only one in the
Ganglia design that *doesn't* scale well (kudos, Matt & co.). I'm
sure that we can work this out if we just keep banging those rocks
together. :)
Clever ;)
Just a little homage to the late Douglas Adams. For those of you at home
keeping score, the original line goes something like this:
"... this is the Sub-Etha News Network, broadcasting to civilized beings
throughout the galaxy. For those of you still living in caves, the secret
is to keep banging those rocks together, guys..."
Do people like the java-like dot notation for hierarchical names, like
"host1.cpu.cache.size", or the unix filesystem forward-slash notation:
"host1/cpu/cache/size". I like the slashes because its easy to tell if
you're talking about a leaf or a branch: "host1/cpu/" is clearly a
branch, while "host1.cpu." is a little harder to read. But either way
would work.
I always liked URLs. Besides, everybody knows what a URL looks like. At
least, everybody who's running gmond does, I hope ...
gmond://host[.cluster?]/cpu/1/idle_percentage
[etc.]
I'm psyched about this change, and I am ready to dive in right after the
2.5.0 release.
Me too, I am itching to write up a platform-agnostic /proc-walking algo
I've been kicking around for a few days but I don't want to even get into
it until 2.5.0 gets out the door.
