Federico Sacerdoti wrote:
To continue our interesting exchange,
I can start reading the phone book at any time, folks.
On Tuesday, September 3, 2002, at 03:37 PM, Steven Wagner wrote:
I am particularly interested in laying the groundwork for that so that
we can upgrade the monitoring core here with a vanilla RPM and not
worry about hosing any future proprietary metrics.
Definitely.
In this discussion, lets assume a node has received a metric from a host
named "subetha", that describes the size of its CPU1's cache. The fully
qualified metric name is "cpu/cache/size".
The node hostnames here are 12 characters long (and the hostname
metric reports their FQDNs). I like the idea of identifying the hosts
separately using a metric transmitted sparingly instead of tacking it
on to every metric name the host transmits... although I definitely
think it's the right way to go as far as naming schemes go
(debug/metadata display), I believe the hostname should be decoupled
from the transmitted metric.
So the host name is actually implied. You don't need to include it in
any metric, since the source address of the XDR packet itself identifies
the originating host. The transmitted metric name only needs to locate
the metric in the host's tree hierarchy, like "cpu/cache/size".
Maintaining a separate hash (of (strlen(metricname) + 1 +
sizeof(uint32_t)) * num_of_metrics) bytes in size) that has XDR metric
names as keys and an internal metric index as values doesn't strike me
as being too icky from a memory/performance footprint either.
I agree that adding the hash is not difficult or slow. It would fit
right into our methodology of processing metrics. My problem is with
failure resistance. What do we do if we just crashed and don't know
about any branches?
The logic is already in the monitoring core to retransmit all metrics if a
node's "gmond_started" metric timestamp changes compared to the last time
in its metric hash. So, the primary (oldest) node takes it upon itself to
retransmit the tree.
I suggest an extension to this - as I said a couple ranty e-mails back, I
think that monitoring cores should wait before transmitting any metric data
beyond "gmond_started", "gmond_version", "hostname" (maybe) and "heartbeat"
... say, ten or fifteen seconds. This should provide ample time to build
up the metric tree, even if there's 85 metrics in there.
And even if there is a failure, it's only on one node. There's still n-1
nodes out there that have an accurate picture of the cluster.
At some point a node is going to need each branch that contains a
metric to be described to it in detail.
Wouldn't it be easier if the metric name self-described its own
branches? Imagine that each metric name contains an absolute path
(filesystem analogy). Then the branches would not have to be described
in detail, they could be automatically created as necessary. The real
problem in my mind is shared fate. If the "branch-description" message
is separate from the "metric" message, then we can potentially receive
one without the other. If the branch description is included in the
metric packet (via a fully qualified name), then this is not a problem.
See above. I also proposed (about four emails ago) a DNS-like metric
resolver function that allows a libganglia-using client to submit a request
for description of a metric ... with answers being provided by the oldest
or second-oldest node.
Anyway, let's say that doesn't work, or your six-fig Cisco monkey shoved a
banana in a switch somewhere and the "create-branch" message arrives after
the metric itself.
At this point we have two options:
* Discard the metric data, process the create-branch data, wait for the
next metric transmission. Straightforward but it means a hole in the data
for up to t_max and that'd be a bummer if it's one of those 15-minute metrics.
* Guess at adding the metric data based on the payload type of the XDR.
If you win and we have a string in there at least naming the actual metric,
then we sock it into an "uncategorized" branch and query/wait for the
branch data. After the create-branch data is received, we update the
lookup hash and the metric hash to move the guessed metric into its
rightful place. This is quite a bit more complicated, obviously. And we
can't report this metric until its rightful place is secured.
A third option springs to mind, which is really a more extreme version of
the second option:
* Freak out and send a request for the entire create_tree to be
retransmitted (or, if the freaking node is the eldest, request the
secondary node or the *new metric's originating node* to transmit its
entire metric tree). The idea being, if *one* metric isn't in the tree,
you can't be sure how many other branches are missing in the tree, so ask
the most authoritative host available.
I'm pretty sure at least half of the above is not insane.
I would further like to point out that the minimum packet size in the
Ethernet protocol is 64 bytes. As most metrics are numeric types that
require only 4 bytes or so, most of the packet will be empty. We have
plenty of room available for a fully qualified name, and that space will
otherwise be wasted.
It's the idea of having the XDR metric key being of a known, fixed length
that I like.
Have we ever investigated the idea of packing metrics into a struct or
array of some kind (possibly on a branch basis) and transmitting them at
the same time? If we're starting to consider logical metric grouping in
storage and data output, might as well transmit them, as well. Besides,
atomic data sets like "cpu/idle_pct" and "cpu/user_pct" should probably be
collected *AND* transmitted as a set, don't you think? :)
If we start packing metrics like this, it might pave the way for
transmitting tabular data (such as "active processes 'ps' data", which I've
found is TOTALLY portable across our currently supported platforms, or
"individual CPU usage", "Disk partition status", "running job data for this
node according to some proprietary batch scheduler" etc. ...), which is
something I'd like to see.
Anyway...
