On 10/7/05, Rick Mohr <[EMAIL PROTECTED]> wrote: > > > On Thu, 6 Oct 2005, Peter Bodik wrote: > > > The user-defined metrics are less-efficient, right? So having 1000s (from > > each host) could cause some problems ... I'd like to know what kind of > > problems we can expect and what do to about them. > > Short answer: You are most likely correct > > Long answer: We have been playing around with adding extra metrics for our > clusters at OSC. Our situation is slightly different: we have a large > number of nodes with a small number of extra metrics added via gmetric > (about 5 or so per host). But in total, this translates into about 1500 > extra metrics. > > For our initial testing, we kept things simple and had an hourly cron job > that reported all these metrics. The metrics were set to expire (and > hence disappear from Ganglia) in a little over an hour. I soon noticed > that there were always a few nodes that were missing some metrics. But > when I checked the metrics listed by the local gmond processes on these > nodes. The values were always there. > > After several different tests, I came to the conclusion that UDP packets > were being lost when the flood of metrics came in every hour. I decided > to add a line to my cron jobs to sleep for a random interval of 0 - 10 > secs. This helped spread out the incoming packets and alleviate the > problem. > > But in your case, such a solution is not possible. Since I don't know > what kind of hardware and networking you have, I can't say for sure that > you would see the same problem. If adding these metrics to your Java code > isn't too much work, the best thing to do might be to just add them and > run some tests and see what happens.
So the key here is that UDP may not be reliable an enough of a solution for your situation, and in that case, a custom TCP may be a better idea. Now, there is one thing to note -- if the packets are being sent every second, if one packet it lost, it should be noted there'll be another one in the next second. That said, because ganglia may send simple differential packets, if one gets lost, then the whole results recieved by e.g. the web interface may drift off the actual value. (That said, the full value is resent every hour or few hours or so, IIRC.) -- ~Mike - Just my two cents - No man is an island, and no man is unable.
