On Thu, 6 Oct 2005, Peter Bodik wrote: > The user-defined metrics are less-efficient, right? So having 1000s (from > each host) could cause some problems ... I'd like to know what kind of > problems we can expect and what do to about them.
Short answer: You are most likely correct Long answer: We have been playing around with adding extra metrics for our clusters at OSC. Our situation is slightly different: we have a large number of nodes with a small number of extra metrics added via gmetric (about 5 or so per host). But in total, this translates into about 1500 extra metrics. For our initial testing, we kept things simple and had an hourly cron job that reported all these metrics. The metrics were set to expire (and hence disappear from Ganglia) in a little over an hour. I soon noticed that there were always a few nodes that were missing some metrics. But when I checked the metrics listed by the local gmond processes on these nodes. The values were always there. After several different tests, I came to the conclusion that UDP packets were being lost when the flood of metrics came in every hour. I decided to add a line to my cron jobs to sleep for a random interval of 0 - 10 secs. This helped spread out the incoming packets and alleviate the problem. But in your case, such a solution is not possible. Since I don't know what kind of hardware and networking you have, I can't say for sure that you would see the same problem. If adding these metrics to your Java code isn't too much work, the best thing to do might be to just add them and run some tests and see what happens. --Rick -------------------------- Rick Mohr Systems Developer Ohio Supercomputer Center
