On Wed, Jun 15, 2005 at 04:46:21PM -0600, Stephen Cartwright wrote: > Is multicast really advantageous? I know for what I am trying to do I > wron't be just monitoring a few nodes on a subnet... I want to monitor > them all. Why then multicast with the added compexity and having to > deal with having possibly incompatible hardware?
On all the systems I'm running, multicast just worked. I don't even notice the traffic on my 225 node cluster. It's just not an issue. > Is it possible to use broadcast and not multicast and is that a good > idea? Or is the unicast suggestion above the optimal solution in terms > of overhead? If you don't want multicast, just use unicast. -- Brooks > Thanks in advance! > > Steve > > > On 6/15/05, Jason A. Smith <[EMAIL PROTECTED]> wrote: > > I am surprised that you can monitor about 600 nodes with a disk based > > rrd directory, especially since you added nearly twice the standard > > number of metrics. A few years ago, with normal hardware at that time, > > I couldn't go above a few hundred nodes without loading down the gmetad > > server, even with tricks like raid or a mounted file via loopback for > > the rrd directory. > > > > Today we are monitoring nearly 2,000 nodes, split into 10 clusters, with > > no load problems using a tmpfs filesystem for the rrds. I should note > > that we also increased the gmetad polling interval from the default 15 > > seconds to 60, which helps. To prevent loss of data, every night gmetad > > is stopped, the rrd directory is copied to disk and gmetad is restarted > > which only takes a few seconds. > > > > We also don't notice any problem with multicast, even with our largest > > cluster of over 400 nodes, we just set a higher ttl and enabled > > multicast forwarding on our router. We also have igmp enabled on a > > switched network, so the multicast is only sent where it is supposed to > > be sent. This allows gmetad to fail-over to another gmond for that > > cluster's data if the one it was polling does not respond. > > > > A quick check with tcpdump shows about 150 packets/sec average on our > > largest cluster which is a little more than 10kB/s or less than 0.1% of > > the available bandwidth on a 100 Base-T network. The exact packet rate > > will vary depending on the usage of the nodes, basically how quickly the > > metric parameters reach their thresholds. > > > > In general, I don't really see any reason to be concerned about extra > > overhead with ganglia except some slight increase in network traffic, > > which should be negligible on a modern 100 Base-T or Gig-E full-duplex, > > switched network. The extra processing of received multicast data is > > also negligible and won't significantly increase gmond's overhead. I > > believe the internal memory hash gmond uses to store metrics is > > statically defined at run time and can be tweaked in the config file if > > really necessary. > > > > ~Jason > > > > > > On Wed, 2005-06-15 at 15:46 -0400, Rick Mohr wrote: > > > Steve, > > > > > > Ganglia 3.0 supports unicast as well as multicast. This has been a big > > > help for me in terms of the type of overhead that you mentioned. Rather > > > than have every node in the cluster keep track of every other node, I have > > > set things up so that each node sends its data to only two places: itself, > > > and a central "collection" server. > > > > > > I run an instance of gmond on the central server that does not monitor > > > itself, but simply listens for metrics from the cluster nodes. I do this > > > for each separate cluster. (All these different gmonds listen on > > > different ports.) > > > > > > Then I have gmetad on the central server contact only the gmonds on the > > > central server to get reports on cluster data. Right now, we use this to > > > monitor about 600 nodes over 4 different clusters. This only generates an > > > incoming network traffic load of about 60 KB/s on the central server (and > > > that even includes about 45 extra custom metrics beyond the standard > > > ones). The outgoing ganglia network traffic for the individual nodes is > > > very tiny. > > > > > > I do have one useful suggestion: Do not have gmetad write to an ext3 > > > filesystem. The journaling overhead is a killer. When we started with > > > ext3, the load on the central server was constantly around 4-5 (even > > > before we added our custom metrics). Moving to ext2 caused the load to > > > drop to about 1.4. > > > > > > --Rick > > > > > > -------------------------- > > > Rick Mohr > > > Systems Developer > > > Ohio Supercomputer Center > > > > > > On Tue, 14 Jun 2005, Stephen Cartwright wrote: > > > > > > > Thanks... That was helpful... I'm not sure why I got it into my head > > > > that they were alternatives and not complementary. > > > > > > > > Also I have some more technical quesitons about Ganglia... > > > > > > > > Am I correct in that Ganglia only uses multicast, and so to monitor > > > > two clusters on different subnets you must have a router that supports > > > > multicast. > > > > > > > > Also how scalable is Ganglia. I understand that Ganglia caches > > > > information about every other machine on each node... at least when I > > > > telnet to gmond on a machine all the machines information appears. > > > > Would this not generate a lot of overhead in a cluster with thousands > > > > of nodes? > > > > > > > > Thank you for your time! > > > > > > > > Steve > > > > > > > > > > > > On 6/14/05, Dan Rich <[EMAIL PROTECTED]> wrote: > > > > > Stephen Cartwright wrote: > > > > > > > > > > >I am looking at both Ganglia and Nagios... any comments on how they > > > > > >compare? > > > > > > > > > > > > > > > > > You're comparing apples and oranges. > > > > > > > > > > Nagios is an active system monitor, it's along the lines of HP > > > > > Openview > > > > > or OpenNMS. It doesn't do any graphing, it is focused on service and > > > > > host uptime monitoring. > > > > > > > > > > Ganglia is better used as a data collector and trending tool, along > > > > > the > > > > > lines of cricket, cacti, mrtg, or any other SNMP monitoring tool that > > > > > generates graphs based on collected data. > > > > > > > > > > We run both (in fact, three of the above, Nagios, Cricket, and > > > > > Ganglia); > > > > > as they all give you different views into the system. Nagios is what > > > > > pages me in the middle of the night. Cricket is where we go for > > > > > long-term trending and data collection for non-Linux boxen (routers, > > > > > air > > > > > handlers, etc.). And Ganglia gives us high precision system > > > > > statistics. Most SNMP monitors only run every 5 minutes, whereas > > > > > Ganglia gives you statistics much more frequently. > > > > > > > > > > -- > > > > > Dan Rich <[EMAIL PROTECTED]> | http://www.employees.org/~drich/ > > > > > | "Step up to red alert!" "Are you > > > > > sure, sir? > > > > > | It means changing the bulb in the > > > > > sign..." > > > > > > > > > > > > > > ------------------------------------------------------- > > > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > > > from IBM. Find simple to follow Roadmaps, straightforward articles, > > > informative Webcasts and more! Get everything you need to get up to > > > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > > > _______________________________________________ > > > Ganglia-general mailing list > > > [email protected] > > > https://lists.sourceforge.net/lists/listinfo/ganglia-general > > > > > -- > > /------------------------------------------------------------------\ > > | Jason A. Smith Email: [EMAIL PROTECTED] | > > | Atlas Computing Facility, Bldg. 510M Phone: (631)344-4226 | > > | Brookhaven National Lab, P.O. Box 5000 Fax: (631)344-7616 | > > | Upton, NY 11973-5000 | > > \------------------------------------------------------------------/ > > > > > > > > > ------------------------------------------------------- > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > from IBM. Find simple to follow Roadmaps, straightforward articles, > informative Webcasts and more! Get everything you need to get up to > speed, fast. http://ads.osdn.com/?ad_idt77&alloc_id492&opÌk > _______________________________________________ > Ganglia-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/ganglia-general
