Is multicast really advantageous? I know for what I am trying to do I wron't be just monitoring a few nodes on a subnet... I want to monitor them all. Why then multicast with the added compexity and having to deal with having possibly incompatible hardware?
Is it possible to use broadcast and not multicast and is that a good idea? Or is the unicast suggestion above the optimal solution in terms of overhead? Thanks in advance! Steve On 6/15/05, Jason A. Smith <[EMAIL PROTECTED]> wrote: > I am surprised that you can monitor about 600 nodes with a disk based > rrd directory, especially since you added nearly twice the standard > number of metrics. A few years ago, with normal hardware at that time, > I couldn't go above a few hundred nodes without loading down the gmetad > server, even with tricks like raid or a mounted file via loopback for > the rrd directory. > > Today we are monitoring nearly 2,000 nodes, split into 10 clusters, with > no load problems using a tmpfs filesystem for the rrds. I should note > that we also increased the gmetad polling interval from the default 15 > seconds to 60, which helps. To prevent loss of data, every night gmetad > is stopped, the rrd directory is copied to disk and gmetad is restarted > which only takes a few seconds. > > We also don't notice any problem with multicast, even with our largest > cluster of over 400 nodes, we just set a higher ttl and enabled > multicast forwarding on our router. We also have igmp enabled on a > switched network, so the multicast is only sent where it is supposed to > be sent. This allows gmetad to fail-over to another gmond for that > cluster's data if the one it was polling does not respond. > > A quick check with tcpdump shows about 150 packets/sec average on our > largest cluster which is a little more than 10kB/s or less than 0.1% of > the available bandwidth on a 100 Base-T network. The exact packet rate > will vary depending on the usage of the nodes, basically how quickly the > metric parameters reach their thresholds. > > In general, I don't really see any reason to be concerned about extra > overhead with ganglia except some slight increase in network traffic, > which should be negligible on a modern 100 Base-T or Gig-E full-duplex, > switched network. The extra processing of received multicast data is > also negligible and won't significantly increase gmond's overhead. I > believe the internal memory hash gmond uses to store metrics is > statically defined at run time and can be tweaked in the config file if > really necessary. > > ~Jason > > > On Wed, 2005-06-15 at 15:46 -0400, Rick Mohr wrote: > > Steve, > > > > Ganglia 3.0 supports unicast as well as multicast. This has been a big > > help for me in terms of the type of overhead that you mentioned. Rather > > than have every node in the cluster keep track of every other node, I have > > set things up so that each node sends its data to only two places: itself, > > and a central "collection" server. > > > > I run an instance of gmond on the central server that does not monitor > > itself, but simply listens for metrics from the cluster nodes. I do this > > for each separate cluster. (All these different gmonds listen on > > different ports.) > > > > Then I have gmetad on the central server contact only the gmonds on the > > central server to get reports on cluster data. Right now, we use this to > > monitor about 600 nodes over 4 different clusters. This only generates an > > incoming network traffic load of about 60 KB/s on the central server (and > > that even includes about 45 extra custom metrics beyond the standard > > ones). The outgoing ganglia network traffic for the individual nodes is > > very tiny. > > > > I do have one useful suggestion: Do not have gmetad write to an ext3 > > filesystem. The journaling overhead is a killer. When we started with > > ext3, the load on the central server was constantly around 4-5 (even > > before we added our custom metrics). Moving to ext2 caused the load to > > drop to about 1.4. > > > > --Rick > > > > -------------------------- > > Rick Mohr > > Systems Developer > > Ohio Supercomputer Center > > > > On Tue, 14 Jun 2005, Stephen Cartwright wrote: > > > > > Thanks... That was helpful... I'm not sure why I got it into my head > > > that they were alternatives and not complementary. > > > > > > Also I have some more technical quesitons about Ganglia... > > > > > > Am I correct in that Ganglia only uses multicast, and so to monitor > > > two clusters on different subnets you must have a router that supports > > > multicast. > > > > > > Also how scalable is Ganglia. I understand that Ganglia caches > > > information about every other machine on each node... at least when I > > > telnet to gmond on a machine all the machines information appears. > > > Would this not generate a lot of overhead in a cluster with thousands > > > of nodes? > > > > > > Thank you for your time! > > > > > > Steve > > > > > > > > > On 6/14/05, Dan Rich <[EMAIL PROTECTED]> wrote: > > > > Stephen Cartwright wrote: > > > > > > > > >I am looking at both Ganglia and Nagios... any comments on how they > > > > >compare? > > > > > > > > > > > > > > You're comparing apples and oranges. > > > > > > > > Nagios is an active system monitor, it's along the lines of HP Openview > > > > or OpenNMS. It doesn't do any graphing, it is focused on service and > > > > host uptime monitoring. > > > > > > > > Ganglia is better used as a data collector and trending tool, along the > > > > lines of cricket, cacti, mrtg, or any other SNMP monitoring tool that > > > > generates graphs based on collected data. > > > > > > > > We run both (in fact, three of the above, Nagios, Cricket, and Ganglia); > > > > as they all give you different views into the system. Nagios is what > > > > pages me in the middle of the night. Cricket is where we go for > > > > long-term trending and data collection for non-Linux boxen (routers, air > > > > handlers, etc.). And Ganglia gives us high precision system > > > > statistics. Most SNMP monitors only run every 5 minutes, whereas > > > > Ganglia gives you statistics much more frequently. > > > > > > > > -- > > > > Dan Rich <[EMAIL PROTECTED]> | http://www.employees.org/~drich/ > > > > | "Step up to red alert!" "Are you > > > > sure, sir? > > > > | It means changing the bulb in the > > > > sign..." > > > > > > > > > > ------------------------------------------------------- > > SF.Net email is sponsored by: Discover Easy Linux Migration Strategies > > from IBM. Find simple to follow Roadmaps, straightforward articles, > > informative Webcasts and more! Get everything you need to get up to > > speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click > > _______________________________________________ > > Ganglia-general mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/ganglia-general > > > -- > /------------------------------------------------------------------\ > | Jason A. Smith Email: [EMAIL PROTECTED] | > | Atlas Computing Facility, Bldg. 510M Phone: (631)344-4226 | > | Brookhaven National Lab, P.O. Box 5000 Fax: (631)344-7616 | > | Upton, NY 11973-5000 | > \------------------------------------------------------------------/ > > >
