Steve, Ganglia 3.0 supports unicast as well as multicast. This has been a big help for me in terms of the type of overhead that you mentioned. Rather than have every node in the cluster keep track of every other node, I have set things up so that each node sends its data to only two places: itself, and a central "collection" server.
I run an instance of gmond on the central server that does not monitor itself, but simply listens for metrics from the cluster nodes. I do this for each separate cluster. (All these different gmonds listen on different ports.) Then I have gmetad on the central server contact only the gmonds on the central server to get reports on cluster data. Right now, we use this to monitor about 600 nodes over 4 different clusters. This only generates an incoming network traffic load of about 60 KB/s on the central server (and that even includes about 45 extra custom metrics beyond the standard ones). The outgoing ganglia network traffic for the individual nodes is very tiny. I do have one useful suggestion: Do not have gmetad write to an ext3 filesystem. The journaling overhead is a killer. When we started with ext3, the load on the central server was constantly around 4-5 (even before we added our custom metrics). Moving to ext2 caused the load to drop to about 1.4. --Rick -------------------------- Rick Mohr Systems Developer Ohio Supercomputer Center On Tue, 14 Jun 2005, Stephen Cartwright wrote: > Thanks... That was helpful... I'm not sure why I got it into my head > that they were alternatives and not complementary. > > Also I have some more technical quesitons about Ganglia... > > Am I correct in that Ganglia only uses multicast, and so to monitor > two clusters on different subnets you must have a router that supports > multicast. > > Also how scalable is Ganglia. I understand that Ganglia caches > information about every other machine on each node... at least when I > telnet to gmond on a machine all the machines information appears. > Would this not generate a lot of overhead in a cluster with thousands > of nodes? > > Thank you for your time! > > Steve > > > On 6/14/05, Dan Rich <[EMAIL PROTECTED]> wrote: > > Stephen Cartwright wrote: > > > > >I am looking at both Ganglia and Nagios... any comments on how they > > >compare? > > > > > > > > You're comparing apples and oranges. > > > > Nagios is an active system monitor, it's along the lines of HP Openview > > or OpenNMS. It doesn't do any graphing, it is focused on service and > > host uptime monitoring. > > > > Ganglia is better used as a data collector and trending tool, along the > > lines of cricket, cacti, mrtg, or any other SNMP monitoring tool that > > generates graphs based on collected data. > > > > We run both (in fact, three of the above, Nagios, Cricket, and Ganglia); > > as they all give you different views into the system. Nagios is what > > pages me in the middle of the night. Cricket is where we go for > > long-term trending and data collection for non-Linux boxen (routers, air > > handlers, etc.). And Ganglia gives us high precision system > > statistics. Most SNMP monitors only run every 5 minutes, whereas > > Ganglia gives you statistics much more frequently. > > > > -- > > Dan Rich <[EMAIL PROTECTED]> | http://www.employees.org/~drich/ > > | "Step up to red alert!" "Are you sure, > > sir? > > | It means changing the bulb in the > > sign..." > >
