On Tue, 2004-09-07 at 17:38, Vineet Mokashi wrote: > This issue has been addressed earlier, but no fool-proof solution was > suggested. > Now that I have been monitoring our 32-node Beowulf, running RHEL-AS with > 2.4.20-20.9smp kernel, the problem has started to look grave. > > Upto 60% of the packets on each node can be attributed to gmond and the > inter-process communication is getting affected due to this. Turning on 'deaf' > mode does not help, since gmetad is unable connect to 8649 to get any data > from > the gmond daemons.
Whats the total bandwidth of those packets? and what kind of networking are you using? I have a decent sized cluster with multiple "clusters" defined in ganglia. Each one has its own multicast channel. The largest multicast channel is 64 nodes. They receive a little over 1 kB/s (when you average it out over time) from ganglia. When running on a gigabit network with a maximum speed of over 120MB/s in each direction, that's barely noticable. All of my gmonds are listening and all of them are broadcasting. > > As a probably workaround, I (painfully) configured gmond with a separate > multicast channel for each node, say 239.2.11.40-72, and removed all > trusted_host entries (but for the login node, which runs gmetad). > > This solves a part of the problem - every node's gmond communicating with > every > other node to probe data. The bandwidth consumption has decreased > significantly. The broken link with this set-up is that gmetad is unable to > query data from any node except the one running on the default multi-cast > channel. All the gmond daemons use the default XML port : 8649. You can define multiple data sources in your gmetad.conf. Each data source can have multiple IP:port comibnations specified. However, gmetad will only poll from one at a time. They're used for redundancy, not for each having a part of the 'cluster' data. However, if you have multiple data sources defined, they'll all show up next to each other on the ganglia webpage. For using a different port, just specify it as IP:port (ie 10.10.1.1:8652)
