I was unable to track down the exact bandwidth usage by gmond daemons. The packet count I was talking about was found out using ethereal. Its able to decode those packets and show requests and replies from each node. And, we have a GigE interconnect for the 32 node cluster and an off-shelf ethernet router for a 10 node cluster.
Now, by assigning a different port number to small clusters like we have is managable. This approach completely mars the scalability of ganglia. Unicast will be the best solution for this issue and thats what 2.6.0 will have. This issue was highlighted in a previous thread: https://sourceforge.net/mailarchive/forum.php?thread_id=4578097&forum_id=7186 None of the suggested approaches worked for me. Thanks for your attention. Regards, Vineet. Quoting Sean Dilda <[EMAIL PROTECTED]>: > On Tue, 2004-09-07 at 17:38, Vineet Mokashi wrote: > > This issue has been addressed earlier, but no fool-proof solution was > suggested. > > Now that I have been monitoring our 32-node Beowulf, running RHEL-AS with > > 2.4.20-20.9smp kernel, the problem has started to look grave. > > > > Upto 60% of the packets on each node can be attributed to gmond and the > > inter-process communication is getting affected due to this. Turning on > 'deaf' > > mode does not help, since gmetad is unable connect to 8649 to get any data > from > > the gmond daemons. > > Whats the total bandwidth of those packets? and what kind of networking > are you using? I have a decent sized cluster with multiple "clusters" > defined in ganglia. Each one has its own multicast channel. The > largest multicast channel is 64 nodes. They receive a little over 1 > kB/s (when you average it out over time) from ganglia. When running on > a gigabit network with a maximum speed of over 120MB/s in each > direction, that's barely noticable. > > All of my gmonds are listening and all of them are broadcasting. > > > > > As a probably workaround, I (painfully) configured gmond with a separate > > multicast channel for each node, say 239.2.11.40-72, and removed all > > trusted_host entries (but for the login node, which runs gmetad). > > > > This solves a part of the problem - every node's gmond communicating with > every > > other node to probe data. The bandwidth consumption has decreased > > significantly. The broken link with this set-up is that gmetad is unable > to > > query data from any node except the one running on the default multi-cast > > channel. All the gmond daemons use the default XML port : 8649. > > You can define multiple data sources in your gmetad.conf. Each data > source can have multiple IP:port comibnations specified. However, > gmetad will only poll from one at a time. They're used for redundancy, > not for each having a part of the 'cluster' data. > > However, if you have multiple data sources defined, they'll all show up > next to each other on the ganglia webpage. > > For using a different port, just specify it as IP:port (ie > 10.10.1.1:8652) > ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
