Ben - I'm not sure you've hit any unicast scalability elbow. Until we had bridging for multicast put in to cross our multiple VLANs, we used unicast with much bigger clusters of machines.
I suspect that there's some flapping going on. I would have more than one spokesmachine available for gmetad to poke, IMHO, but that should be unrelated. Are all of your non-spokesman machines consistent with their configs ? and are they all *only* doing unicast to the spokesman node ? (i.e. no multicast at all ?) I would run gstat on all of them to make sure that they all look consistent, and then maybe telnet to the spokesman's gmond port periodically to see if you can't find a pattern with the nodes coming in/out... -john ----- Original Message ---- > From: Ben Rockwood <[EMAIL PROTECTED]> > To: Ganglia <[email protected]> > Sent: Wednesday, March 19, 2008 12:18:18 AM > Subject: [Ganglia-general] Unicast Scalability > > I'm having some trouble with my unicast Ganglia setup. I have a single > gmetad watching 8 different clusters, clusters range from 5 nodes to 55 > nodes. Small clusters operate as expected. Larger clusters of 20 nodes > upwards flag a lot of nodes down. > > The strange thing is that if you just refresh the gmeta cluster page > you'll see nodes going up and down in a sine fashion... 15 down 10 up, > refresh, 14 down 11 up, refresh, 13 down 12 up, [...]. > > I'm thinking that I've hit a scaling threshold but none of the > processes, gmetad or gmond's, seem to struggle much. I've tuned the > server thread count on gmetad up. In all the setups there is a single > unicast head node to which all the other gmond's in the cluster report > to and is poked by gmetad. > > I saw an issue like this in the archives several years ago (something > about TN) but nothing seemed helpful. > > Can anyone offer suggestions on where to look to better understand this > issue or share similar experiences? > > benr. > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > Ganglia-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/ganglia-general > ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Ganglia-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ganglia-general
