On 10/7/05, Rick Mohr <[EMAIL PROTECTED]> wrote: > > On Fri, 7 Oct 2005, michael chang wrote: > > > So the key here is that UDP may not be reliable an enough of a > > solution for your situation, and in that case, a custom TCP may be a > > better idea. > > > > Now, there is one thing to note -- if the packets are being sent every > > second, if one packet it lost, it should be noted there'll be another > > one in the next second. That said, because ganglia may send simple > > differential packets, if one gets lost, then the whole results > > recieved by e.g. the web interface may drift off the actual value. > > (That said, the full value is resent every hour or few hours or so, > > IIRC.) > > I agree that a TCP solution would probably be needed for this particular > application. Basically, it boils down to understanding the application > well enough to know whether UDP loss is acceptable. > > Because Ganglia allows you to do things like set thresholds for sending > data, expiring data if it gets too old, etc. we have been able to work > around such limitations. For my applications, I just assume that some > data will be lost. Then I just have to figure out how much loss is > acceptable, and send some sort of notification via Nagios when it passes > the acceptable limit. > > Although it would be really cool if Ganglia implemented some TCP support > for reliable communications. But if there are a large number of TCP > connections, then there might have to be some sort of "tree" structure. > Rather than having 1000 nodes trying to open connections to the a single > gmond to send their metrics, there would probably need to be "collectors" > so that these 100 nodes sent data to collector #1, then next 100 sent it > to collector #2, etc. Then there could be a main collector that collects > metrics from the sub-collectors.... > > But that would be quite a bit of work.
The thing is that Ganglia was designed around multicast + UDP. This involves fiddling with TTL, but saves in terms of bandwidth per packet. A TCP hierarchy may make more sense with the new unicast structure, but on internal communications it may be a bit much. I know that all of my gmond setups have a gmetad daemon also, and about 1/3 of them run the web interface (because it's a WAN network... I use OpenVPN). It's funky in terms of the way it behaves, but it works, for the most part. With network hierarchies, it's hard to tell who to send data to without wasting information or routing inefficiently (think the Freenet mess). What I will say, is that TCP Unicast wouldn't work in this setup very well, or at the very least, it would scale horridly (and I do plan on scaling it exponentially). -- ~Mike - Just my two cents - No man is an island, and no man is unable.
