RE: [Ganglia-general] windows gmond client
Exactly. I should have been clearer. The default windows/cygwin client is neither correct enough (cygwin's fault) nor provides all the metrics we want (in fact, because some of our farms are not just HPC farms, we want some other metrics as well). I remain grateful to whoever developed it, none-the-less. I am not a windows man, but we are looking at the possibility of developing a fully native (no cygwin) client ourselves. The reason for the TCP question is that my feeling was that it would be much easier to produce a native first pass windows gmond client deliverying TCP only, rather that all that clever UDP stuff as well. But of course with the TCP route, I have fears of scaling. But there is a GEM in Martins reply (and a Doh moment for me), in that I assumed that every node would have to be polled by a gmetad to get the cluster info. But you remind me this is not so, I can do the structural equivalent of the udp unicast to a head node using TCP to a head node, that gmetad then interogates. Have I got this right guys? And the other thing for the community is asking whether anyone else out there is considering developing a native windows gmond. Kind regards, Richard -Original Message- From: michael chang [mailto:[EMAIL PROTECTED] Sent: 08 November 2005 00:21 To: [EMAIL PROTECTED] Cc: Grevis, Richard: IT (LDN); ganglia-general@lists.sourceforge.net Subject: Re: [Ganglia-general] windows gmond client On 11/7/05, Martin Knoblauch [EMAIL PROTECTED] wrote: If we were to cheat, and create a windows agent that only produced the XML via the tcp interface, and not the udp niceness, can anyone give me an idea of how this will scale? This obviously moves more work to gmetad. Will gmetad poop with 5 data sources, 100? Not knowing the Cygwin implementation at all, but what is wrong with using the unicast TCP setup. Just select one or two nodes per *cluster* to run gmond in TCP receive mode and let all other nodes send data to them. Use the selected node(s) as data source for gmetad. Much better network usage compared to the multicast mode, which produces traffic going up with N*N. And you don't have to worry about switches blocking IGMP traffic. I think he means that Ganglia on Cygwin is inaccurate because Cygwin supposedly misreports certain metrics or can't report others. That would make sense, since Cygwin is a POSIX emulation layer (or whatever you call it). That said, I'm not sure about the validity of that statement. I think he wants to know if there is a Windows-specific Ganglia client that e.g. uses metrics provided by the Windows kernel subsystems (or similar) that works better or is more accurate. [Which makes some sense, since there are reporting-mechanisms of some sort for Windows, I'm sure, since the System Resource Monitor (I know this was on 9x, forget about XP and the like) and the Task Manager (XP, or at least I believe 2000 and later versions or something) can show e.g. CPU usage. Whether these are internal-use-only or not, I have no clue.] -- ~Mike - Just my two cents - No man is an island, and no man is unable. For more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons.
RE: [Ganglia-general] windows gmond client
--- [EMAIL PROTECTED] wrote: Exactly. I should have been clearer. The default windows/cygwin client is neither correct enough (cygwin's fault) nor provides all the metrics we want (in fact, because some of our farms are not just HPC farms, we want some other metrics as well). I remain grateful to whoever developed it, none-the-less. As I said, I do not know the Cygwin client very well. I am not a windows man, but we are looking at the possibility of developing a fully native (no cygwin) client ourselves. The reason for the TCP question is that my feeling was that it would be much easier to produce a native first pass windows gmond client deliverying TCP only, rather that all that clever UDP stuff as well. Not really liking Windows myself, I believe the contribution of a native client would be very welcome. But of course with the TCP route, I have fears of scaling. But there is a GEM in Martins reply (and a Doh moment for me), in that I assumed that every node would have to be polled by a gmetad to get the cluster info. But you remind me this is not so, I can do the structural equivalent of the udp unicast to a head node using TCP to a head node, that gmetad then interogates. Have I got this right guys? Unfortunatelly I think the answer is no. I made the mistake to somehow associate gmond unicast with TCP which is wrong. Communication between the gmonds in a host group is always UDP. One ore more clients listen, while all push their data out (either multicast, or unicast). But you are right that gmetad only needs to communicate with the heads of the host groups. This communication is TCP. And the other thing for the community is asking whether anyone else out there is considering developing a native windows gmond. not me :-) Cheers Martin -- Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
Re: [Ganglia-general] windows gmond client
On 11/10/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Exactly. I should have been clearer. The default windows/cygwin client is neither correct enough (cygwin's fault) nor provides all the metrics we want (in fact, because some of our farms are not just HPC farms, we want some other metrics as well). I remain grateful to whoever developed it, none-the-less. Ganglia is awesome, and developed by a great team, in my opinion. I am not a windows man, but we are looking at the possibility of developing a fully native (no cygwin) client ourselves. The reason for the TCP question is that my feeling was that it would be much easier to produce a native first pass windows gmond client deliverying TCP only, rather that all that clever UDP stuff as well. Ganglia uses UDP - a TCP client would be harder, I would say. If you create a native client, that would be great! It would be nice if it could compile on MinGW, but any sort of native client 'aught to do, I suppose. But of course with the TCP route, I have fears of scaling. But there is a GEM in Martins reply (and a Doh moment for me), in that I assumed that every node would have to be polled by a gmetad to get the cluster info. But you remind me this is not so, I can do the structural equivalent of the udp unicast to a head node using TCP to a head node, that gmetad then interogates. Have I got this right guys? Unicast is UDP based, as much as it looks otherwise. That said, since Unicast is supposed to reduce network bandwidth, the overhead of TCP doesn't look so bad so long as starts are staggered or something; the only limiting factor is the insanely low TCP connections maximum imposed in Operating Systems (presumably to avoid a random person flooding, I'd bet). Of course, a TCP client would require relevant changes in gmond on other platforms... And the other thing for the community is asking whether anyone else out there is considering developing a native windows gmond. Kind regards, Richard -Original Message- From: michael chang [mailto:[EMAIL PROTECTED] Sent: 08 November 2005 00:21 To: [EMAIL PROTECTED] Cc: Grevis, Richard: IT (LDN); ganglia-general@lists.sourceforge.net Subject: Re: [Ganglia-general] windows gmond client On 11/7/05, Martin Knoblauch [EMAIL PROTECTED] wrote: If we were to cheat, and create a windows agent that only produced the XML via the tcp interface, and not the udp niceness, can anyone give me an idea of how this will scale? This obviously moves more work to gmetad. Will gmetad poop with 5 data sources, 100? Not knowing the Cygwin implementation at all, but what is wrong with using the unicast TCP setup. Just select one or two nodes per *cluster* to run gmond in TCP receive mode and let all other nodes send data to them. Use the selected node(s) as data source for gmetad. Much better network usage compared to the multicast mode, which produces traffic going up with N*N. And you don't have to worry about switches blocking IGMP traffic. I think he means that Ganglia on Cygwin is inaccurate because Cygwin supposedly misreports certain metrics or can't report others. That would make sense, since Cygwin is a POSIX emulation layer (or whatever you call it). That said, I'm not sure about the validity of that statement. I think he wants to know if there is a Windows-specific Ganglia client that e.g. uses metrics provided by the Windows kernel subsystems (or similar) that works better or is more accurate. [Which makes some sense, since there are reporting-mechanisms of some sort for Windows, I'm sure, since the System Resource Monitor (I know this was on 9x, forget about XP and the like) and the Task Manager (XP, or at least I believe 2000 and later versions or something) can show e.g. CPU usage. Whether these are internal-use-only or not, I have no clue.] -- ~Mike - Just my two cents - No man is an island, and no man is unable. For more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. -- ~Mike - Just my two cents - No man is an island, and no man is unable.
Re: [Ganglia-general] windows gmond client
Hi Richard, --- [EMAIL PROTECTED] wrote: All, against all probability, but for reasonable historical reasons, we run windows based HPC applications. What kind of HPC stuff is a financial institution running? Just curious :-) If we were to cheat, and create a windows agent that only produced the XML via the tcp interface, and not the udp niceness, can anyone give me an idea of how this will scale? This obviously moves more work to gmetad. Will gmetad poop with 5 data sources, 100? Not knowing the Cygwin implementation at all, but what is wrong with using the unicast TCP setup. Just select one or two nodes per *cluster* to run gmond in TCP receive mode and let all other nodes send data to them. Use the selected node(s) as data source for gmetad. Much better network usage compared to the multicast mode, which produces traffic going up with N*N. And you don't have to worry about switches blocking IGMP traffic. 5 Datasources schould be no problem for gmetad. I have no idea about 100 or more. Can someone suggest something clever to get windows node producing ganglia data in a lightweight way? This likely needs a native client. Cheers Martin -- Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
