Re: _ping_more / dropping pings
Personaly I did not like the idea of running my whole process as user root and so implemented mass ping using Wheel::Run and fping. Unfortunately I don't have it wrappered up into a nice distro. But it was not too hard to do. On Wed, Feb 11, 2009 at 12:46 PM, wrote: > i use POE::Component::Client::Ping to emit lots of pings > > and i find that i have to throttle the number of pings i have > outstanding; otherwise, POE::Component::Client::Ping doesn't see the > responses > > using a sniffer (tshark running on the box hosting my script), i can > see that the ICMP Echo Reply frames arrive just fine (as far as tshark > is concerned) > > i suppose that i could be seeing an effect somewhere in the OS, such > that if the flow of returning ICMP Echoes exceeds some threshold, the > NIC sees them, tshark sees them ... but somewhere higher up, a buffer > overflows and Perl never gets them > > or, i suppose that the flood of responses overwhelms some buffer > inside POE::Component::Client::Ping > > or perhaps somewhere in between: perhaps there is an OS buffer which > holds the ICMP Echo Reply frames and unless POE pulls them out of the > buffer fast enough, the buffer will overflow and POE won't see those > responses > > i notice that sometimes i can get away with several hundred > outstanding pings at once; sometimes, not much more than thirty. i > generally run with a ceiling of twenty (20). the variability depends > on load on my box? i don't know > > this isn't a problem -- the total time it takes to ping the ~350 > addresses i'm using as my test bed right now doesn't change, > regardless of the size of my 'ping heap' > > but i'm curious. would anyone know where the overflowing buffer > lives? if in the OS, how would i look to see its size? of inside > POE, again, how would i identify its size? > > i'm running Perl 5.8.8, POE 1.03, POE::Component::Client::Ping 1.14 > under CentOS 5.2 > > my code is visible at https://vishnu.fhcrc.org/ping/mass-ping > > the 'ping_nodes' subroutine is the master pinger ... with 'sub > _client_start', 'sub _ping_more', and 'sub _client_got_pong' playing > the usual roles > > --sk > > stuart kendrick > fhcrc > seattle, wa > >
Re: _ping_more / dropping pings
i played around with BufferSize, increasing it from 65536 to 1048576 by powers of two; i didn't see an effect on the number of dropped pings (my test 'mass-ping' session was pinging 510 addresses, of which ~360 were responding) i wanted 'netstat -s' output to show me dropped ICMP messages, i.e. messages overwritten in a buffer ... but 'wanting' and 'happening' are too different things ... i don't see a counter tracking such an event: Icmp: 588970 ICMP messages received 285250 input ICMP message failed. ICMP input histogram: destination unreachable: 362369 timeout in transit: 21028 echo requests: 11771 echo replies: 193799 2375973 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 29077 echo request: 2335125 echo replies: 11771 IcmpMsg: InType0: 237002 InType3: 363685 InType8: 11812 InType11: 21028 OutType0: 11812 OutType3: 29098 OutType8: 2396328 i tried assigning '100' to $POE::Component::Client::Ping::PKTSIZE ($POE::Component::Client::Ping::PKTSIZE = 100); i didn't notice a change in behavior setting Parallelism makes a difference though; i'm at 120 at the moment, which is larger, and simpler, than the ~30+ i can reach using my home-built throttling scheme i'm trying to think of additional ways in which i can explore this issue. -printing the value of BufferSize, as a way of uncovering what the OS default is?what i really want is to find some counter which tracks how many buffer slots get overwritten ... and of course, that counter may not exist -i've poked through /proc/sys/net/ipv4/icmp* -- i see ways to throttle *outbound* ICMP replies ... but nothing related to how the box accepts inbound frames -dig up copies of Stevens' and Comer's books, see if they discuss how this is typically handled additional suggestions? --sk Tony Cook wrote: [...] or perhaps somewhere in between: perhaps there is an OS buffer which holds the ICMP Echo Reply frames and unless POE pulls them out of the buffer fast enough, the buffer will overflow and POE won't see those responses I'd be inclined to think it was this. POE::Component::Client::Ping->spawn has a BufferSize parameter you could use to check - try setting it fairly large, like 65536, and see if that reduces the number of lost ping responses. Tony
Re: _ping_more / dropping pings
[email protected] wrote: or perhaps somewhere in between: perhaps there is an OS buffer which holds the ICMP Echo Reply frames and unless POE pulls them out of the buffer fast enough, the buffer will overflow and POE won't see those responses This is almost certainly the issue. i notice that sometimes i can get away with several hundred outstanding pings at once; sometimes, not much more than thirty. i generally run with a ceiling of twenty (20). the variability depends on load on my box? i don't know I believe it'll be related to MTU and the amount of other network traffic occurring at the moment the pings are sent. this isn't a problem -- the total time it takes to ping the ~350 addresses i'm using as my test bed right now doesn't change, regardless of the size of my 'ping heap' but i'm curious. would anyone know where the overflowing buffer lives? if in the OS, how would i look to see its size? of inside POE, again, how would i identify its size? It'll be controlled by something in /proc/sys/net/ipv4/icmp*, I think.. Good luck, Rob
Re: _ping_more / dropping pings
On Wed, Feb 11, 2009 at 11:46:31AM -0800, [email protected] wrote: > i use POE::Component::Client::Ping to emit lots of pings > > and i find that i have to throttle the number of pings i have > outstanding; otherwise, POE::Component::Client::Ping doesn't see the > responses > > using a sniffer (tshark running on the box hosting my script), i can > see that the ICMP Echo Reply frames arrive just fine (as far as tshark > is concerned) > > i suppose that i could be seeing an effect somewhere in the OS, such > that if the flow of returning ICMP Echoes exceeds some threshold, the > NIC sees them, tshark sees them ... but somewhere higher up, a buffer > overflows and Perl never gets them > > or, i suppose that the flood of responses overwhelms some buffer > inside POE::Component::Client::Ping > > or perhaps somewhere in between: perhaps there is an OS buffer which > holds the ICMP Echo Reply frames and unless POE pulls them out of the > buffer fast enough, the buffer will overflow and POE won't see those > responses I'd be inclined to think it was this. POE::Component::Client::Ping->spawn has a BufferSize parameter you could use to check - try setting it fairly large, like 65536, and see if that reduces the number of lost ping responses. Tony
