Bill, I will try increasing the timeout to 9 seconds. Because this is not always happening to the same device, I will try it on our backbone map and a couple of building maps that have many switches, this way I can see if it helps.
Is there an easy way to change the timeout for all devices? Thanks, Steve -----Original Message----- From: [email protected] [mailto:[EMAIL PROTECTED] On Behalf Of William W. Fisher Sent: Monday, January 24, 2005 5:06 PM To: InterMapper Discussion Subject: Re: [IM-Talk] DOWN outages because of no response to ICMP Echo polling - has anyone else had this type of problem I don't see anything out of the ordinary in the snoop output. It looks like the checksums are okay, and the ICMP requests are making it out to the network -- there just isn't a reply for three requests in a row. The number of entries in the debug log for "ICMP pkt not handled from xx.xx.xx.xx" is a little suspicious. This debug message means that InterMapper received an ICMP echo reply when there was no device on any map expecting it. Under normal circumstances, this can happen if you close a map or remove a device while a ping is "in flight". The map is closed when the ping reply comes back, so the packet is recorded as "not handled". The "ICMP pkt not handled" message can also be logged if the ping reply is really late or there are two or more replies to a single echo request. The number of "ICMP pkt not handled" messages does not correlate to your significant packet loss, however. What happens to InterMapper's packet loss measurements if you triple the probe timeout from 3 seconds to 9 seconds for one of the ping devices? Thanks, Bill Fisher Dartware, LLC On Jan 24, 2005, at 2:53 PM, Steven Good wrote: > Bill, > > Here you go.... > > I dumped the data from the snoop using the "-v" option. It looks like > all > the data you need is available with the snoop -v option (I was using > the > -V before that is a bit less verbose). > > Since I had quite a lot of snoop data, I am only including the time > frame > that includes one good ICMP poll, one failed (three retires) poll, then > one more good poll that occurs after the failed poll period. > > > The Debug grep data follows after the snoop data. > > ETHER: ----- Ether Header ----- > ETHER: > ETHER: Packet 21 arrived at 9:36:50.78 > ETHER: Packet size = 62 bytes > ETHER: Destination = 0:0:c:7:ac:1, Cisco > ETHER: Source = 8:0:20:c4:e:56, Sun > ETHER: Ethertype = 0800 (IP) > ETHER: > IP: ----- IP Header ----- > IP: > IP: Version = 4 > IP: Header length = 20 bytes > IP: Type of service = 0x00 > IP: xxx. .... = 0 (precedence) > IP: ...0 .... = normal delay > IP: .... 0... = normal throughput > IP: .... .0.. = normal reliability > IP: .... ..0. = not ECN capable transport > IP: .... ...0 = no ECN congestion experienced > IP: Total length = 48 bytes > IP: Identification = 40892 > IP: Flags = 0x4 > IP: .1.. .... = do not fragment > IP: ..0. .... = last fragment > IP: Fragment offset = 0 bytes > IP: Time to live = 255 seconds/hops > IP: Protocol = 1 (ICMP) > IP: Header checksum = 91d5 > IP: Source address = 129.21.6.1, rit010n19 > IP: Destination address = 129.21.66.15, x009-066m-15n3.rit.edu > IP: No options > IP: > ICMP: ----- ICMP Header ----- > ICMP: > ICMP: Type = 8 (Echo request) > ICMP: Code = 0 (ID: 27552 Sequence number: 43572) > ICMP: Checksum = 9cc8 > ICMP: > > ETHER: ----- Ether Header ----- > ETHER: > ETHER: Packet 22 arrived at 9:36:50.78 > ETHER: Packet size = 62 bytes > ETHER: Destination = 8:0:20:c4:e:56, Sun > ETHER: Source = aa:0:4:0:fd:7, DEC (DECNET) > ETHER: Ethertype = 0800 (IP) > ETHER: > IP: ----- IP Header ----- > IP: > IP: Version = 4 > IP: Header length = 20 bytes > IP: Type of service = 0x00 > IP: xxx. .... = 0 (precedence) > IP: ...0 .... = normal delay > IP: .... 0... = normal throughput > IP: .... .0.. = normal reliability > IP: .... ..0. = not ECN capable transport > IP: .... ...0 = no ECN congestion experienced > IP: Total length = 48 bytes > IP: Identification = 40892 > IP: Flags = 0x4 > IP: .1.. .... = do not fragment > IP: ..0. .... = last fragment > IP: Fragment offset = 0 bytes > IP: Time to live = 252 seconds/hops > IP: Protocol = 1 (ICMP) > IP: Header checksum = 94d5 > IP: Source address = 129.21.66.15, x009-066m-15n3.rit.edu > IP: Destination address = 129.21.6.1, rit010n19 > IP: No options > IP: > ICMP: ----- ICMP Header ----- > ICMP: > ICMP: Type = 0 (Echo reply) > ICMP: Code = 0 (ID: 27552 Sequence number: 43572) > ICMP: Checksum = a4c8 > ICMP: > > ETHER: ----- Ether Header ----- > ETHER: > ETHER: Packet 23 arrived at 9:38:50.79 > ETHER: Packet size = 62 bytes > ETHER: Destination = 0:0:c:7:ac:1, Cisco > ETHER: Source = 8:0:20:c4:e:56, Sun > ETHER: Ethertype = 0800 (IP) > ETHER: > IP: ----- IP Header ----- > IP: > IP: Version = 4 > IP: Header length = 20 bytes > IP: Type of service = 0x00 > IP: xxx. .... = 0 (precedence) > IP: ...0 .... = normal delay > IP: .... 0... = normal throughput > IP: .... .0.. = normal reliability > IP: .... ..0. = not ECN capable transport > IP: .... ...0 = no ECN congestion experienced > IP: Total length = 48 bytes > IP: Identification = 40895 > IP: Flags = 0x4 > IP: .1.. .... = do not fragment > IP: ..0. .... = last fragment > IP: Fragment offset = 0 bytes > IP: Time to live = 255 seconds/hops > IP: Protocol = 1 (ICMP) > IP: Header checksum = 91d2 > IP: Source address = 129.21.6.1, rit010n19 > IP: Destination address = 129.21.66.15, x009-066m-15n3.rit.edu > IP: No options > IP: > ICMP: ----- ICMP Header ----- > ICMP: > ICMP: Type = 8 (Echo request) > ICMP: Code = 0 (ID: 52560 Sequence number: 45139) > ICMP: Checksum = f72 > ICMP: > > ETHER: ----- Ether Header ----- > ETHER: > ETHER: Packet 24 arrived at 9:38:53.79 > ETHER: Packet size = 62 bytes > ETHER: Destination = 0:0:c:7:ac:1, Cisco > ETHER: Source = 8:0:20:c4:e:56, Sun > ETHER: Ethertype = 0800 (IP) > ETHER: > IP: ----- IP Header ----- > IP: > IP: Version = 4 > IP: Header length = 20 bytes > IP: Type of service = 0x00 > IP: xxx. .... = 0 (precedence) > IP: ...0 .... = normal delay > IP: .... 0... = normal throughput > IP: .... .0.. = normal reliability > IP: .... ..0. = not ECN capable transport > IP: .... ...0 = no ECN congestion experienced > IP: Total length = 48 bytes > IP: Identification = 40896 > IP: Flags = 0x4 > IP: .1.. .... = do not fragment > IP: ..0. .... = last fragment > IP: Fragment offset = 0 bytes > IP: Time to live = 255 seconds/hops > IP: Protocol = 1 (ICMP) > IP: Header checksum = 91d1 > IP: Source address = 129.21.6.1, rit010n19 > IP: Destination address = 129.21.66.15, x009-066m-15n3.rit.edu > IP: No options > IP: > ICMP: ----- ICMP Header ----- > ICMP: > ICMP: Type = 8 (Echo request) > ICMP: Code = 0 (ID: 52561 Sequence number: 45172) > ICMP: Checksum = b57 > ICMP: > > ETHER: ----- Ether Header ----- > ETHER: > ETHER: Packet 25 arrived at 9:38:56.79 > ETHER: Packet size = 62 bytes > ETHER: Destination = 0:0:c:7:ac:1, Cisco > ETHER: Source = 8:0:20:c4:e:56, Sun > ETHER: Ethertype = 0800 (IP) > ETHER: > IP: ----- IP Header ----- > IP: > IP: Version = 4 > IP: Header length = 20 bytes > IP: Type of service = 0x00 > IP: xxx. .... = 0 (precedence) > IP: ...0 .... = normal delay > IP: .... 0... = normal throughput > IP: .... .0.. = normal reliability > IP: .... ..0. = not ECN capable transport > IP: .... ...0 = no ECN congestion experienced > IP: Total length = 48 bytes > IP: Identification = 40897 > IP: Flags = 0x4 > IP: .1.. .... = do not fragment > IP: ..0. .... = last fragment > IP: Fragment offset = 0 bytes > IP: Time to live = 255 seconds/hops > IP: Protocol = 1 (ICMP) > IP: Header checksum = 91d0 > IP: Source address = 129.21.6.1, rit010n19 > IP: Destination address = 129.21.66.15, x009-066m-15n3.rit.edu > IP: No options > IP: > ICMP: ----- ICMP Header ----- > ICMP: > ICMP: Type = 8 (Echo request) > ICMP: Code = 0 (ID: 52562 Sequence number: 45226) > ICMP: Checksum = f3b > ICMP: > > ETHER: ----- Ether Header ----- > ETHER: > ETHER: Packet 26 arrived at 9:40:50.79 > ETHER: Packet size = 62 bytes > ETHER: Destination = 0:0:c:7:ac:1, Cisco > ETHER: Source = 8:0:20:c4:e:56, Sun > ETHER: Ethertype = 0800 (IP) > ETHER: > IP: ----- IP Header ----- > IP: > IP: Version = 4 > IP: Header length = 20 bytes > IP: Type of service = 0x00 > IP: xxx. .... = 0 (precedence) > IP: ...0 .... = normal delay > IP: .... 0... = normal throughput > IP: .... .0.. = normal reliability > IP: .... ..0. = not ECN capable transport > IP: .... ...0 = no ECN congestion experienced > IP: Total length = 48 bytes > IP: Identification = 40900 > IP: Flags = 0x4 > IP: .1.. .... = do not fragment > IP: ..0. .... = last fragment > IP: Fragment offset = 0 bytes > IP: Time to live = 255 seconds/hops > IP: Protocol = 1 (ICMP) > IP: Header checksum = 91cd > IP: Source address = 129.21.6.1, rit010n19 > IP: Destination address = 129.21.66.15, x009-066m-15n3.rit.edu > IP: No options > IP: > ICMP: ----- ICMP Header ----- > ICMP: > ICMP: Type = 8 (Echo request) > ICMP: Code = 0 (ID: 12128 Sequence number: 46721) > ICMP: Checksum = a37f > ICMP: > > ETHER: ----- Ether Header ----- > ETHER: > ETHER: Packet 27 arrived at 9:40:50.79 > ETHER: Packet size = 62 bytes > ETHER: Destination = 8:0:20:c4:e:56, Sun > ETHER: Source = aa:0:4:0:fd:7, DEC (DECNET) > ETHER: Ethertype = 0800 (IP) > ETHER: > IP: ----- IP Header ----- > IP: > IP: Version = 4 > IP: Header length = 20 bytes > IP: Type of service = 0x00 > IP: xxx. .... = 0 (precedence) > IP: ...0 .... = normal delay > IP: .... 0... = normal throughput > IP: .... .0.. = normal reliability > IP: .... ..0. = not ECN capable transport > IP: .... ...0 = no ECN congestion experienced > IP: Total length = 48 bytes > IP: Identification = 40900 > IP: Flags = 0x4 > IP: .1.. .... = do not fragment > IP: ..0. .... = last fragment > IP: Fragment offset = 0 bytes > IP: Time to live = 252 seconds/hops > IP: Protocol = 1 (ICMP) > IP: Header checksum = 94cd > IP: Source address = 129.21.66.15, x009-066m-15n3.rit.edu > IP: Destination address = 129.21.6.1, rit010n19 > IP: No options > IP: > ICMP: ----- ICMP Header ----- > ICMP: > ICMP: Type = 0 (Echo reply) > ICMP: Code = 0 (ID: 12128 Sequence number: 46721) > ICMP: Checksum = ab7f > ICMP: > > $ egrep ICMP Debug* > Debug200501130000.txt:18:10:53 ICMP pkt not handled from 129.21.51.38, > ID > = 72 > Debug200501130000.txt:21:36:01 ICMP pkt not handled from 129.21.51.62, > ID > = 6fa2 > Debug200501130000.txt:21:57:40 ICMP pkt not handled from 129.21.51.51, > ID > = bba2 > Debug200501130000.txt:23:55:12 ICMP pkt not handled from 129.21.1.18, > ID = > ad02 > Debug200501130000.txt:23:55:12 ICMP pkt not handled from 129.21.1.18, > ID = > ad12 > Debug200501140000.txt:05:32:46 ICMP pkt not handled from 129.21.51.1, > ID = > c172 > Debug200501140000.txt:08:56:09 ICMP pkt not handled from 129.21.51.65, > ID > = 80b2 > Debug200501140000.txt:09:46:36 ICMP pkt not handled from 129.21.51.73, > ID > = 82f2 > Debug200501140000.txt:13:14:49 ICMP pkt not handled from 129.21.51.9, > ID = > e6d2 > Debug200501140000.txt:16:15:28 ICMP pkt not handled from 129.21.51.9, > ID = > 57c2 > Debug200501140000.txt:17:30:12 ICMP pkt not handled from 129.21.1.18, > ID = > 5792 > Debug200501140000.txt:17:46:41 ICMP pkt not handled from 129.21.51.8, > ID = > 2572 > Debug200501140000.txt:22:55:24 ICMP pkt not handled from 129.21.51.83, > ID > = e52 > Debug200501150000.txt:01:35:12 ICMP pkt not handled from 129.21.1.18, > ID = > bda2 > Debug200501150000.txt:01:35:12 ICMP pkt not handled from 129.21.1.18, > ID = > bdb2 > Debug200501150000.txt:02:39:34 ICMP pkt not handled from 129.21.51.82, > ID > = bbe2 > Debug200501150000.txt:07:40:10 ICMP pkt not handled from 129.21.1.18, > ID = > 61a2 > Debug200501150000.txt:07:40:10 ICMP pkt not handled from 129.21.1.18, > ID = > 61b2 > Debug200501150000.txt:09:21:09 ICMP pkt not handled from 129.21.51.89, > ID > = b6f2 > Debug200501150000.txt:10:48:53 ICMP pkt not handled from 129.21.51.85, > ID > = 4802 > Debug200501150000.txt:14:43:44 ICMP pkt not handled from 129.21.51.86, > ID > = 9292 > Debug200501150000.txt:18:05:11 ICMP pkt not handled from 129.21.1.18, > ID = > c8b2 > Debug200501150000.txt:18:05:11 ICMP pkt not handled from 129.21.1.18, > ID = > c8d2 > Debug200501160000.txt:05:03:08 ICMP pkt not handled from 129.21.51.16, > ID > = c072 > Debug200501160000.txt:12:47:37 ICMP pkt not handled from 129.21.51.18, > ID > = 4992 > Debug200501160000.txt:17:02:29 ICMP pkt not handled from 129.21.51.5, > ID = > e152 > Debug200501160000.txt:21:08:08 ICMP pkt not handled from 129.21.51.42, > ID > = a710 > Debug200501160000.txt:22:23:57 ICMP pkt not handled from 129.21.51.24, > ID > = c3f2 > Debug200501170000.txt:01:48:55 ICMP pkt not handled from 129.21.51.4, > ID = > e762 > Debug200501170000.txt:02:24:43 ICMP pkt not handled from 129.21.51.24, > ID > = 5482 > Debug200501170000.txt:04:48:21 ICMP pkt not handled from 129.21.51.3, > ID = > 5d2 > Debug200501170000.txt:05:13:15 ICMP pkt not handled from 129.21.51.84, > ID > = 8792 > Debug200501170000.txt:13:57:37 ICMP pkt not handled from 129.21.51.81, > ID > = 3692 > Debug200501170000.txt:16:06:21 ICMP pkt not handled from 129.21.51.30, > ID > = 3372 > Debug200501170000.txt:19:58:52 ICMP pkt not handled from 129.21.51.85, > ID > = ad2 > Debug200501170000.txt:22:51:14 ICMP pkt not handled from 129.21.51.3, > ID = > e032 > Debug200501180000.txt:02:42:25 ICMP pkt not handled from > 129.21.155.253, > ID = dd22 > Debug200501180000.txt:02:53:57 ICMP pkt not handled from > 129.21.169.252, > ID = 33d2 > Debug200501180000.txt:04:23:31 ICMP pkt not handled from 129.21.51.36, > ID > = a530 > Debug200501180000.txt:06:34:25 ICMP pkt not handled from 129.21.51.89, > ID > = a622 > Debug200501180000.txt:08:09:58 ICMP pkt not handled from 129.21.51.80, > ID > = 6a32 > Debug200501180000.txt:12:56:06 ICMP pkt not handled from 129.21.51.86, > ID > = a1a2 > Debug200501180000.txt:17:16:52 ICMP pkt not handled from 129.21.51.65, > ID > = 8332 > Debug200501180000.txt:20:10:15 ICMP pkt not handled from 129.21.1.18, > ID = > 3ed2 > Debug200501180000.txt:20:10:15 ICMP pkt not handled from 129.21.1.18, > ID = > 3ee2 > Debug200501180000.txt:20:13:59 ICMP pkt not handled from 129.21.51.83, > ID > = dd92 > Debug200501180000.txt:20:35:13 ICMP pkt not handled from 129.21.1.18, > ID = > 34d2 > Debug200501180000.txt:20:35:13 ICMP pkt not handled from 129.21.1.18, > ID = > 34e2 > Debug200501190000.txt:06:35:12 ICMP pkt not handled from 129.21.1.18, > ID = > 5082 > Debug200501190000.txt:06:35:12 ICMP pkt not handled from 129.21.1.18, > ID = > 5092 > Debug200501190000.txt:09:36:32 ICMP pkt not handled from 129.21.51.24, > ID > = 76a2 > Debug200501190000.txt:10:59:30 ICMP pkt not handled from 129.21.51.82, > ID > = 78d2 > Debug200501190000.txt:11:59:47 ICMP pkt not handled from 129.21.51.82, > ID > = 5c52 > Debug200501190000.txt:13:55:34 ICMP pkt not handled from 129.21.51.1, > ID = > cad2 > Debug200501190000.txt:17:09:04 ICMP pkt not handled from 129.21.51.10, > ID > = d542 > Debug200501190000.txt:17:25:12 ICMP pkt not handled from 129.21.1.18, > ID = > 1a12 > Debug200501190000.txt:17:25:12 ICMP pkt not handled from 129.21.1.18, > ID = > 1a22 > Debug200501190000.txt:19:05:11 ICMP pkt not handled from 129.21.1.18, > ID = > 6f72 > Debug200501190000.txt:19:05:11 ICMP pkt not handled from 129.21.1.18, > ID = > 6f82 > Debug200501190000.txt:19:08:48 ICMP pkt not handled from 129.21.51.11, > ID > = 1142 > Debug200501190000.txt:19:15:13 ICMP pkt not handled from 129.21.1.18, > ID = > e22 > Debug200501190000.txt:19:15:13 ICMP pkt not handled from 129.21.1.18, > ID = > e32 > Debug200501190000.txt:20:10:13 ICMP pkt not handled from 129.21.1.18, > ID = > f572 > Debug200501190000.txt:20:10:13 ICMP pkt not handled from 129.21.1.18, > ID = > f582 > Debug200501190000.txt:21:30:11 ICMP pkt not handled from 129.21.1.18, > ID = > 8ff2 > Debug200501190000.txt:21:30:11 ICMP pkt not handled from 129.21.1.18, > ID = > 9002 > Debug200501190000.txt:21:39:08 ICMP pkt not handled from 129.21.51.49, > ID > = fc22 > Debug200501190000.txt:23:09:38 ICMP pkt not handled from 129.21.51.11, > ID > = 5d02 > Debug200501190000.txt:23:25:12 ICMP pkt not handled from 129.21.1.18, > ID = > 50b2 > Debug200501190000.txt:23:25:12 ICMP pkt not handled from 129.21.1.18, > ID = > 50c2 > Debug200501200000.txt:01:00:11 ICMP pkt not handled from 129.21.1.18, > ID = > ef2 > Debug200501200000.txt:02:30:12 ICMP pkt not handled from 129.21.1.18, > ID = > a792 > Debug200501200000.txt:02:30:12 ICMP pkt not handled from 129.21.1.18, > ID = > a7b2 > Debug200501200000.txt:05:24:12 ICMP pkt not handled from 129.21.51.65, > ID > = 7022 > Debug200501200000.txt:15:17:18 ICMP pkt not handled from 129.21.1.30, > ID = > 5600 > Debug200501200000.txt:16:50:53 ICMP pkt not handled from 129.21.51.53, > ID > = bcc2 > Debug200501200000.txt:17:40:12 ICMP pkt not handled from 129.21.1.18, > ID = > 1922 > Debug200501200000.txt:19:48:05 ICMP pkt not handled from 129.21.51.8, > ID = > 6802 > Debug200501200000.txt:XSocketICMP_BSD.cp: sendto errno 11 > Debug200501210000.txt:06:07:29 ICMP pkt not handled from 129.21.51.82, > ID > = 48d2 > Debug200501210000.txt:17:23:10 ICMP pkt not handled from > 129.21.51.101, ID > = 18b0 > Debug200501220000.txt:02:46:27 ICMP pkt not handled from 129.21.51.97, > ID > = 810 > Debug200501230000.txt:23:56:55 ICMP pkt not handled from 129.21.51.92, > ID > = 17c0 > Debug200501240000.txt:00:57:10 ICMP pkt not handled from > 129.21.51.108, ID > = 8c90 > Debug200501240000.txt:13:38:09 ICMP pkt not handled from > 129.21.51.111, ID > = 9ed0 > > Thanks, > > Steve > > -----Original Message----- > From: [email protected] > [mailto:[EMAIL PROTECTED] On Behalf Of William W. > Fisher > Sent: Monday, January 24, 2005 2:07 PM > To: InterMapper Discussion > Subject: Re: [IM-Talk] DOWN outages because of no response to ICMP Echo > polling - has anyone else had this type of problem > > > On Jan 24, 2005, at 12:39 PM, Steven Good wrote: > >> I am including the snoop data. I already removed any lines not >> relevant >> to this device. I am only including the snoop for this time period >> showing a successful poll both before and after the poll with out >> replies. >> > > Steven: > > Can you collect a more detailed trace showing the ICMP requests and > responses, perhaps using tcpdump to a file? There are some things I > would like to check in the outgoing ICMP echo requests: > > - IP TOS > - IP header checksum > - ICMP checksum > etc. > > These are things in the echo request where a misstep would lead to no > response. I would also recommend grep'ing InterMapper's debug log for > "ICMP" to check if any API errors have been logged. > > Thanks, > > Bill Fisher > Dartware, LLC > > > ____________________________________________________________________ > List archives: > http://www.mail-archive.com/intermapper-talk%40list.dartware.com/ > To unsubscribe: send email to: [EMAIL PROTECTED] ____________________________________________________________________ List archives: http://www.mail-archive.com/intermapper-talk%40list.dartware.com/ To unsubscribe: send email to: [EMAIL PROTECTED]
smime.p7s
Description: S/MIME cryptographic signature
