Re: [c-nsp] Cursed IP address
On 11/30/2014 12:22 AM, Victor Sudakov wrote: Daniel Roesen wrote: Check for enabled IGMP snooping on this switch. Try disabling snooping. We have already tried that, with no effect. Besides, IGMP snooping should not affect packets with only one unicast source address on only one switch. Should not and does not are two different things. pt ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Daniel Roesen wrote: On Sat, Nov 29, 2014 at 09:41:45PM +0600, Victor Sudakov wrote: We have set up port monitor sessions in various parts of the network and have found out the following. One of the C3560X-24P in the chain of identical switches does not let through packets with src=10.65.127.246dst=224.0.0.5. Neither when such packets transit the switch nor when it's configured on one of it's own Vlan interfaces. Other switches in the chain are identical from the hardware/software point of view, and configured almost identically, but do not suffer from the problem. Check for enabled IGMP snooping on this switch. Try disabling snooping. We have already tried that, with no effect. Besides, IGMP snooping should not affect packets with only one unicast source address on only one switch. Wouldn't outrule broken hardware... Some bit flip error or so. We may try to change the hardware some day, but for the present it's easier to just avoid one particular IP address. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Cool story. We have set up port monitor sessions in various parts of the network and have found out the following. One of the C3560X-24P in the chain of identical switches does not let through packets with src=10.65.127.246dst=224.0.0.5. Neither when such packets transit the switch nor when it's configured on one of it's own Vlan interfaces. Other switches in the chain are identical from the hardware/software point of view, and configured almost identically, but do not suffer from the problem. Just reload that box, those things are not hardware defects usually. Lukas ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Lukas Tribus wrote: Are there any IP filters on the layer 2 side of this? Are you using CoPP and the IP is denied there? No. PASOLINK does not do IP filtering. It can only do some Ethernet frame filtering, like filtering out LLDP or STP frames, but no such filters are even configured. Just because its not configured or not configurable doesn't mean its not actually doing it (it may be a bug). I have an Ethernet over FrameRelay Radio PTMP system that after a few months of uptime inserts the UDP payload of customer A into the TCP payload of customer B (customer B using this TCP session to transfer files to a AS400 that doesn't check TCP checksums and therefor the UDP payload of customer A makes it into the application of customer B). The only fix here is to reload the whole system. The very same PTMP system sometimes drops traffic of a certain mac address, although there are no layer 2 rules (other that different vlans). Cool story. We have set up port monitor sessions in various parts of the network and have found out the following. One of the C3560X-24P in the chain of identical switches does not let through packets with src=10.65.127.246dst=224.0.0.5. Neither when such packets transit the switch nor when it's configured on one of it's own Vlan interfaces. Other switches in the chain are identical from the hardware/software point of view, and configured almost identically, but do not suffer from the problem. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
On Sat, Nov 29, 2014 at 09:41:45PM +0600, Victor Sudakov wrote: We have set up port monitor sessions in various parts of the network and have found out the following. One of the C3560X-24P in the chain of identical switches does not let through packets with src=10.65.127.246dst=224.0.0.5. Neither when such packets transit the switch nor when it's configured on one of it's own Vlan interfaces. Other switches in the chain are identical from the hardware/software point of view, and configured almost identically, but do not suffer from the problem. Check for enabled IGMP snooping on this switch. Try disabling snooping. Wouldn't outrule broken hardware... Some bit flip error or so. Best regards, Daniel -- CLUE-RIPE -- Jabber: d...@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0 ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Hello Victor Seems to be some multicast receiving problem 224.0.0.5/6? Are there filters / IGMP stuff? What kind of L2 design do you have in the segment? Some years ago we had problems with multicast and SDH MUX systems. Regards Gregor Good point, thank you. After looking at debug OSPF hello, we have found out that Hello packets with src=10.65.127.246 are being sent by the device Nov 27 05:29:35.923: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.246 but they are not Rcv-ed by other devices on the network segment. Hello packets with any other src IP address, e.g. 10.65.127.243, are being received as expected. We are at a loss what could be blocking packets with this particular src addr. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Friedrich, Gregor wrote: Seems to be some multicast receiving problem 224.0.0.5/6? Seems like it. The main engima is why only multicast packets with src=10.65.127.246 are affected and not from other source addresses. Packets from x.x.x.246 addresses in other networks also work without problems. Are there filters / IGMP stuff? No. What kind of L2 design do you have in the segment? Some years ago we had problems with multicast and SDH MUX systems. iPASOLINK radio relay towers, iPASOLINK IDU/ODU. Below is the OSPF hello debug output from 10.65.127.243 where you can see that 10.65.127.243 does not receive hellos from 10.65.127.246. Then it marks 10.65.127.246 as down and then immediately receives a unicast (?) hello from 10.65.127.246. Then the problem is repeated. Nov 27 09:14:17.133: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from LOADING to FULL, Loading Done Nov 27 09:14:26.780: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:14:36.300: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:14:45.586: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:14:55.501: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:02.421: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from FULL to DOWN, Neighbor Down: Dead timer expired Nov 27 09:15:05.257: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:05.257: OSPF: Rcv hello from 10.65.127.246 area 0 from Vlan333 10.65.127.246 Nov 27 09:15:05.257: OSPF: Send immediate hello to nbr 10.65.127.246, src address 10.65.127.246, on Vlan333 Nov 27 09:15:05.257: OSPF: Send hello to 10.65.127.246 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:05.257: OSPF: End of hello processing Nov 27 09:15:05.273: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from LOADING to FULL, Loading Done Nov 27 09:15:14.945: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:24.860: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:34.859: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:44.413: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:52.868: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from FULL to DOWN, Neighbor Down: Dead timer expired Nov 27 09:15:54.043: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:54.051: OSPF: Rcv hello from 10.65.127.246 area 0 from Vlan333 10.65.127.246 Nov 27 09:15:54.051: OSPF: Send immediate hello to nbr 10.65.127.246, src address 10.65.127.246, on Vlan333 Nov 27 09:15:54.051: OSPF: Send hello to 10.65.127.246 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:54.051: OSPF: End of hello processing Nov 27 09:15:54.059: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from LOADING to FULL, Loading Done Nov 27 09:16:04.041: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:13.235: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:22.781: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:32.025: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:41.805: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:43.374: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from FULL to DOWN, Neighbor Down: Dead timer expired Nov 27 09:16:51.569: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:51.578: OSPF: Rcv hello from 10.65.127.246 area 0 from Vlan333 10.65.127.246 Nov 27 09:16:51.578: OSPF: Send immediate hello to nbr 10.65.127.246, src address 10.65.127.246, on Vlan333 Nov 27 09:16:51.578: OSPF: Send hello to 10.65.127.246 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:51.578: OSPF: End of hello processing Nov 27 09:16:51.586: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from LOADING to FULL, Loading Done Nov 27 09:17:01.350: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:10.560: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:20.509: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:30.063: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:36.690: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from FULL to DOWN, Neighbor Down: Dead timer expired Nov 27 09:17:39.902: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:39.902: OSPF: Rcv hello from 10.65.127.246 area 0 from Vlan333 10.65.127.246 Nov 27 09:17:39.902: OSPF: Send immediate hello to nbr 10.65.127.246, src address 10.65.127.246, on Vlan333 Nov 27 09:17:39.902: OSPF: Send hello to 10.65.127.246 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:39.902: OSPF:
Re: [c-nsp] Cursed IP address
Do you have control of the devices at each L2 hop? Can you run packet captures and see where the hello is dropped? Date: Thu, 27 Nov 2014 17:08:45 +0600 From: v...@mpeks.tomsk.su To: friedr...@pdv-sachsen.net CC: vlaso...@sibptus.tomsk.ru; cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Cursed IP address Friedrich, Gregor wrote: Seems to be some multicast receiving problem 224.0.0.5/6? Seems like it. The main engima is why only multicast packets with src=10.65.127.246 are affected and not from other source addresses. Packets from x.x.x.246 addresses in other networks also work without problems. Are there filters / IGMP stuff? No. What kind of L2 design do you have in the segment? Some years ago we had problems with multicast and SDH MUX systems. iPASOLINK radio relay towers, iPASOLINK IDU/ODU. Below is the OSPF hello debug output from 10.65.127.243 where you can see that 10.65.127.243 does not receive hellos from 10.65.127.246. Then it marks 10.65.127.246 as down and then immediately receives a unicast (?) hello from 10.65.127.246. Then the problem is repeated. Nov 27 09:14:17.133: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from LOADING to FULL, Loading Done Nov 27 09:14:26.780: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:14:36.300: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:14:45.586: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:14:55.501: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:02.421: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from FULL to DOWN, Neighbor Down: Dead timer expired Nov 27 09:15:05.257: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:05.257: OSPF: Rcv hello from 10.65.127.246 area 0 from Vlan333 10.65.127.246 Nov 27 09:15:05.257: OSPF: Send immediate hello to nbr 10.65.127.246, src address 10.65.127.246, on Vlan333 Nov 27 09:15:05.257: OSPF: Send hello to 10.65.127.246 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:05.257: OSPF: End of hello processing Nov 27 09:15:05.273: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from LOADING to FULL, Loading Done Nov 27 09:15:14.945: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:24.860: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:34.859: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:44.413: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:52.868: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from FULL to DOWN, Neighbor Down: Dead timer expired Nov 27 09:15:54.043: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:54.051: OSPF: Rcv hello from 10.65.127.246 area 0 from Vlan333 10.65.127.246 Nov 27 09:15:54.051: OSPF: Send immediate hello to nbr 10.65.127.246, src address 10.65.127.246, on Vlan333 Nov 27 09:15:54.051: OSPF: Send hello to 10.65.127.246 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:15:54.051: OSPF: End of hello processing Nov 27 09:15:54.059: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from LOADING to FULL, Loading Done Nov 27 09:16:04.041: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:13.235: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:22.781: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:32.025: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:41.805: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:43.374: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from FULL to DOWN, Neighbor Down: Dead timer expired Nov 27 09:16:51.569: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:51.578: OSPF: Rcv hello from 10.65.127.246 area 0 from Vlan333 10.65.127.246 Nov 27 09:16:51.578: OSPF: Send immediate hello to nbr 10.65.127.246, src address 10.65.127.246, on Vlan333 Nov 27 09:16:51.578: OSPF: Send hello to 10.65.127.246 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:16:51.578: OSPF: End of hello processing Nov 27 09:16:51.586: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from LOADING to FULL, Loading Done Nov 27 09:17:01.350: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:10.560: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:20.509: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:30.063: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.243 Nov 27 09:17:36.690: %OSPF-5-ADJCHG: Process 20, Nbr 10.65.127.246 on Vlan333 from FULL to DOWN, Neighbor Down: Dead timer
Re: [c-nsp] Cursed IP address
Lars Fenneberg wrote: We are at a loss what could be blocking packets with this particular src addr. Are there any IP filters on the layer 2 side of this? Are you using CoPP and the IP is denied there? No. PASOLINK does not do IP filtering. It can only do some Ethernet frame filtering, like filtering out LLDP or STP frames, but no such filters are even configured. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Are there any IP filters on the layer 2 side of this? Are you using CoPP and the IP is denied there? No. PASOLINK does not do IP filtering. It can only do some Ethernet frame filtering, like filtering out LLDP or STP frames, but no such filters are even configured. Just because its not configured or not configurable doesn't mean its not actually doing it (it may be a bug). I have an Ethernet over FrameRelay Radio PTMP system that after a few months of uptime inserts the UDP payload of customer A into the TCP payload of customer B (customer B using this TCP session to transfer files to a AS400 that doesn't check TCP checksums and therefor the UDP payload of customer A makes it into the application of customer B). The only fix here is to reload the whole system. The very same PTMP system sometimes drops traffic of a certain mac address, although there are no layer 2 rules (other that different vlans). Just because the box isn't supposed to interact with a specific OSI layer doesn't mean its not actually doing it. 2 suggestions: - use point-to-point OSPF links for everything (especially for WAN links) - exclude your WAN link in your testing Regards, Lukas ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Murat Kaipov wrote: Hello Victor. Very interesting issue. Can you provide show ip ospf 12 database issue like this can occur when you have 10.65.127.246 address on other device. Here you are from two routers (sw-vertikos and Raskino): sw-vertikos#sh ip ospf 12 neighbor Neighbor ID Pri State Dead Time Address Interface 10.65.127.7 130 2WAY/DROTHER849 msec10.65.127.249 Vlan22 10.65.127.9 130 FULL/DR 916 msec10.65.127.250 Vlan22 10.65.127.10 1 2WAY/DROTHER715 msec10.65.127.248 Vlan22 10.65.127.12 1 2WAY/DROTHER799 msec10.65.127.252 Vlan22 10.65.127.13 1 INIT/DROTHER690 msec10.65.127.235 Vlan22 10.65.127.14 1 INIT/DROTHER916 msec10.65.127.245 Vlan22 10.65.127.15130 INIT/DROTHER682 msec10.65.127.251 Vlan22 10.65.127.17 1 2WAY/DROTHER900 msec10.65.127.241 Vlan22 10.65.127.19 1 2WAY/DROTHER774 msec10.65.127.238 Vlan22 10.65.127.21 1 2WAY/DROTHER883 msec10.65.127.244 Vlan22 10.65.127.22 0 INIT/DROTHER849 msec10.65.127.243 Vlan22 10.65.127.23 1 2WAY/DROTHER782 msec10.65.127.230 Vlan22 10.65.127.24 1 2WAY/DROTHER706 msec10.65.127.231 Vlan22 10.65.155.11 1 2WAY/DROTHER823 msec10.65.127.253 Vlan22 172.16.146.11 1 2WAY/DROTHER840 msec10.65.127.247 Vlan22 sw-vertikos#sh ip ospf 12 interface vlan 22 Vlan22 is up, line protocol is up Internet Address 10.65.127.246/27, Area 0 Process ID 12, Router ID 10.65.127.246, Network Type BROADCAST, Cost: 1 Topology-MTIDCostDisabledShutdown Topology Name 0 1 no noBase Transmit Delay is 1 sec, State DROTHER, Priority 1 Designated Router (ID) 10.65.127.9, Interface address 10.65.127.250 Backup Designated router (ID) 10.65.127.9, Interface address 10.65.127.250 Timer intervals configured, Hello 333 msec, Dead 1, Wait 1, Retransmit 5 oob-resync timeout 40 Hello due in 46 msec Supports Link-local Signaling (LLS) Cisco NSF helper support enabled IETF NSF helper support enabled Index 1/1, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 1, maximum is 1 Last flood scan time is 0 msec, maximum is 0 msec Neighbor Count is 15, Adjacent neighbor count is 1 Adjacent with neighbor 10.65.127.9 (Designated Router) Suppress hello for 0 neighbor(s) Raskino#sh ip ospf interface gi 0/2 GigabitEthernet0/2 is up, line protocol is up Internet Address 10.65.127.235/27, Area 0, Attached via Network Statement Process ID 10, Router ID 10.65.127.13, Network Type BROADCAST, Cost: 10 Topology-MTIDCostDisabledShutdown Topology Name 0 10no noBase Transmit Delay is 1 sec, State DROTHER, Priority 1 Designated Router (ID) 10.65.127.15, Interface address 10.65.127.251 Backup Designated router (ID) 10.65.127.9, Interface address 10.65.127.250 Timer intervals configured, Hello 333 msec, Dead 1, Wait 1, Retransmit 5 oob-resync timeout 40 Hello due in 112 msec Supports Link-local Signaling (LLS) Cisco NSF helper support enabled IETF NSF helper support enabled Index 3/3, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 0, maximum is 9 Last flood scan time is 0 msec, maximum is 4 msec Neighbor Count is 14, Adjacent neighbor count is 2 Adjacent with neighbor 10.65.127.9 (Backup Designated Router) Adjacent with neighbor 10.65.127.15 (Designated Router) Suppress hello for 0 neighbor(s) Raskino#sh ip ospf database OSPF Router with ID (10.65.127.13) (Process ID 10) Router Link States (Area 0) Link ID ADV Router Age Seq# Checksum Link count 10.65.127.7 10.65.127.7 418 0x805C 0x00CC30 13 10.65.127.9 10.65.127.9 657 0x8031 0x003D08 6 10.65.127.1010.65.127.10274 0x8059 0x00FD1D 6 10.65.127.1210.65.127.1211450x8047 0x00078B 5 10.65.127.1310.65.127.1318760x8035 0x001B1C 4 10.65.127.1410.65.127.14988 0x8034 0x00F4C9 4 10.65.127.1510.65.127.15867 0x8030 0x005DC0 4 10.65.127.1610.65.127.16666 0x8030 0x00E590 3 10.65.127.1710.65.127.17672 0x8031 0x00F1C1 4 10.65.127.1810.65.127.18742 0x8030 0x009ED3 3 10.65.127.1910.65.127.19609 0x8032 0x0039C7 5 10.65.127.2110.65.127.2110870x8044 0x005B3F 3 10.65.127.2210.65.127.22609 0x8033 0x004311 3 10.65.127.2310.65.127.2315040x8040 0x00D793 3 10.65.127.2410.65.127.2415140x8040 0x00F76F 3 10.65.127.246 10.65.127.246 271 0x8003 0x00A084 1 10.65.155.1110.65.155.1116780x803D
Re: [c-nsp] Cursed IP address
Antonio Querubin wrote: There are a dozen routers connected to a common 10.65.127.224/27 L2 backbone. All are running OSPF area 0. Any router which has the IP address 10.65.127.246 cannot establish OSPF adjacency with some other routers, showing them forever in the INIT/DROTHER or EXSTART/DROTHER state. When a different IP address is configured on the same router, the problem is solved. More over, when 10.65.127.246 is configured on ANY router in the segment, it experiences adjacency problems. We are currently using a workaround of never assigning 10.65.127.246 to any router. Is this Sauron's IP address, or is there some kind of curse thereon? Are you absolutely positively sure 10.65.127.246 still does not exist somewhere else on one of the devices on that network, for example as a stale router ID? I know that a duplicate router ID should cause a message like %OSPF-4-DUP_RTRID, and I have not seen this message in the logs. So am I not absolutely positively sure, but I am pretty sure. And I have posted a show ip ospf database output in another message. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Joshua Riesenweber wrote: Out of curiosity, is this the highest IP address on the router? On some routers there is no loopback interface, so yes, on some routers 10.65.127.246 may become the router ID. Is it possible that this is being chosen as the OSPF router ID and causing problems? What's wrong with 10.65.127.246 being an OSPF router ID? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Hello Victor. Very interesting issue. Can you provide show ip ospf 12 database issue like this can occur when you have 10.65.127.246 address on other device. Отправлено с iPad 26 нояб. 2014 г., в 9:42, Victor Sudakov v...@mpeks.tomsk.su написал(а): Colleagues, We have found a very interesting problem. There are a dozen routers connected to a common 10.65.127.224/27 L2 backbone. All are running OSPF area 0. Any router which has the IP address 10.65.127.246 cannot establish OSPF adjacency with some other routers, showing them forever in the INIT/DROTHER or EXSTART/DROTHER state. When a different IP address is configured on the same router, the problem is solved. More over, when 10.65.127.246 is configured on ANY router in the segment, it experiences adjacency problems. We are currently using a workaround of never assigning 10.65.127.246 to any router. Is this Sauron's IP address, or is there some kind of curse thereon? Below is a typical output sw-bptoik#sh ip ospf 12 neighbor Neighbor ID Pri State Dead Time Address Interface 10.65.127.7 130 2WAY/DROTHER984 msec10.65.127.249 Vlan22 10.65.127.9 130 FULL/DR 707 msec10.65.127.250 Vlan22 10.65.127.10 1 2WAY/DROTHER942 msec10.65.127.248 Vlan22 10.65.127.12 1 2WAY/DROTHER841 msec10.65.127.252 Vlan22 10.65.127.13 1 INIT/DROTHER942 msec10.65.127.235 Vlan22 10.65.127.14 1 INIT/DROTHER782 msec10.65.127.245 Vlan22 10.65.127.15130 INIT/DROTHER908 msec10.65.127.251 Vlan22 10.65.127.17 1 2WAY/DROTHER866 msec10.65.127.241 Vlan22 10.65.127.19 1 2WAY/DROTHER959 msec10.65.127.238 Vlan22 10.65.127.21 1 2WAY/DROTHER740 msec10.65.127.244 Vlan22 10.65.127.22 0 INIT/DROTHER766 msec10.65.127.243 Vlan22 10.65.127.23 1 2WAY/DROTHER891 msec10.65.127.230 Vlan22 10.65.127.24 1 2WAY/DROTHER682 msec10.65.127.231 Vlan22 10.65.155.11 1 2WAY/DROTHER816 msec10.65.127.253 Vlan22 172.16.146.11 1 2WAY/DROTHER858 msec10.65.127.247 Vlan22 sw-bptoik#sh ip ospf int Vlan22 Vlan22 is up, line protocol is up Internet Address 10.65.127.246/27, Area 0 Process ID 12, Router ID 10.65.127.246, Network Type BROADCAST, Cost: 1 Topology-MTIDCostDisabledShutdown Topology Name 0 1 no noBase Transmit Delay is 1 sec, State DROTHER, Priority 1 Designated Router (ID) 10.65.127.9, Interface address 10.65.127.250 Backup Designated router (ID) 10.65.127.9, Interface address 10.65.127.250 Timer intervals configured, Hello 333 msec, Dead 1, Wait 1, Retransmit 5 oob-resync timeout 40 Hello due in 264 msec Supports Link-local Signaling (LLS) Cisco NSF helper support enabled IETF NSF helper support enabled Index 1/1, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 1, maximum is 1 Last flood scan time is 0 msec, maximum is 0 msec Neighbor Count is 15, Adjacent neighbor count is 1 Adjacent with neighbor 10.65.127.9 (Designated Router) Suppress hello for 0 neighbor(s) -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Hi Victor, Nothing is wrong with that IP address specifically, but I was wondering if it is being chosen as the ID whether there is a conflict with another router ID, perhaps manually set. Regards,Josh Date: Wed, 26 Nov 2014 15:46:15 +0600 From: v...@mpeks.tomsk.su To: joshua.riesenwe...@outlook.com; cisco-nsp@puck.nether.net Subject: Re: [c-nsp] Cursed IP address Joshua Riesenweber wrote: Out of curiosity, is this the highest IP address on the router? On some routers there is no loopback interface, so yes, on some routers 10.65.127.246 may become the router ID. Is it possible that this is being chosen as the OSPF router ID and causing problems? What's wrong with 10.65.127.246 being an OSPF router ID? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Joshua Riesenweber wrote: Nothing is wrong with that IP address specifically, but I was wondering if it is being chosen as the ID whether there is a conflict with another router ID, perhaps manually set. A conflict of router IDs causes a well known message which I am not observing. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
If you sh arp do you see the right MAC address for the ip. Could that address be in use on some dumb device? Sent from my iPhone On Nov 26, 2014, at 2:43, Victor Sudakov v...@mpeks.tomsk.su wrote: Colleagues, We have found a very interesting problem. There are a dozen routers connected to a common 10.65.127.224/27 L2 backbone. All are running OSPF area 0. Any router which has the IP address 10.65.127.246 cannot establish OSPF adjacency with some other routers, showing them forever in the INIT/DROTHER or EXSTART/DROTHER state. When a different IP address is configured on the same router, the problem is solved. More over, when 10.65.127.246 is configured on ANY router in the segment, it experiences adjacency problems. We are currently using a workaround of never assigning 10.65.127.246 to any router. Is this Sauron's IP address, or is there some kind of curse thereon? Below is a typical output sw-bptoik#sh ip ospf 12 neighbor Neighbor ID Pri State Dead Time Address Interface 10.65.127.7 130 2WAY/DROTHER984 msec10.65.127.249 Vlan22 10.65.127.9 130 FULL/DR 707 msec10.65.127.250 Vlan22 10.65.127.10 1 2WAY/DROTHER942 msec10.65.127.248 Vlan22 10.65.127.12 1 2WAY/DROTHER841 msec10.65.127.252 Vlan22 10.65.127.13 1 INIT/DROTHER942 msec10.65.127.235 Vlan22 10.65.127.14 1 INIT/DROTHER782 msec10.65.127.245 Vlan22 10.65.127.15130 INIT/DROTHER908 msec10.65.127.251 Vlan22 10.65.127.17 1 2WAY/DROTHER866 msec10.65.127.241 Vlan22 10.65.127.19 1 2WAY/DROTHER959 msec10.65.127.238 Vlan22 10.65.127.21 1 2WAY/DROTHER740 msec10.65.127.244 Vlan22 10.65.127.22 0 INIT/DROTHER766 msec10.65.127.243 Vlan22 10.65.127.23 1 2WAY/DROTHER891 msec10.65.127.230 Vlan22 10.65.127.24 1 2WAY/DROTHER682 msec10.65.127.231 Vlan22 10.65.155.11 1 2WAY/DROTHER816 msec10.65.127.253 Vlan22 172.16.146.11 1 2WAY/DROTHER858 msec10.65.127.247 Vlan22 sw-bptoik#sh ip ospf int Vlan22 Vlan22 is up, line protocol is up Internet Address 10.65.127.246/27, Area 0 Process ID 12, Router ID 10.65.127.246, Network Type BROADCAST, Cost: 1 Topology-MTIDCostDisabledShutdown Topology Name 0 1 no noBase Transmit Delay is 1 sec, State DROTHER, Priority 1 Designated Router (ID) 10.65.127.9, Interface address 10.65.127.250 Backup Designated router (ID) 10.65.127.9, Interface address 10.65.127.250 Timer intervals configured, Hello 333 msec, Dead 1, Wait 1, Retransmit 5 oob-resync timeout 40 Hello due in 264 msec Supports Link-local Signaling (LLS) Cisco NSF helper support enabled IETF NSF helper support enabled Index 1/1, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 1, maximum is 1 Last flood scan time is 0 msec, maximum is 0 msec Neighbor Count is 15, Adjacent neighbor count is 1 Adjacent with neighbor 10.65.127.9 (Designated Router) Suppress hello for 0 neighbor(s) -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Antonio Querubin wrote: Are you absolutely positively sure 10.65.127.246 still does not exist somewhere else on one of the devices on that network, for example as a stale router ID? I know that a duplicate router ID should cause a message like %OSPF-4-DUP_RTRID, and I have not seen this message in the logs. So am I not absolutely positively sure, but I am pretty sure. And I have posted a show ip ospf database output in another message. Understood, but I have seen OSPF do weird things with router IDs in cases where IP addresses on router interfaces have been changed/removed without resetting the database by doing a 'clear ip ospf ...' or just rebooting the router. We have even recreated the problem in a separate VRF, created as a testbed specifically for the purpose. The problem is 100% reproducible. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Harold 'Buz' Dale wrote: If you sh arp do you see the right MAC address for the ip. Certainly. I can even ping it from any other device and telnet into it. Could that address be in use on some dumb device? No, it could not. We have even recreated the problem in a separate VRF, created as a testbed specifically for the purpose. The problem is 100% reproducible. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
As no one asked yet, is it possible to get ospf related debug from the not working router and the DR router of the subnet? Am 26.11.2014 14:44, schrieb Victor Sudakov: Harold 'Buz' Dale wrote: If you sh arp do you see the right MAC address for the ip. Certainly. I can even ping it from any other device and telnet into it. Could that address be in use on some dumb device? No, it could not. We have even recreated the problem in a separate VRF, created as a testbed specifically for the purpose. The problem is 100% reproducible. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Davis, Philip wrote: When you made your testbed did you have the same problem if the subnet mask was different? The problem persists regardless of the subnet mask: sw-parabel#sh ip ospf 20 neighbor Neighbor ID Pri State Dead Time Address Interface 10.65.127.241 1 FULL/DR 958 msec10.65.127.241 Vlan333 10.65.127.242 1 FULL/BDR916 msec10.65.127.242 Vlan333 10.65.127.243 1 INIT/DROTHER975 msec10.65.127.243 Vlan333 10.65.127.244 1 INIT/DROTHER899 msec10.65.127.244 Vlan333 10.65.127.245 1 INIT/DROTHER706 msec10.65.127.245 Vlan333 10.65.127.247 1 INIT/DROTHER748 msec10.65.127.247 Vlan333 sw-parabel#sh ip ospf interface vlan 333 Vlan333 is up, line protocol is up Internet Address 10.65.127.246/24, Area 0 Process ID 20, Router ID 10.65.127.246, Network Type BROADCAST, Cost: 1 Topology-MTIDCostDisabledShutdown Topology Name 0 1 no noBase Transmit Delay is 1 sec, State DROTHER, Priority 1 Designated Router (ID) 10.65.127.241, Interface address 10.65.127.241 Backup Designated router (ID) 10.65.127.242, Interface address 10.65.127.242 Timer intervals configured, Hello 333 msec, Dead 1, Wait 1, Retransmit 5 oob-resync timeout 40 Hello due in 170 msec Supports Link-local Signaling (LLS) Cisco NSF helper support enabled IETF NSF helper support enabled Index 1/1, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 0, maximum is 7 Last flood scan time is 0 msec, maximum is 8 msec Neighbor Count is 6, Adjacent neighbor count is 2 Adjacent with neighbor 10.65.127.241 (Designated Router) Adjacent with neighbor 10.65.127.242 (Backup Designated Router) Suppress hello for 0 neighbor(s) sw-parabel#sh arp vrf OSPF Protocol Address Age (min) Hardware Addr Type Interface Internet 10.65.127.241 4 6073.5cac.1cc5 ARPA Vlan333 Internet 10.65.127.242 4 4403.a7cb.2644 ARPA Vlan333 Internet 10.65.127.243 4 4403.a7d1.7dc3 ARPA Vlan333 Internet 10.65.127.244 4 4403.a7cc.72c2 ARPA Vlan333 Internet 10.65.127.245 4 4403.a7d1.64c2 ARPA Vlan333 Internet 10.65.127.246 - e4d3.f1cc.f2c1 ARPA Vlan333 Internet 10.65.127.247 4 4403.a7cf.74c3 ARPA Vlan333 sw-rrs-nvartovsk#sh arp vrf OSPF Protocol Address Age (min) Hardware Addr Type Interface Internet 10.65.127.241 0 6073.5cac.1cc5 ARPA Vlan333 Internet 10.65.127.242 0 4403.a7cb.2644 ARPA Vlan333 Internet 10.65.127.243 0 4403.a7d1.7dc3 ARPA Vlan333 Internet 10.65.127.244 0 4403.a7cc.72c2 ARPA Vlan333 Internet 10.65.127.245 0 4403.a7d1.64c2 ARPA Vlan333 Internet 10.65.127.246 0 e4d3.f1cc.f2c1 ARPA Vlan333 Internet 10.65.127.247 - 4403.a7cf.74c3 ARPA Vlan333 sw-rrs-nvartovsk#sh ip ospf 20 neighbor Neighbor ID Pri State Dead Time Address Interface 10.65.127.241 1 FULL/DR 1000 msec10.65.127.241 Vlan333 10.65.127.242 1 FULL/BDR748 msec10.65.127.242 Vlan333 10.65.127.243 1 2WAY/DROTHER865 msec10.65.127.243 Vlan333 10.65.127.244 1 2WAY/DROTHER916 msec10.65.127.244 Vlan333 10.65.127.245 1 2WAY/DROTHER723 msec10.65.127.245 Vlan333 sw-rrs-nvartovsk#sh ip ospf interface vlan 333 Vlan333 is up, line protocol is up Internet Address 10.65.127.247/24, Area 0 Process ID 20, Router ID 10.65.127.247, Network Type BROADCAST, Cost: 1 Topology-MTIDCostDisabledShutdown Topology Name 0 1 no noBase Transmit Delay is 1 sec, State DROTHER, Priority 1 Designated Router (ID) 10.65.127.241, Interface address 10.65.127.241 Backup Designated router (ID) 10.65.127.242, Interface address 10.65.127.242 Timer intervals configured, Hello 333 msec, Dead 1, Wait 1, Retransmit 5 oob-resync timeout 40 Hello due in 22 msec Supports Link-local Signaling (LLS) Cisco NSF helper support enabled IETF NSF helper support enabled Index 1/1, flood queue length 0 Next 0x0(0)/0x0(0) Last flood scan length is 0, maximum is 1 Last flood scan time is 0 msec, maximum is 8 msec Neighbor Count is 5, Adjacent neighbor count is 2 Adjacent with neighbor 10.65.127.241 (Designated Router) Adjacent with neighbor 10.65.127.242 (Backup Designated Router) Suppress hello for 0 neighbor(s) -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Cursed IP address
Karsten Thomann wrote: As no one asked yet, is it possible to get ospf related debug from the not working router and the DR router of the subnet? Good point, thank you. After looking at debug OSPF hello, we have found out that Hello packets with src=10.65.127.246 are being sent by the device Nov 27 05:29:35.923: OSPF: Send hello to 224.0.0.5 area 0 on Vlan333 from 10.65.127.246 but they are not Rcv-ed by other devices on the network segment. Hello packets with any other src IP address, e.g. 10.65.127.243, are being received as expected. We are at a loss what could be blocking packets with this particular src addr. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
