Hi Remi,
Just tried with 1 FW and static route, yes it's still reproducible.
The previous e-mail outputs were all related to my lab, when you see
IP's like 192.168.98.x, it's lab.
Regarding host1/2 interface configs:
Host1:
host1# more
/etc/hostname.em0
up
inet 192.168.98.202 255.255.255.0 NONE
!route add default 192.168.98.1
host1# more
/etc/hostname.carp1
up
inet 10.10.10.10 255.255.255.255 NONE vhid 1 pass 4k5EKt2svYNzSf carpdev
em0 advskew 0
host1#
Host2:
host2# more
/etc/hostname.em0
up
inet 192.168.98.203 255.255.255.0 NONE
!route add default 192.168.98.1
host2# more
/etc/hostname.carp1
up
inet 10.10.10.10 255.255.255.255 NONE vhid 1 pass 4k5EKt2svYNzSf carpdev
em0 advskew 128
host2#
The interface I kill/restore is the em0. I do it on VMware VSphere by
disabling it for Host1.
So, the steps to reproduce with one FW and static route are the same,
but I'll detail them again.
Starting and working condition:
fw1# uptime && ospfctl show rib
12:10PM up 15 days, 1:40, 1 user, load averages: 0.00, 0.00, 0.00
Destination Nexthop Path Type Type Cost
Uptime
10.10.10.10/32 192.168.98.202 Intra-Area Network 20
00:01:03
192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
00:01:08
fw1# uptime && route -n show | grep 10.10.10
12:11PM up 15 days, 1:41, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32 192.168.98.202 UG 0 0 - 32
em0
10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
em0
fw1#
Route is set to Host1 without MPATH flag.
Now I kill em0 in VSphere for Host1:
fw1#
fw1# uptime && ospfctl show rib
12:13PM up 15 days, 1:43, 1 user, load averages: 0.00, 0.00, 0.00
Destination Nexthop Path Type Type Cost
Uptime
10.10.10.10/32 192.168.98.202 Intra-Area Network 20
00:03:57
10.10.10.10/32 192.168.98.203 Intra-Area Network 20
00:00:23
192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
00:04:02
fw1# uptime && route -n show | grep 10.10.10
12:13PM up 15 days, 1:43, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32 192.168.98.202 UGP 0 6 - 32
em0
10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
em0
10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
em0
fw1#
We are working with standard OSPF timers in LAB, so the dead route stays
for 40s and then is removed.
The MPATH flag appears because this was true for 40s, although the route
for host1 results now in a blackhole, we had temporarily 2 routes with
exact metrics, only with different nexthop.
The old route is then cleared, but MPATH flag stays on the new route.
fw1# uptime && ospfctl show rib
12:14PM up 15 days, 1:43, 1 user, load averages: 0.00, 0.00, 0.00
Destination Nexthop Path Type Type Cost
Uptime
10.10.10.10/32 192.168.98.203 Intra-Area Network 20
00:00:43
192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
00:04:22
fw1# uptime && route -n show | grep 10.10.10
12:14PM up 15 days, 1:44, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
em0
10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
em0
fw1#
fw1#
But at this stage we are still with the correct nexthop in the FIB
We now bring up the Host1 em0 in VSphere:
fw1# uptime && ospfctl show rib
12:22PM up 15 days, 1:52, 1 user, load averages: 0.00, 0.00, 0.00
Destination Nexthop Path Type Type Cost
Uptime
10.10.10.10/32 192.168.98.202 Intra-Area Network 20
00:00:03
192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
00:12:28
fw1# uptime && route -n show | grep 10.10.10
12:22PM up 15 days, 1:52, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
em0
10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
em0
fw1#
fw1#
You receive the route from Host1, but is not installed on FIB.
fw1# uptime && ospfctl show rib
1:39PM up 15 days, 3:09, 1 user, load averages: 0.00, 0.00, 0.00
Destination Nexthop Path Type Type Cost
Uptime
10.10.10.10/32 192.168.98.202 Intra-Area Network 20
01:17:11
192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
01:29:36
fw1# uptime && route -n show | grep 10.10.10
1:39PM up 15 days, 3:09, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32 192.168.98.203 UGP 0 7 - 32
em0
10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
em0
fw1#
This is still true after more then 1h.
We now reload the FIB:
fw1# ospfctl fib reload
reload request sent.
fw1#
fw1# uptime && ospfctl show rib
1:40PM up 15 days, 3:10, 1 user, load averages: 0.00, 0.00, 0.00
Destination Nexthop Path Type Type Cost
Uptime
192.168.98.0/24 0.0.0.0 C Intra-Area Network 10
00:00:02
fw1# uptime && route -n show | grep 10.10.10
1:40PM up 15 days, 3:10, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
em0
fw1#
fw1#
fw1# uptime && ospfctl show rib
1:41PM up 15 days, 3:11, 1 user, load averages: 0.00, 0.00, 0.00
Destination Nexthop Path Type Type Cost
Uptime
10.10.10.10/32 192.168.98.202 Intra-Area Network 20
00:00:19
192.168.98.0/24 192.168.98.200 C Intra-Area Network 10
00:00:24
fw1# uptime && route -n show | grep 10.10.10
1:41PM up 15 days, 3:11, 1 user, load averages: 0.00, 0.00, 0.00
10.10.10.10/32 192.168.98.202 UG 0 0 - 32
em0
10.10.10.10/32 192.168.98.204 UGS 0 0 - 40
em0
fw1#
And we get to the good state condition.
Hope this helps.
Best regards,
João
On 05.11.19 23:19, Remi Locherer wrote:
> On Thu, Oct 24, 2019 at 02:09:09PM +0200, Joao Alves wrote:
>> Hi Remi,
>>
>> I've installed a lab with OpenBSD6.6 VM's to see if would happen in the
>> newer version.
>>
>> I was able to reproduce it again, but in slightly different manner.
>>
>> First of all, you need to have BGP running in FW's also, and have the
>> same route received through BGP, otherwise the issue is not
>> reproducible, because the MPATH flag will behave well with OSPF only.
>> Without the MPATH issue you can't reproduce the rest.
> Can you also reproduce without BGP but with a static route added like this:
> route add -net 10.10.10.10 -netmask 255.255.255.255 192.168.98.204 -priority
> 40
>
>> So LAB setup(all openbsd6.6 VM's):
>>
>> 2x fw/router
> Can you also reproduce with only 1 fw or are both fw nodes needed to
> reproduce?
>
>> 2x host
>> 1x bgp router
>>
>> fw1:192.168.98.200
>>
>> fw2:192.168.98.201
>>
>> host1:192.168.98.202
>>
>> host2:192.168.98.203
>>
>> bgp:192.168.98.204
>>
>>
>> In the hosts I run carp with VIP 10.10.10.10/32, carp configured with
>> preempt in kernel.
>>
>> ospf config for host1/2 is (only router id change):
>>
>> host1# more
>> /etc/ospfd.conf
>>
>>
>>
>>
>> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>>
>> # macros
>> id="192.168.98.202"
>>
>> # global configuration
>> router-id $id
>> # fib-update no
>> # stub router no
>> # spf-delay 1
>> # spf-holdtime 5
>>
>> # auth-key secret
>> # auth-type simple
>> # hello-interval 10
>> metric 10
>> # retransmit-interval 5
>> # router-dead-time 40
>> router-priority 0
>> # transmit-delay 1
>>
>> # rtlabel "DMZ" external-tag 1
>>
>> # areas
>> area 0.0.0.0 {
>> interface em0 {
>> auth-type simple
>> auth-key secret
>> }
>>
>> interface carp1 {
>> passive
>> }
>> }
>> host1#
> Please show the configs of em0 and carp1.
>
>> For FW1/2 is(router ID and router priority change, in FW2 priority is
>> 10, so BDR):
>>
>> fw1# more
>> /etc/ospfd.conf
>>
>>
>>
>>
>> # $OpenBSD: ospfd.conf,v 1.2 2018/08/07 07:06:20 claudio Exp $
>>
>> # macros
>> id="192.168.98.200"
>>
>> # global configuration
>> router-id $id
>> # fib-update no
>> # stub router no
>> # spf-delay 1
>> # spf-holdtime 5
>>
>> # auth-key secret
>> # auth-type simple
>> # hello-interval 10
>> metric 10
>> # retransmit-interval 5
>> # router-dead-time 40
>> router-priority 100
>> # transmit-delay 1
>>
>> # rtlabel "DMZ" external-tag 1
>>
>> # areas
>> area 0.0.0.0 {
>> interface em0 {
>> auth-type simple
>> auth-key secret
>> }
>> }
>> fw1#
>>
>> For BGPD configs:
>>
>> FW1/2:
>>
>>
>> fw1# more
>> /etc/bgpd.conf
>>
>>
>>
>>
>> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
>> # example bgpd configuration file, see bgpd.conf(5)
>>
>> # define our own ASN as a macro
>> ASN="65123"
>>
>> # global configuration
>> AS $ASN
>> router-id 192.168.98.200
>>
>> # list of networks that may be originated by our ASN
>> prefix-set mynetworks { \
>> 192.0.6.0/24 \
>> 2001:db8:abef::/48 \
>> }
>>
>> # define bogon prefixes which should not be part of the DFZ
>> prefix-set bogons {
>> 0.0.0.0/8 or-longer # 'this' network [RFC1122]
>> 10.0.0.0/8 or-longer # private space [RFC1918]
>> 100.64.0.0/10 or-longer # CGN Shared [RFC6598]
>> 127.0.0.0/8 or-longer # localhost [RFC1122]
>> 169.254.0.0/16 or-longer # link local [RFC3927]
>> 172.16.0.0/12 or-longer # private space [RFC1918]
>> 192.0.2.0/24 or-longer # TEST-NET-1 [RFC5737]
>> 192.88.99.0/24 or-longer # 6to4 anycast relay [RFC7526]
>> 192.168.0.0/16 or-longer # private space [RFC1918]
>> 198.18.0.0/15 or-longer # benchmarking [RFC2544]
>> 198.51.100.0/24 or-longer # TEST-NET-2 [RFC5737]
>> 203.0.113.0/24 or-longer # TEST-NET-3 [RFC5737]
>> 224.0.0.0/4 or-longer # multicast
>> 240.0.0.0/4 or-longer # reserved for future use
>> ::/8 or-longer # RFC 4291 IPv4-compatible,
>> loopback, et al
>> 0100::/64 or-longer # Discard-Only [RFC6666]
>> 2001:2::/48 or-longer # BMWG [RFC5180]
>> 2001:10::/28 or-longer # ORCHID [RFC4843]
>> 2001:db8::/32 or-longer # docu range [RFC3849]
>> 2002::/16 or-longer # 6to4 anycast relay [RFC7526]
>> 3ffe::/16 or-longer # old 6bone
>> fc00::/7 or-longer # unique local unicast
>> fe80::/10 or-longer # link local unicast
>> fec0::/10 or-longer # old site local unicast
>> ff00::/8 or-longer # multicast
>> }
>>
>> # Generate routes for the networks our ASN will originate.
>> # The communities (read 'tags') are later used to match on what
>> # is announced to EBGP neighbors
>> network prefix-set mynetworks set large-community $ASN:1:1
>>
>> # assume simple network with 3 routers in IBGP full mesh
>> group "ibgp mesh v4" {
>> remote-as $ASN
>> # use loopback for IBGP sessions, assume its distributed in OSPF
>> local-address 192.168.98.200
>> neighbor 192.168.98.204 # router 2 ipv4
>> # neighbor 192.168.98.201 # router 3 ipv4
>> }
>> # define the IPv6 IBGP sessions
>> group "ibgp mesh v6" {
>> remote-as $ASN
>> local-address 2001:db8:abcd::1
>> neighbor 2001:db8:abcd::2 # router 2 ipv6
>> neighbor 2001:db8:abcd::3 # router 3 ipv6
>> }
>>
>> # upstream providers
>> group "upstreams" {
>> neighbor 203.0.113.1 {
>> remote-as 65002
>> descr "IPv4 Transit Provider A"
>> }
>> neighbor 198.51.100.0 {
>> remote-as 65123
>> descr "IPv4 Transit provider B"
>> }
>> neighbor 2001:db8:666::2 {
>> remote-as 65123
>> descr "IPv6 Transit provider B"
>> }
>> }
>>
>> ## rules section
>>
>> # uncomment the following two lines to accept a default route from upstreams
>> #allow from group upstreams prefix 0.0.0.0/0
>> #allow from group upstreams prefix ::/0
>>
>> ### for simple BGP setups, no editing below this line is required ###
>>
>> # Outbound EBGP: only allow self originated networks to ebgp peers
>> # Don't leak any routes from upstream or peering sessions. This is done
>> # by checking for routes that are tagged with the large-community $ASN:1:1
>> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>>
>> # deny more-specifics of our own originated prefixes
>> deny quick from ebgp prefix-set mynetworks or-longer
>>
>> # IBGP: allow all updates to and from our IBGP neighbors
>> allow from ibgp
>> allow to ibgp
>>
>> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
>> # https://tools.ietf.org/html/rfc7454#section-11
>> match from ebgp set { community delete $ASN:* }
>> match from ebgp set { large-community delete $ASN:*:* }
>>
>> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
>> # and longer than 48 or shorter than 16 bits for IPv6.
>> allow from any inet prefixlen 8 - 24
>> allow from any inet6 prefixlen 16 - 48
>>
>> # Honor requests to gracefully shutdown BGP sessions
>> # https://tools.ietf.org/html/rfc8326
>> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>>
>> #deny quick from any prefix-set bogons
>>
>> # filter bogon AS numbers
>> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
>> # missconfiguration. Additionally Private or Reserved ASNs have no place in
>> # the public DFZ.
>> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
>> #deny quick from any AS 23456
>> #deny quick from any AS 64496 - 131071
>> #deny quick from any AS 4200000000 - 4294967295
>>
>> # filter out too long paths
>> deny from any max-as-len 100
>> fw1#
>>
>>
>> For BGP router:
>>
>>
>> bgp1# more /etc/bgpd.conf
>> # $OpenBSD: bgpd.conf,v 1.15 2018/11/17 17:22:38 deraadt Exp $
>> # example bgpd configuration file, see bgpd.conf(5)
>>
>> # define our own ASN as a macro
>> ASN="65123"
>>
>> # global configuration
>> AS $ASN
>> router-id 192.168.98.204
>>
>> # list of networks that may be originated by our ASN
>> prefix-set mynetworks { \
>> 192.0.2.0/24 \
>> 2001:db8:abcd::/48 \
>> 10.10.10.10/32 \
>> }
>>
>> # define bogon prefixes which should not be part of the DFZ
>> prefix-set bogons {
>> 0.0.0.0/8 or-longer # 'this' network [RFC1122]
>> 10.0.0.0/8 or-longer # private space [RFC1918]
>> 100.64.0.0/10 or-longer # CGN Shared [RFC6598]
>> 127.0.0.0/8 or-longer # localhost [RFC1122]
>> 169.254.0.0/16 or-longer # link local [RFC3927]
>> 172.16.0.0/12 or-longer # private space [RFC1918]
>> 192.0.2.0/24 or-longer # TEST-NET-1 [RFC5737]
>> 192.88.99.0/24 or-longer # 6to4 anycast relay [RFC7526]
>> 192.168.0.0/16 or-longer # private space [RFC1918]
>> 198.18.0.0/15 or-longer # benchmarking [RFC2544]
>> 198.51.100.0/24 or-longer # TEST-NET-2 [RFC5737]
>> 203.0.113.0/24 or-longer # TEST-NET-3 [RFC5737]
>> 224.0.0.0/4 or-longer # multicast
>> 240.0.0.0/4 or-longer # reserved for future use
>> ::/8 or-longer # RFC 4291 IPv4-compatible,
>> loopback, et al
>> 0100::/64 or-longer # Discard-Only [RFC6666]
>> 2001:2::/48 or-longer # BMWG [RFC5180]
>> 2001:10::/28 or-longer # ORCHID [RFC4843]
>> 2001:db8::/32 or-longer # docu range [RFC3849]
>> 2002::/16 or-longer # 6to4 anycast relay [RFC7526]
>> 3ffe::/16 or-longer # old 6bone
>> fc00::/7 or-longer # unique local unicast
>> fe80::/10 or-longer # link local unicast
>> fec0::/10 or-longer # old site local unicast
>> ff00::/8 or-longer # multicast
>> }
>>
>> # Generate routes for the networks our ASN will originate.
>> # The communities (read 'tags') are later used to match on what
>> # is announced to EBGP neighbors
>> network prefix-set mynetworks set large-community $ASN:1:1
>>
>> # assume simple network with 3 routers in IBGP full mesh
>> group "ibgp mesh v4" {
>> remote-as $ASN
>> # use loopback for IBGP sessions, assume its distributed in OSPF
>> local-address 192.168.98.204
>> neighbor 192.168.98.200 # router 2 ipv4
>> neighbor 192.168.98.201 # router 3 ipv4
>> }
>> # define the IPv6 IBGP sessions
>> group "ibgp mesh v6" {
>> remote-as $ASN
>> local-address 2001:db8:abcd::1
>> neighbor 2001:db8:abcd::2 # router 2 ipv6
>> neighbor 2001:db8:abcd::3 # router 3 ipv6
>> }
>>
>> # upstream providers
>> group "upstreams" {
>> neighbor 203.0.113.1 {
>> remote-as 65002
>> descr "IPv4 Transit Provider A"
>> }
>> neighbor 198.51.100.0 {
>> remote-as 65123
>> descr "IPv4 Transit provider B"
>> }
>> neighbor 2001:db8:666::2 {
>> remote-as 65123
>> descr "IPv6 Transit provider B"
>> }
>> }
>>
>> ## rules section
>>
>> # uncomment the following two lines to accept a default route from upstreams
>> #allow from group upstreams prefix 0.0.0.0/0
>> #allow from group upstreams prefix ::/0
>>
>> ### for simple BGP setups, no editing below this line is required ###
>>
>> # Outbound EBGP: only allow self originated networks to ebgp peers
>> # Don't leak any routes from upstream or peering sessions. This is done
>> # by checking for routes that are tagged with the large-community $ASN:1:1
>> allow to ebgp prefix-set mynetworks large-community $ASN:1:1
>>
>> # deny more-specifics of our own originated prefixes
>> deny quick from ebgp prefix-set mynetworks or-longer
>>
>> # IBGP: allow all updates to and from our IBGP neighbors
>> allow from ibgp
>> allow to ibgp
>>
>> # Scrub normal and large communities relevant to our ASN from EBGP neighbors
>> # https://tools.ietf.org/html/rfc7454#section-11
>> match from ebgp set { community delete $ASN:* }
>> match from ebgp set { large-community delete $ASN:*:* }
>>
>> # filter out prefixes longer than 24 or shorter than 8 bits for IPv4
>> # and longer than 48 or shorter than 16 bits for IPv6.
>> allow from any inet prefixlen 8 - 24
>> allow from any inet6 prefixlen 16 - 48
>>
>> # Honor requests to gracefully shutdown BGP sessions
>> # https://tools.ietf.org/html/rfc8326
>> #match from any community GRACEFUL_SHUTDOWN set { localpref 0 }
>>
>> #deny quick from any prefix-set bogons
>>
>> # filter bogon AS numbers
>> # AS_TRANS (23456) is not supposed to show up in any path and indicates a
>> # missconfiguration. Additionally Private or Reserved ASNs have no place in
>> # the public DFZ.
>> http://www.iana.org/assignments/as-numbers/as-numbers.xhtml
>> #deny quick from any AS 23456
>> #deny quick from any AS 64496 - 131071
>> #deny quick from any AS 4200000000 - 4294967295
>>
>> # filter out too long paths
>> deny from any max-as-len 100
>> bgp1#
>>
>>
>>
>> Reproduce the issue:
>>
>>
>> The problem in this case is faced in the recovery of the failed host
>> instead.
>>
>> start condition:
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UG 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1#
>>
>> We now force the MPATH flag by temporarly advertise the network from
>> host 2. For this we do "ifconfig carp1 state master" on the backup host2:
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UG 0 3 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UG 0 3 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
>> em0
>> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
>> em0
>> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1#
>>
>>
>> We now have the MPATH flag active, even having only one OSPF route.
>>
>>
>> So, we now kill the interface of host1:
> Which interface? How do you kill it?
>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 3 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 6 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
>> 00:58:16
>> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
>> 00:00:07
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 9 - 32
>> em0
>> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
>> 00:58:23
>> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
>> 00:00:14
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 9 - 32
>> em0
>> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 9 - 32
>> em0
>> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UGP 0 9 - 32
>> em0
>> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
>> 00:00:24
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1#
>>
>>
>> In this output we see that temporarily we have both OSPF routes, and
>> then the first one timesout living the one related to host2.
>> This behaviour is different from what we have in production, so in this
>> case, at this stage we still have service because route ended up with
>> correct next-hop for host2.
>>
>> But again, MPATH flag was kept active when it shouldn't.
>>
>> so current status is:
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.203 Intra-Area Network 20
>> 00:06:11
>> fw1#
>>
>>
>> Now we raise interface on host1 and get service down:
> Again: which interface and how?
>
>>
>>
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.203 Intra-Area Network 65545
>> 00:07:19
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.203 Intra-Area Network 65545
>> 00:07:24
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
>> 00:00:04
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
>> 00:00:10
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
>> 00:00:15
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl show rib | grep 10.10.10
>> 10.10.10.10/32 192.168.98.202 Intra-Area Network 20
>> 00:00:19
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 6 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1#
> This output (and following similar ones) is a bit hard to read. I'd prefere
> to see the output for the 1st state and then the output for the 2nd state.
> Maybe with a comment about the elapsed time in between.
>
>>
>>
>> Notice that the FIB next-hop wasn't updated to 192.168.98.202.
>>
>>
>>
>> Now the fix (commands issued repeatedly):
>>
>>
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UGP 0 9 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# ospfctl fib
>> reload
>>
>>
>>
>>
>> reload request sent.
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UG 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.203 UG 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UG 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1# route -n show | grep 10.10.10.10
>> 10.10.10.10/32 192.168.98.202 UG 0 0 - 32
>> em0
>> 10.10.10.10/32 192.168.98.204 UG 0 0 - 48
>> em0
>> fw1#
>>
>>
>> We still see temporarily the wrong next-hop, but it then converges to
>> the correct one, and we now don't have the MPATH flag anymore.
> Is this about your prod or your lab setup?
>
>>
>> Hope this helps in your lab setup, as my real scenario is a bit
>> different behaviour, prbably because I'm using quagga to advertise the
>> prefix with network command, and the prefix is configured on same
>> physical interface.
>>
>> In this lab scenario, the interface with the VIP is diferent from the
>> main OSPF interface, and because of that I was able to announce it with
>> passive command.
> In ospfd carp interfaces are always announced as passive.
>
>> For the quagga's, there are constantly hello's being sent for the
>> prefix, even then no adjacency is formed on that prefix subnet.
>> In this LAB scenario you see updates being trade about the prefix.
>>
>> This is the captures in the lab environment when I issue the ospfctl fib
>> reload (captures being taken on fw2):
>>
>>
>> 13:46:36.739839 192.168.98.200 > 224.0.0.5: OSPFv2-hello 44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 34942, len 64)
>> 13:46:36.740057 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 20542, len 76)
>> 13:46:36.740182 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 47644, len 76)
>> 13:46:36.740269 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30044, len 76)
>> 13:46:36.740550 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 100: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
>> 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs 192.168.98.202
>> 192.168.98.203 192.168.98.201 } { E S 8000000D age 1 rtr 192.168.98.201
>> } [|ospf] [tos 0xc0] [ttl 1] (id 26274, len 120)
>> 13:46:36.741005 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 59174, len 76)
>> 13:46:36.741692 192.168.98.200 > 224.0.0.5: OSPFv2-hello 44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 48099, len 64)
>> 13:46:38.743078 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
>> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 55607, len 84)
>> 13:46:38.744309 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000001 age 1 net dr
>> 192.168.98.201 if 192.168.98.201 } { E S 8000000D age 1 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 31253, len 84)
>> 13:46:41.752541 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd 76: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000021 age 6 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 9379, len 96)
>> 13:46:41.752841 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 76: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 52657, len 96)
>> 13:46:41.753134 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd 76: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000017 age 6 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 6608, len 96)
>> 13:46:41.753335 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 76: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 22390, len 96)
>> 13:46:43.747722 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd 76:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.201 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 29312, len 96)
>> 13:46:43.747795 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd 76:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.201 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 42301, len 96)
>> 13:46:43.748098 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 9 rtr
>> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 56865, len 64)
>> 13:46:43.748136 192.168.98.202 > 192.168.98.201: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 9 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 43582, len 64)
>> 13:46:43.762045 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 44: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000017 age 7 rtr
>> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 1486, len 64)
>> 13:46:43.763373 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 44: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000021 age 7 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 2479, len 64)
>> 13:46:46.742883 192.168.98.200 > 224.0.0.5: OSPFv2-hello 44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 nbrs [tos 0xc0] [ttl 1] (id 41604, len 64)
>> 13:46:46.743098 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 732, len 76)
>> 13:46:46.743099 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.201 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 28204, len 76)
>> 13:46:46.743141 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.201 nbrs 192.168.98.200 192.168.98.202
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 35443, len 76)
>> 13:46:46.744985 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 nbrs 192.168.98.203 192.168.98.201
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 33192, len 76)
>> 13:46:46.745048 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222A
>> [tos 0xc0] [ttl 1] (id 6154, len 52)
>> 13:46:46.745344 192.168.98.201 > 192.168.98.200: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF222B
>> [tos 0xc0] [ttl 1] (id 38910, len 52)
>> 13:46:46.745345 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17052, len 76)
>> 13:46:46.745399 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.203 [tos 0xc0] [ttl 1] (id 30271, len 76)
>> 13:46:46.745399 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 nbrs 192.168.98.201 192.168.98.200
>> 192.168.98.202 [tos 0xc0] [ttl 1] (id 49111, len 76)
>> 13:46:46.745530 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 100: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> { E S 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 }
>> [|ospf] [tos 0xc0] [ttl 1] (id 42835, len 120)
>> 13:46:46.745579 192.168.98.203 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
>> [tos 0xc0] [ttl 1] (id 39759, len 52)
>> 13:46:46.745580 192.168.98.202 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
>> [tos 0xc0] [ttl 1] (id 15662, len 52)
>> 13:46:46.745654 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 41333, len 76)
>> 13:46:46.745657 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 35399, len 76)
>> 13:46:46.745657 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 5018, len 76)
>> 13:46:46.745783 192.168.98.201 > 192.168.98.203: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
>> [tos 0xc0] [ttl 1] (id 19213, len 52)
>> 13:46:46.745930 192.168.98.200 > 192.168.98.201: OSPFv2-dd 132: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF222B { E S
>> 80000013 age 10 rtr 192.168.98.200 } { E S 8000000C age 13:17 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9643, len 152)
>> 13:46:46.745954 192.168.98.201 > 192.168.98.202: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
>> [tos 0xc0] [ttl 1] (id 48123, len 52)
>> 13:46:46.746257 192.168.98.201 > 192.168.98.200: OSPFv2-dd 132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF222C { E
>> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 60171, len 152)
>> 13:46:46.746524 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222C [tos
>> 0xc0] [ttl 1] (id 20148, len 52)
>> 13:46:46.746593 192.168.98.201 > 192.168.98.200: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF222D [tos
>> 0xc0] [ttl 1] (id 29634, len 52)
>> 13:46:46.746857 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF222D [tos
>> 0xc0] [ttl 1] (id 15253, len 52)
>> 13:46:46.746858 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req 60:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 } {
>> rtr 192.168.98.202 } { rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 8075,
>> len 80)
>> 13:46:46.747077 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req 48:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
>> net dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 29745,
>> len 68)
>> 13:46:46.747152 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd 160:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> { E S 80000017 age 12 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl 1]
>> (id 21137, len 180)
>> 13:46:46.747581 192.168.98.200 > 192.168.98.201: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000017 age 12
>> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 7102, len 64)
>> 13:46:46.747666 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 112: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } { E [|ospf]
>> [tos 0xc0] [ttl 1] (id 40964, len 132)
>> 13:46:46.747696 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BFEEF2
>> [tos 0xc0] [ttl 1] (id 16937, len 52)
>> 13:46:46.747813 192.168.98.201 > 192.168.98.200: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 99BF5FD9
>> [tos 0xc0] [ttl 1] (id 3855, len 52)
>> 13:46:46.747848 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 96: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 1 rtr
>> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
>> { E S 8000000C age 1 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
>> [tos 0xc0] [ttl 1] (id 45786, len 116)
>> 13:46:46.748155 192.168.98.200 > 192.168.98.201: OSPFv2-dd 152: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E M mtu 1500 S 99BF5FD9 { E S
>> 80000014 age 0 rtr 192.168.98.200 } { E S 8000000D age 12 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24835, len 172)
>> 13:46:46.748551 192.168.98.201 > 192.168.98.200: OSPFv2-dd 132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M/MS mtu 1500 S 99BF5FDA { E
>> S 80000012 age 13:21 rtr 192.168.98.200 } { E S 8000000E age 0 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 47662, len 152)
>> 13:46:46.748693 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDA [tos
>> 0xc0] [ttl 1] (id 21631, len 52)
>> 13:46:46.748799 192.168.98.201 > 192.168.98.200: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E MS mtu 1500 S 99BF5FDB [tos
>> 0xc0] [ttl 1] (id 48058, len 52)
>> 13:46:46.748938 192.168.98.200 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mtu 1500 S 99BF5FDB [tos
>> 0xc0] [ttl 1] (id 16434, len 52)
>> 13:46:46.748962 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req 36:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 28004, len 56)
>> 13:46:46.749045 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req 60:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
>> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33841, len 80)
>> 13:46:46.749209 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd 64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 1 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> [tos 0xc0] [ttl 1] (id 5235, len 84)
>> 13:46:48.752528 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack 104: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000D age 12 rtr
>> 192.168.98.201 } { E S 80000017 age 13 rtr 192.168.98.202 } { E S
>> 80000022 age 1 rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 50978, len 124)
>> 13:46:48.752670 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack 64: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
>> 192.168.98.203 } { E S 80000001 age 13 net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 17804, len 84)
>> 13:46:48.752696 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 44: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000022 age 2 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 60705, len 64)
>> 13:46:48.752887 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000013 age 11 rtr
>> 192.168.98.200 } { E S 8000000B age 11 net dr 192.168.98.200 if
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 30728, len 84)
>> 13:46:51.756275 192.168.98.201 > 192.168.98.203: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DF7C38
>> [tos 0xc0] [ttl 1] (id 13078, len 52)
>> 13:46:51.756285 192.168.98.201 > 192.168.98.202: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 3610C5FF
>> [tos 0xc0] [ttl 1] (id 53410, len 52)
>> 13:46:51.756310 192.168.98.201 > 192.168.98.200: OSPFv2-ls_req 60:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.200 } {
>> net dr 192.168.98.200 if 192.168.98.200 } { net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 33659, len 80)
>> 13:46:51.756826 192.168.98.202 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E I/M/MS mtu 1500 S 361091BE
>> [tos 0xc0] [ttl 1] (id 64021, len 52)
>> 13:46:51.756827 192.168.98.203 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E I/M/MS mtu 1500 S E1DEE91C
>> [tos 0xc0] [ttl 1] (id 12508, len 52)
>> 13:46:51.756827 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd 96:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
>> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
>> [tos 0xc0] [ttl 1] (id 13626, len 116)
>> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_req 36:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 41975, len 56)
>> 13:46:51.756828 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd 132:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 { dr 192.168.98.200 if 192.168.98.200 tos 0 metric 10 } }
>> { E S 8000000C age 6 net dr 192.168.98.200 if 192.168.98.200 } [|ospf]
>> [tos 0xc0] [ttl 1] (id 52527, len 152)
>> 13:46:51.757543 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 76: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 7 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 2414, len 96)
>> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 49941, len 64)
>> 13:46:51.757553 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000C age 6 net
>> dr 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 54649, len 64)
>> 13:46:51.757554 192.168.98.201 > 192.168.98.202: OSPFv2-dd 132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S 361091BE { E S
>> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 35843, len 152)
>> 13:46:51.757555 192.168.98.201 > 192.168.98.200: OSPFv2-ls_upd 64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 { net 192.168.98.0 mask 255.255.255.0 tos 0 metric 10 } }
>> [tos 0xc0] [ttl 1] (id 14024, len 84)
>> 13:46:51.757556 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 64: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
>> dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
>> 192.168.98.202 192.168.98.203 192.168.98.201 } [tos 0xc0] [ttl 1] (id
>> 22555, len 84)
>> 13:46:51.757583 192.168.98.201 > 192.168.98.203: OSPFv2-dd 132: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E M mtu 1500 S E1DEE91C { E S
>> 80000012 age 13:26 rtr 192.168.98.200 } { E S 8000000E age 5 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 33297, len 152)
>> 13:46:51.757954 192.168.98.202 > 192.168.98.201: OSPFv2-dd 152: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E M/MS mtu 1500 S 361091BF { E
>> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 9420, len 172)
>> 13:46:51.758049 192.168.98.201 > 192.168.98.202: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091BF [tos
>> 0xc0] [ttl 1] (id 26856, len 52)
>> 13:46:51.758204 192.168.98.202 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E MS mtu 1500 S 361091C0 [tos
>> 0xc0] [ttl 1] (id 29314, len 52)
>> 13:46:51.758286 192.168.98.203 > 192.168.98.201: OSPFv2-dd 152: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E M/MS mtu 1500 S E1DEE91D { E
>> S 80000014 age 6 rtr 192.168.98.200 } { E S 8000000D age 16 rtr
>> 192.168.98.201 } { E [tos 0xc0] [ttl 1] (id 24179, len 172)
>> 13:46:51.758304 192.168.98.201 > 192.168.98.202: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S 361091C0 [tos
>> 0xc0] [ttl 1] (id 18366, len 52)
>> 13:46:51.758353 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req 36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
>> [tos 0xc0] [ttl 1] (id 39434, len 56)
>> 13:46:51.758392 192.168.98.201 > 192.168.98.203: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91D [tos
>> 0xc0] [ttl 1] (id 19583, len 52)
>> 13:46:51.758574 192.168.98.203 > 192.168.98.201: OSPFv2-dd 32: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E MS mtu 1500 S E1DEE91E [tos
>> 0xc0] [ttl 1] (id 46477, len 52)
>> 13:46:51.758655 192.168.98.201 > 192.168.98.203: OSPFv2-dd 32: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mtu 1500 S E1DEE91E [tos
>> 0xc0] [ttl 1] (id 10148, len 52)
>> 13:46:51.758695 192.168.98.201 > 192.168.98.203: OSPFv2-ls_req 36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.203 }
>> [tos 0xc0] [ttl 1] (id 20293, len 56)
>> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req 36:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 8976, len 56)
>> 13:46:51.758725 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd 76:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000018 age 1 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 38525, len 96)
>> 13:46:51.758990 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req 36:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 3058, len 56)
>> 13:46:51.759005 192.168.98.203 > 192.168.98.201: OSPFv2-ls_upd 76:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 1 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 6586, len 96)
>> 13:46:53.760281 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 84: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
>> 192.168.98.200 } { E S 80000018 age 7 rtr 192.168.98.202 } [tos 0xc0]
>> [ttl 1] (id 44630, len 104)
>> 13:46:53.760438 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack 104: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
>> 192.168.98.200 } { E S 80000001 age 18 net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 60174, len 124)
>> 13:46:53.760534 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack 104: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000018 age 6 rtr
>> 192.168.98.202 } { E S 8000000E age 6 rtr 192.168.98.201 } { E S
>> 80000001 age 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos
>> 0xc0] [ttl 1] (id 59994, len 124)
>> 13:46:53.760642 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000014 age 6 rtr
>> 192.168.98.200 } { E S 8000000C age 6 net dr 192.168.98.200 if
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 11941, len 84)
>> 13:46:56.751086 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 17219, len 76)
>> 13:46:56.751111 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54762, len 76)
>> 13:46:56.751228 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17270, len 76)
>> 13:46:56.751383 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 10624, len 76)
>> 13:46:56.765328 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd 64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 29796, len 84)
>> 13:46:56.765380 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd 64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 5472, len 84)
>> 13:46:56.765418 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req 36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
>> [tos 0xc0] [ttl 1] (id 56664, len 56)
>> 13:46:56.765519 192.168.98.203 > 192.168.98.201: OSPFv2-ls_req 36:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 20557, len 56)
>> 13:46:56.765519 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd 76: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 55930, len 96)
>> 13:46:56.765597 192.168.98.202 > 192.168.98.201: OSPFv2-ls_req 36:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { rtr 192.168.98.201 }
>> [tos 0xc0] [ttl 1] (id 60479, len 56)
>> 13:46:56.765737 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 6 rtr
>> 192.168.98.203 } [tos 0xc0] [ttl 1] (id 10666, len 64)
>> 13:46:56.765796 192.168.98.201 > 224.0.0.5: OSPFv2-ls_upd 64: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 13632, len 84)
>> 13:46:56.765819 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 116: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 7 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
>> [tos 0xc0] [ttl 1] (id 18213, len 136)
>> 13:46:58.769384 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 } { E S 8000000D age 1 net dr 192.168.98.200 if
>> 192.168.98.200 } [tos 0xc0] [ttl 1] (id 16783, len 84)
>> 13:46:58.769522 192.168.98.201 > 224.0.0.5: OSPFv2-ls_ack 44: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" { E S 8000000D age 1 net dr
>> 192.168.98.200 if 192.168.98.200 } [tos 0xc0] [ttl 1] (id 18538, len 64)
>> 13:46:58.769542 192.168.98.200 > 224.0.0.5: OSPFv2-ls_ack 44: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 8000000F age 1 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 29090, len 64)
>> 13:46:58.769738 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 124: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000E age 6 rtr
>> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
>> 192.168.98.201 } { E S 80000023 age 7 rtr 192.168.98.203 } [tos 0xc0]
>> [ttl 1] (id 11688, len 144)
>> 13:47:01.770218 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd 76:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000023 age 12
>> rtr 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
>> 65535 } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } [tos
>> 0xc0] [ttl 1] (id 39965, len 96)
>> 13:47:01.770242 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd 76: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 32673, len 96)
>> 13:47:01.770449 192.168.98.201 > 192.168.98.202: OSPFv2-ls_req 36:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { rtr 192.168.98.202 }
>> [tos 0xc0] [ttl 1] (id 8167, len 56)
>> 13:47:01.770476 192.168.98.201 > 192.168.98.202: OSPFv2-ls_upd 64:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 4781, len 84)
>> 13:47:01.770490 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd 64: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000E age 12 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } [tos 0xc0] [ttl 1] (id 42500, len 84)
>> 13:47:01.770518 192.168.98.200 > 224.0.0.5: OSPFv2-ls_upd 76: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 7 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 15481, len 96)
>> 13:47:01.770628 192.168.98.201 > 192.168.98.203: OSPFv2-ls_upd 100:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
>> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 mask 255.255.255.0 rtrs
>> 192.168.98.202 192.168.98.203 192.168.98.201 } { E S 8000000F age 6 rtr
>> 192.168.98.201 } [|ospf] [tos 0xc0] [ttl 1] (id 18229, len 120)
>> 13:47:01.770739 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 12
>> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 23688, len 64)
>> 13:47:01.770839 192.168.98.202 > 192.168.98.201: OSPFv2-ls_upd 76:
>> rtrid 192.168.98.202 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
>> 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 10 }
>> { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos 0xc0]
>> [ttl 1] (id 4824, len 96)
>> 13:47:01.771012 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 6 rtr
>> 192.168.98.202 } [tos 0xc0] [ttl 1] (id 48592, len 64)
>> 13:47:01.771070 192.168.98.203 > 192.168.98.201: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 50511, len 64)
>> 13:47:03.778476 192.168.98.203 > 224.0.0.6: OSPFv2-ls_ack 104: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 80000001 age 1:00:00 net
>> dr 192.168.98.201 if 192.168.98.201 } { E S 8000000F age 7 rtr
>> 192.168.98.201 } { E S 80000001 age 1:00:00 net dr 192.168.98.201 if
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 63751, len 124)
>> 13:47:03.778985 192.168.98.202 > 224.0.0.6: OSPFv2-ls_ack 44: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 8000000F age 6 rtr
>> 192.168.98.201 } [tos 0xc0] [ttl 1] (id 34324, len 64)
>> 13:47:06.759348 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 28952, len 76)
>> 13:47:06.759349 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 25646, len 76)
>> 13:47:06.759524 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 3310, len 76)
>> 13:47:06.759532 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 23021, len 76)
>> 13:47:06.788050 192.168.98.203 > 224.0.0.6: OSPFv2-ls_upd 112: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" { E S 8000000F age 13 rtr
>> 192.168.98.201 { dr 192.168.98.200 if 192.168.98.201 tos 0 metric 65535
>> } } { E S 80000019 age 13 rtr 192.168.98.202 } [|ospf] [tos 0xc0] [ttl
>> 1] (id 12815, len 132)
>> 13:47:06.788051 192.168.98.200 > 192.168.98.201: OSPFv2-ls_upd 76:
>> rtrid 192.168.98.200 backbone auth "secret^@^@" { E S 80000019 age 12
>> rtr 192.168.98.202 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric
>> 10 } { dr 192.168.98.200 if 192.168.98.202 tos 0 metric 10 } } [tos
>> 0xc0] [ttl 1] (id 31226, len 96)
>> 13:47:06.788317 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 8000000F age 13
>> rtr 192.168.98.201 } [tos 0xc0] [ttl 1] (id 26313, len 64)
>> 13:47:06.788367 192.168.98.201 > 192.168.98.203: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 13
>> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 15304, len 64)
>> 13:47:06.788414 192.168.98.201 > 192.168.98.200: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000019 age 12
>> rtr 192.168.98.202 } [tos 0xc0] [ttl 1] (id 22491, len 64)
>> 13:47:06.789275 192.168.98.202 > 224.0.0.6: OSPFv2-ls_upd 112: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" { E S 80000023 age 18 rtr
>> 192.168.98.203 { net 10.10.10.10 mask 255.255.255.255 tos 0 metric 65535
>> } { dr 192.168.98.200 if 192.168.98.203 tos 0 metric 10 } } { E [|ospf]
>> [tos 0xc0] [ttl 1] (id 22696, len 132)
>> 13:47:06.789496 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000023 age 18
>> rtr 192.168.98.203 } [tos 0xc0] [ttl 1] (id 777, len 64)
>> 13:47:06.789497 192.168.98.201 > 192.168.98.202: OSPFv2-ls_ack 44:
>> rtrid 192.168.98.201 backbone auth "secret^@^@" { E S 80000001 age
>> 1:00:00 net dr 192.168.98.201 if 192.168.98.201 } [tos 0xc0] [ttl 1] (id
>> 26560, len 64)
>> 13:47:16.771753 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 16260, len 76)
>> 13:47:16.772049 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9947, len 76)
>> 13:47:16.772050 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 54548, len 76)
>> 13:47:16.772050 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 21814, len 76)
>> 13:47:26.774278 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 50490, len 76)
>> 13:47:26.774423 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 17263, len 76)
>> 13:47:26.774518 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 24212, len 76)
>> 13:47:26.774518 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 49902, len 76)
>> 13:47:36.778146 192.168.98.201 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.201 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 10 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.200
>> 192.168.98.202 192.168.98.203 [tos 0xc0] [ttl 1] (id 27885, len 76)
>> 13:47:36.778382 192.168.98.200 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.200 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 100 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.203
>> 192.168.98.201 192.168.98.202 [tos 0xc0] [ttl 1] (id 9857, len 76)
>> 13:47:36.778618 192.168.98.202 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.202 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.203 [tos 0xc0] [ttl 1] (id 57404, len 76)
>> 13:47:36.778618 192.168.98.203 > 224.0.0.5: OSPFv2-hello 56: rtrid
>> 192.168.98.203 backbone auth "secret^@^@" E mask 255.255.255.0 int 10
>> pri 0 dead 40 dr 192.168.98.200 bdr 192.168.98.201 nbrs 192.168.98.201
>> 192.168.98.200 192.168.98.202 [tos 0xc0] [ttl 1] (id 23172, len 76)
>> ^C
>> 3693 packets received by filter
>> 0 packets dropped by kernel
>> fw2#
>>
>>
>>
>> This post is very extensive already, let me know in case you need any
>> specific info.
>>
>> I'll keep lab running.
>>
>> Thank you.
>>
