Re: [vpp-dev] VRRP issue when using VRs in different VLANs
Hi Matt, Thanks for the fast reply. Yes, I can confirm that the patch solves this issue. Thanks again, BR/Mechthild From: Matthew Smith Sent: Friday, 18 February 2022 22:43 To: Mechthild Buescher Cc: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] VRRP issue when using VRs in different VLANs Hi Mechthild, It looks like you are running version 21.06 of VPP. This patch was merged last month and may resolve the issue - https://gerrit.fd.io/r/c/vpp/+/34815<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-4e7411000178acf4=1=18a9b9de-69e2-4627-ba7e-b4b0481d9c1f=https%3A%2F%2Fgerrit.fd.io%2Fr%2Fc%2Fvpp%2F%2B%2F34815>. Can you try applying that patch to your build? Let me know if that helps. Thanks, -Matt On Fri, Feb 18, 2022 at 3:06 PM Mechthild Buescher via lists.fd.io<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-56f4fcff43ab917f=1=18a9b9de-69e2-4627-ba7e-b4b0481d9c1f=http%3A%2F%2Flists.fd.io%2F> mailto:ericsson@lists.fd.io>> wrote: Hi all, We are using VPP on two nodes and run VRRP on them. This works fine as long as we use one VLAN. But if we configure VRs in different VLANs we see strange behavior: Every 4 seconds, the backup VRRP VR shortly changes to master and sends an advertisement message to the master and then it changes back to master. The symptom: # while true; do date; vppctl show vrrp vr | grep state; echo; sleep 1; done Fri Feb 18 20:33:15 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:16 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:17 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:18 UTC 2022 state Backup flags: preempt no accept yes unicast no state Master flags: preempt no accept yes unicast no Fri Feb 18 20:33:19 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:20 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:21 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:22 UTC 2022 state Backup flags: preempt no accept yes unicast no state Master flags: preempt no accept yes unicast no Fri Feb 18 20:33:23 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no # tcpdump -nve -r /tmp/my_vppExt0_2vrid_2vlan.pcap | grep -B1 "vrid 232” 00:02:25.805841 00:00:5e:00:01:e8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 50: vlan 102, p 0, ethertype IPv4, (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 32) 172.17.2.2 > 224.0.0.18<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-56c1a0226d9ef03f=1=18a9b9de-69e2-4627-ba7e-b4b0481d9c1f=http%3A%2F%2F224.0.0.18%2F>: vrrp 172.17.2.2 > 224.0.0.18<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-56c1a0226d9ef03f=1=18a9b9de-69e2-4627-ba7e-b4b0481d9c1f=http%3A%2F%2F224.0.0.18%2F>: VRRPv3, Advertisement, vrid 232, prio 100, intvl 100cs, length 12, addrs: 172.17.2.3 -- 00:02:26.190168 00:00:5e:00:01:e8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 60: vlan 102, p 0, ethertype IPv4, (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 32) 172.17.2.3 > 224.0.0.18<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-56c1a0226d9ef03f=1=18a9b9de-69e2-4627-ba7e-b4b0481d9c1f=http%3A%2F%2F224.0.0.18%2F>: vrrp 172.17.2.3 > 224.0.0.18<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-56c1a0226d9ef03f=1=18a9b9de-69e2-4627-ba7e-b4b0481d9c1f=http%3A%2F%2F224.0.0.18%2F>: VRRPv3, Advertisement, vrid 232, prio 200, intvl 100cs, length 12, addrs: 172.17.2.3 -- 00:02:27.194227 00:00:5e:00:01:e8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 60: vlan 102, p 0, ethertype IPv4, (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 32) 172.17.2.3 > 224.0.0.18<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-56c1a0226d9ef03f=1=18a9b9de-69e2-4627-ba7e-b4b0481d9c1f=http%3A%2F%2F224.0.0.18%2F>: vrrp 172.17.2.3 > 224.0.0.18<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-45444731-56c1a0226d9ef03f=1=18a9b9de-69e2-4627-ba7e-b4b0481d9c1f=http%3A%2F%2F224.0.0.18%2F>: VRRPv3, Advertisement, vrid 232, prio 200, intvl 100cs, length 12, addrs: 172.17.2.3 The configuration: BACKUP system: ip table add 26 ip table add 27 set interface state
Re: [vpp-dev] VRRP issue when using VRs in different VLANs
Hi Mechthild, It looks like you are running version 21.06 of VPP. This patch was merged last month and may resolve the issue - https://gerrit.fd.io/r/c/vpp/+/34815. Can you try applying that patch to your build? Let me know if that helps. Thanks, -Matt On Fri, Feb 18, 2022 at 3:06 PM Mechthild Buescher via lists.fd.io wrote: > Hi all, > > > > We are using VPP on two nodes and run VRRP on them. This works fine as > long as we use one VLAN. But if we configure VRs in different VLANs we see > strange behavior: Every 4 seconds, the backup VRRP VR shortly changes to > master and sends an advertisement message to the master and then it changes > back to master. > > The symptom: > # while true; do date; vppctl show vrrp vr | grep state; echo; sleep 1; > done > > Fri Feb 18 20:33:15 UTC 2022 > >state Backup flags: preempt no accept yes unicast no > >state Backup flags: preempt no accept yes unicast no > > > > Fri Feb 18 20:33:16 UTC 2022 > >state Backup flags: preempt no accept yes unicast no > >state Backup flags: preempt no accept yes unicast no > > > > Fri Feb 18 20:33:17 UTC 2022 > >state Backup flags: preempt no accept yes unicast no > >state Backup flags: preempt no accept yes unicast no > > > > Fri Feb 18 20:33:18 UTC 2022 > >state Backup flags: preempt no accept yes unicast no > >state Master flags: preempt no accept yes unicast no > > > > Fri Feb 18 20:33:19 UTC 2022 > >state Backup flags: preempt no accept yes unicast no > >state Backup flags: preempt no accept yes unicast no > > > > Fri Feb 18 20:33:20 UTC 2022 > >state Backup flags: preempt no accept yes unicast no > >state Backup flags: preempt no accept yes unicast no > > > > Fri Feb 18 20:33:21 UTC 2022 > >state Backup flags: preempt no accept yes unicast no > >state Backup flags: preempt no accept yes unicast no > > > > Fri Feb 18 20:33:22 UTC 2022 > >state Backup flags: preempt no accept yes unicast no > >state Master flags: preempt no accept yes unicast no > > > > Fri Feb 18 20:33:23 UTC 2022 > >state Backup flags: preempt no accept yes unicast no > >state Backup flags: preempt no accept yes unicast no > > > > # tcpdump -nve -r /tmp/my_vppExt0_2vrid_2vlan.pcap | grep -B1 "vrid 232” > > 00:02:25.805841 00:00:5e:00:01:e8 > 01:00:5e:00:00:12, ethertype 802.1Q > (0x8100), length 50: vlan 102, p 0, ethertype IPv4, (tos 0x0, ttl 255, id > 0, offset 0, flags [none], proto VRRP (112), length 32) > > 172.17.2.2 > 224.0.0.18: vrrp 172.17.2.2 > 224.0.0.18: VRRPv3, > Advertisement, vrid 232, prio 100, intvl 100cs, length 12, addrs: 172.17.2.3 > > -- > > 00:02:26.190168 00:00:5e:00:01:e8 > 01:00:5e:00:00:12, ethertype 802.1Q > (0x8100), length 60: vlan 102, p 0, ethertype IPv4, (tos 0x0, ttl 255, id > 0, offset 0, flags [none], proto VRRP (112), length 32) > > 172.17.2.3 > 224.0.0.18: vrrp 172.17.2.3 > 224.0.0.18: VRRPv3, > Advertisement, vrid 232, prio 200, intvl 100cs, length 12, addrs: 172.17.2.3 > > -- > > 00:02:27.194227 00:00:5e:00:01:e8 > 01:00:5e:00:00:12, ethertype 802.1Q > (0x8100), length 60: vlan 102, p 0, ethertype IPv4, (tos 0x0, ttl 255, id > 0, offset 0, flags [none], proto VRRP (112), length 32) > > 172.17.2.3 > 224.0.0.18: vrrp 172.17.2.3 > 224.0.0.18: VRRPv3, > Advertisement, vrid 232, prio 200, intvl 100cs, length 12, addrs: 172.17.2.3 > > > > The configuration: > BACKUP system: > ip table add 26 > > ip table add 27 > > set interface state Ext-0 up > > create sub-interfaces Ext-0 101 > > create sub-interfaces Ext-0 102 > > set interface state Ext-0.101 up > > set interface state Ext-0.102 up > > > > set interface ip table Ext-0.101 26 > > set interface ip table Ext-0.102 27 > > set interface ip address Ext-0.101 172.17.1.2/25 > > set interface ip address Ext-0.102 172.17.2.2/25 > > vrrp vr add Ext-0.101 vr_id 231 priority 100 no_preempt accept_mode > 172.17.1.3 > > vrrp vr add Ext-0.102 vr_id 232 priority 100 no_preempt accept_mode > 172.17.2.3 > > > > MASTER system: > ip table add 26 > > ip table add 27 > > set interface state Ext-0 up > > create sub-interfaces Ext-0 101 > > create sub-interfaces Ext-0 102 > > set interface state Ext-0.101 up > > set interface state Ext-0.102 up > > > > set interface ip table Ext-0.101 26 > > set interface ip table Ext-0.102 27 > > set interface ip address Ext-0.101 172.17.1.1/25 > > set interface ip address Ext-0.102 172.17.2.1/25 > > vrrp vr add Ext-0.101 vr_id 231 priority 200 no_preempt accept_mode > 172.17.1.3 > > vrrp vr add Ext-0.102 vr_id 232 priority 200 no_preempt accept_mode > 172.17.2.3 > > > > Running BACKUP config: > > # vppctl show vrrp vr > > [0] sw_if_index 2 VR ID 231 IPv4 > >state Backup flags: preempt no accept yes unicast no > >priority: configured 100 adjusted 100 > >timers: adv interval 100 master adv 100 skew 60 master down 360 > >virtual MAC 00:00:5e:00:01:e7 > >addresses 172.17.1.3 > >peer addresses > >tracked interfaces > > [1] sw_if_index 3 VR ID 232 IPv4 > >
[vpp-dev] VRRP issue when using VRs in different VLANs
Hi all, We are using VPP on two nodes and run VRRP on them. This works fine as long as we use one VLAN. But if we configure VRs in different VLANs we see strange behavior: Every 4 seconds, the backup VRRP VR shortly changes to master and sends an advertisement message to the master and then it changes back to master. The symptom: # while true; do date; vppctl show vrrp vr | grep state; echo; sleep 1; done Fri Feb 18 20:33:15 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:16 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:17 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:18 UTC 2022 state Backup flags: preempt no accept yes unicast no state Master flags: preempt no accept yes unicast no Fri Feb 18 20:33:19 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:20 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:21 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no Fri Feb 18 20:33:22 UTC 2022 state Backup flags: preempt no accept yes unicast no state Master flags: preempt no accept yes unicast no Fri Feb 18 20:33:23 UTC 2022 state Backup flags: preempt no accept yes unicast no state Backup flags: preempt no accept yes unicast no # tcpdump -nve -r /tmp/my_vppExt0_2vrid_2vlan.pcap | grep -B1 "vrid 232" 00:02:25.805841 00:00:5e:00:01:e8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 50: vlan 102, p 0, ethertype IPv4, (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 32) 172.17.2.2 > 224.0.0.18: vrrp 172.17.2.2 > 224.0.0.18: VRRPv3, Advertisement, vrid 232, prio 100, intvl 100cs, length 12, addrs: 172.17.2.3 -- 00:02:26.190168 00:00:5e:00:01:e8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 60: vlan 102, p 0, ethertype IPv4, (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 32) 172.17.2.3 > 224.0.0.18: vrrp 172.17.2.3 > 224.0.0.18: VRRPv3, Advertisement, vrid 232, prio 200, intvl 100cs, length 12, addrs: 172.17.2.3 -- 00:02:27.194227 00:00:5e:00:01:e8 > 01:00:5e:00:00:12, ethertype 802.1Q (0x8100), length 60: vlan 102, p 0, ethertype IPv4, (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto VRRP (112), length 32) 172.17.2.3 > 224.0.0.18: vrrp 172.17.2.3 > 224.0.0.18: VRRPv3, Advertisement, vrid 232, prio 200, intvl 100cs, length 12, addrs: 172.17.2.3 The configuration: BACKUP system: ip table add 26 ip table add 27 set interface state Ext-0 up create sub-interfaces Ext-0 101 create sub-interfaces Ext-0 102 set interface state Ext-0.101 up set interface state Ext-0.102 up set interface ip table Ext-0.101 26 set interface ip table Ext-0.102 27 set interface ip address Ext-0.101 172.17.1.2/25 set interface ip address Ext-0.102 172.17.2.2/25 vrrp vr add Ext-0.101 vr_id 231 priority 100 no_preempt accept_mode 172.17.1.3 vrrp vr add Ext-0.102 vr_id 232 priority 100 no_preempt accept_mode 172.17.2.3 MASTER system: ip table add 26 ip table add 27 set interface state Ext-0 up create sub-interfaces Ext-0 101 create sub-interfaces Ext-0 102 set interface state Ext-0.101 up set interface state Ext-0.102 up set interface ip table Ext-0.101 26 set interface ip table Ext-0.102 27 set interface ip address Ext-0.101 172.17.1.1/25 set interface ip address Ext-0.102 172.17.2.1/25 vrrp vr add Ext-0.101 vr_id 231 priority 200 no_preempt accept_mode 172.17.1.3 vrrp vr add Ext-0.102 vr_id 232 priority 200 no_preempt accept_mode 172.17.2.3 Running BACKUP config: # vppctl show vrrp vr [0] sw_if_index 2 VR ID 231 IPv4 state Backup flags: preempt no accept yes unicast no priority: configured 100 adjusted 100 timers: adv interval 100 master adv 100 skew 60 master down 360 virtual MAC 00:00:5e:00:01:e7 addresses 172.17.1.3 peer addresses tracked interfaces [1] sw_if_index 3 VR ID 232 IPv4 state Backup flags: preempt no accept yes unicast no priority: configured 100 adjusted 100 timers: adv interval 100 master adv 100 skew 60 master down 360 virtual MAC 00:00:5e:00:01:e8 addresses 172.17.2.3 peer addresses tracked interfaces Running MASTER config: # vppctl show vrrp vr [0] sw_if_index 2 VR ID 231 IPv4 state Master flags: preempt no accept yes unicast no priority: configured 200 adjusted 200 timers: adv interval 100 master adv 100 skew 21 master down 321 virtual MAC 00:00:5e:00:01:e7 addresses 172.17.1.3 peer addresses tracked interfaces [1] sw_if_index 3 VR ID 232 IPv4 state Master flags: preempt no accept yes unicast no priority: configured
Re: [vpp-dev] VRRP issue when using interface in a table
Hi Matt, we change the VRRP configuration to use VLANs and ran into the next problem. With the following configuration, we don’t see any VRRP message leaving VPP: set interface state Ext-0 up create sub-interfaces Ext-0 10 set interface state Ext-0.10 up set interface ip address Ext-0.10 192.168.42.43/25 vrrp vr add Ext-0.10 vr_id 41 priority 200 interval 100 no_preempt accept_mode 192.168.42.42 I tried with both NICs, Intel X710 as well as Intel e1000, where in both cases it’s not working. Even a ping from vpp to the peer IP doesn’t send an ARP. This is what I checked: # vppctl show int addr Ext-0 (up): Ext-0.10 (up): L3 192.168.42.43/25 local0 (dn): # vppctl show ip fib ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto flowlabel ] epoch:0 flags:none locks:[default-route:1, ] 0.0.0.0/0 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:0 to:[0:0]] [0] [@0]: dpo-drop ip4 0.0.0.0/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]] [0] [@0]: dpo-drop ip4 192.168.42.0/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:9 to:[0:0]] [0] [@0]: dpo-drop ip4 192.168.42.0/25 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:8 buckets:1 uRPF:16 to:[10:960]] [0] [@4]: ipv4-glean: [src:192.168.42.0/25] Ext-0.10: mtu:9000 next:1 flags:[] 8c47be931910810a0806 192.168.42.43/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:13 to:[0:0]] [0] [@2]: dpo-receive: 192.168.42.43 on Ext-0.10 192.168.42.127/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:11 to:[0:0]] [0] [@0]: dpo-drop ip4 224.0.0.0/4 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]] [0] [@0]: dpo-drop ip4 240.0.0.0/4 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]] [0] [@0]: dpo-drop ip4 255.255.255.255/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]] [0] [@0]: dpo-drop ip4 I don’t know how to trace on outgoing packet. Any help is appreciated Thanks, BR/Mechthild From: vpp-dev@lists.fd.io On Behalf Of Mechthild Buescher via lists.fd.io Sent: Wednesday, 7 July 2021 12:02 To: Matthew Smith Cc: Neale Ranns ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] VRRP issue when using interface in a table Hi Matt, Thanks for the patch. I can confirm that the patch solves the vrrp issue and from what we see until now, it doesn’t break other traffic handled by that interface. So everything is fine Fyi: We applied the patch on VPP 21.06.1 Thank you very much for your help! BR/Mechthild From: Matthew Smith mailto:mgsm...@netgate.com>> Sent: Friday, 2 July 2021 23:01 To: Mechthild Buescher mailto:mechthild.buesc...@ericsson.com>> Cc: Neale Ranns mailto:ne...@graphiant.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] VRRP issue when using interface in a table Hi Mechthild, Here is a patch you can try - https://gerrit.fd.io/r/c/vpp/+/32999<https://protect2.fireeye.com/v1/url?k=7a0bd14a-2590e9a8-7a0b91d1-86073b36ea28-8df48bb092ba60d9=1=b9334abf-d165-4a83-8357-4b2df662477f=https%3A%2F%2Fgerrit.fd.io%2Fr%2Fc%2Fvpp%2F%2B%2F32999>. Let me know if it works for you. I don't have an i40e readily available to test it on and my local build is still using DPDK 21.01 while the current gerrit master branch uses DPDK 21.05 by default. So I could not test it against the current gerrit master branch. The exact same patch works against DPDK 21.01 and the function it modifies was not changed between 21.01 and 21.05 so I expect it ought to work, but. YMMV. I'll set the review score to -2 until I receive confirmation from you that it is working. -Matt On Fri, Jul 2, 2021 at 11:48 AM Mechthild Buescher mailto:mechthild.buesc...@ericsson.com>> wrote: Hi Matt, Thanks for your fast reply. Yes, it seems to be the “source pruning” issue on X710/XL710. When both VRs are in the master state, I can’t see any VRRP messages in the dpdk-input trace. Furthermore, I tried VRRP with Intel e1000 NICs where it behaves correctly: When the interface of the VRRP master goes down, the VRRP backup changes to state master and when the VRRP master recovers (ie. Interface is up), the peer node changes back to state backup. It would be nice if you can tell me how to disable source pruning with DPDK PMD. Thank you, BR/Mechthild From: Matthew Smith mailto:mgsm...@netgate.com>> Sent: Friday, 2 July 2021 16:06 To: Neale Ranns mailto:ne...@graphiant.com>> Cc: Mechthild Buescher mailto:mechthild.buesc...@ericsson.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] VRRP issue when using interface in a table There could be an issue with the NIC: vpp# show hardware-interfaces
Re: [vpp-dev] VRRP issue when using interface in a table
Hi Matt, Thanks for the patch. I can confirm that the patch solves the vrrp issue and from what we see until now, it doesn’t break other traffic handled by that interface. So everything is fine Fyi: We applied the patch on VPP 21.06.1 Thank you very much for your help! BR/Mechthild From: Matthew Smith Sent: Friday, 2 July 2021 23:01 To: Mechthild Buescher Cc: Neale Ranns ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] VRRP issue when using interface in a table Hi Mechthild, Here is a patch you can try - https://gerrit.fd.io/r/c/vpp/+/32999<https://protect2.fireeye.com/v1/url?k=7a0bd14a-2590e9a8-7a0b91d1-86073b36ea28-8df48bb092ba60d9=1=b9334abf-d165-4a83-8357-4b2df662477f=https%3A%2F%2Fgerrit.fd.io%2Fr%2Fc%2Fvpp%2F%2B%2F32999>. Let me know if it works for you. I don't have an i40e readily available to test it on and my local build is still using DPDK 21.01 while the current gerrit master branch uses DPDK 21.05 by default. So I could not test it against the current gerrit master branch. The exact same patch works against DPDK 21.01 and the function it modifies was not changed between 21.01 and 21.05 so I expect it ought to work, but. YMMV. I'll set the review score to -2 until I receive confirmation from you that it is working. -Matt On Fri, Jul 2, 2021 at 11:48 AM Mechthild Buescher mailto:mechthild.buesc...@ericsson.com>> wrote: Hi Matt, Thanks for your fast reply. Yes, it seems to be the “source pruning” issue on X710/XL710. When both VRs are in the master state, I can’t see any VRRP messages in the dpdk-input trace. Furthermore, I tried VRRP with Intel e1000 NICs where it behaves correctly: When the interface of the VRRP master goes down, the VRRP backup changes to state master and when the VRRP master recovers (ie. Interface is up), the peer node changes back to state backup. It would be nice if you can tell me how to disable source pruning with DPDK PMD. Thank you, BR/Mechthild From: Matthew Smith mailto:mgsm...@netgate.com>> Sent: Friday, 2 July 2021 16:06 To: Neale Ranns mailto:ne...@graphiant.com>> Cc: Mechthild Buescher mailto:mechthild.buesc...@ericsson.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] VRRP issue when using interface in a table There could be an issue with the NIC: vpp# show hardware-interfaces NameIdx Link Hardware Ext-0 1 up Ext-0 Link speed: 10 Gbps Ethernet address e4:43:4b:ed:59:10 Intel X710/XL710 Family With certain versions of firmware, these interfaces have a feature called "source pruning" enabled by default. When a MAC address is added on a X710/XL710 interface, packets which arrive with that address as their source MAC address are filtered by the NIC. Since VRRP uses a virtual MAC address as the source address of advertisements sent to peers, source pruning causes problems for it. A VRRP VR entering the master state will add the virtual MAC address to the NIC and henceforth the NIC will filter any higher priority advertisements that a peer might send because they are sourced from the virtual MAC address. I reported a bug to DPDK about it which has more details - https://bugs.dpdk.org/show_bug.cgi?id=648<https://protect2.fireeye.com/v1/url?k=7eb6f901-212dc042-7eb6b99a-861fcb972bfc-dfa86a12f9703310=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=https%3A%2F%2Fbugs.dpdk.org%2Fshow_bug.cgi%3Fid%3D648>. Mechthild, you can check whether this is the issue you're experiencing by taking a packet trace on node 1 when both VRs are in the master state. If source pruning is causing the problem, you will not see any received advertisements from the peer in the trace because they will have been filtered by the hardware and never reach VPP. There is no supported way to disable source pruning when using the DPDK PMD, but if your packet trace indicates that this appears to be the issue I can give you a patch to try which should disable it. If not, please send the output from the packet trace anyway so I can try to diagnose what else might be going on. Thanks, -Matt On Fri, Jul 2, 2021 at 4:04 AM Neale Ranns mailto:ne...@graphiant.com>> wrote: Hi Mechthild, Core VRRP issues I can’t help with, I no next to nothing about VRRP. I’ll hand over to those who do. /neale From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> on behalf of Mechthild Buescher via lists.fd.io<https://protect2.fireeye.com/v1/url?k=2c6b4627-73f07f64-2c6b06bc-861fcb972bfc-cf791770fb17ceed=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=http%3A%2F%2Flists.fd.io%2F> mailto:ericsson@lists.fd.io>> Date: Thursday, 1 July 2021 at 22:55 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> Subject: Re: [vpp-dev] VRRP issue when using interface in a table Hi Neale, I did some deeper investigations on the vrrp issu
Re: [vpp-dev] VRRP issue when using interface in a table
Hi Mechthild, Here is a patch you can try - https://gerrit.fd.io/r/c/vpp/+/32999. Let me know if it works for you. I don't have an i40e readily available to test it on and my local build is still using DPDK 21.01 while the current gerrit master branch uses DPDK 21.05 by default. So I could not test it against the current gerrit master branch. The exact same patch works against DPDK 21.01 and the function it modifies was not changed between 21.01 and 21.05 so I expect it ought to work, but. YMMV. I'll set the review score to -2 until I receive confirmation from you that it is working. -Matt On Fri, Jul 2, 2021 at 11:48 AM Mechthild Buescher < mechthild.buesc...@ericsson.com> wrote: > Hi Matt, > > > > Thanks for your fast reply. Yes, it seems to be the “source pruning” issue > on X710/XL710. > > > > When both VRs are in the master state, I can’t see any VRRP messages in > the dpdk-input trace. Furthermore, I tried VRRP with Intel e1000 NICs where > it behaves correctly: When the interface of the VRRP master goes down, the > VRRP backup changes to state master and when the VRRP master recovers (ie. > Interface is up), the peer node changes back to state backup. > > > > It would be nice if you can tell me how to disable source pruning with > DPDK PMD. > > > > Thank you, > > > > BR/Mechthild > > > > *From:* Matthew Smith > *Sent:* Friday, 2 July 2021 16:06 > *To:* Neale Ranns > *Cc:* Mechthild Buescher ; > vpp-dev@lists.fd.io > *Subject:* Re: [vpp-dev] VRRP issue when using interface in a table > > > > > > There could be an issue with the NIC: > > > > vpp# show hardware-interfaces > NameIdx Link Hardware > Ext-0 1 up Ext-0 > Link speed: 10 Gbps > Ethernet address e4:43:4b:ed:59:10 > Intel X710/XL710 Family > > > > With certain versions of firmware, these interfaces have a feature called > "source pruning" enabled by default. When a MAC address is added on a > X710/XL710 interface, packets which arrive with that address as their > source MAC address are filtered by the NIC. Since VRRP uses a virtual MAC > address as the source address of advertisements sent to peers, source > pruning causes problems for it. A VRRP VR entering the master state will > add the virtual MAC address to the NIC and henceforth the NIC will filter > any higher priority advertisements that a peer might send because they are > sourced from the virtual MAC address. I reported a bug to DPDK about it > which has more details - https://bugs.dpdk.org/show_bug.cgi?id=648 > <https://protect2.fireeye.com/v1/url?k=7eb6f901-212dc042-7eb6b99a-861fcb972bfc-dfa86a12f9703310=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=https%3A%2F%2Fbugs.dpdk.org%2Fshow_bug.cgi%3Fid%3D648> > . > > > > Mechthild, you can check whether this is the issue you're experiencing by > taking a packet trace on node 1 when both VRs are in the master state. If > source pruning is causing the problem, you will not see any received > advertisements from the peer in the trace because they will have been > filtered by the hardware and never reach VPP. There is no supported way to > disable source pruning when using the DPDK PMD, but if your packet trace > indicates that this appears to be the issue I can give you a patch to try > which should disable it. If not, please send the output from the packet > trace anyway so I can try to diagnose what else might be going on. > > > > Thanks, > > -Matt > > > > > > > > On Fri, Jul 2, 2021 at 4:04 AM Neale Ranns wrote: > > > > Hi Mechthild, > > > > Core VRRP issues I can’t help with, I no next to nothing about VRRP. I’ll > hand over to those who do. > > > > /neale > > > > > > *From: *vpp-dev@lists.fd.io on behalf of Mechthild > Buescher via lists.fd.io > <https://protect2.fireeye.com/v1/url?k=2c6b4627-73f07f64-2c6b06bc-861fcb972bfc-cf791770fb17ceed=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=http%3A%2F%2Flists.fd.io%2F> > > *Date: *Thursday, 1 July 2021 at 22:55 > *To: *vpp-dev@lists.fd.io > *Subject: *Re: [vpp-dev] VRRP issue when using interface in a table > > Hi Neale, > > > > I did some deeper investigations on the vrrp issue. What I observed is as > follows: > > > > On one node1 the VRRP config is: > set interface state Ext-0 up > > set interface ip address Ext-0 192.168.61.52/25 > <https://protect2.fireeye.com/v1/url?k=9226f0ca-cdbdc989-9226b051-861fcb972bfc-987c9fef2ec84c30=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=http%3A%2F%2F192.168.61.52%2F25> > > vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode > 192.168.61.5
Re: [vpp-dev] VRRP issue when using interface in a table
Hi Matt, Thanks for your fast reply. Yes, it seems to be the “source pruning” issue on X710/XL710. When both VRs are in the master state, I can’t see any VRRP messages in the dpdk-input trace. Furthermore, I tried VRRP with Intel e1000 NICs where it behaves correctly: When the interface of the VRRP master goes down, the VRRP backup changes to state master and when the VRRP master recovers (ie. Interface is up), the peer node changes back to state backup. It would be nice if you can tell me how to disable source pruning with DPDK PMD. Thank you, BR/Mechthild From: Matthew Smith Sent: Friday, 2 July 2021 16:06 To: Neale Ranns Cc: Mechthild Buescher ; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] VRRP issue when using interface in a table There could be an issue with the NIC: vpp# show hardware-interfaces NameIdx Link Hardware Ext-0 1 up Ext-0 Link speed: 10 Gbps Ethernet address e4:43:4b:ed:59:10 Intel X710/XL710 Family With certain versions of firmware, these interfaces have a feature called "source pruning" enabled by default. When a MAC address is added on a X710/XL710 interface, packets which arrive with that address as their source MAC address are filtered by the NIC. Since VRRP uses a virtual MAC address as the source address of advertisements sent to peers, source pruning causes problems for it. A VRRP VR entering the master state will add the virtual MAC address to the NIC and henceforth the NIC will filter any higher priority advertisements that a peer might send because they are sourced from the virtual MAC address. I reported a bug to DPDK about it which has more details - https://bugs.dpdk.org/show_bug.cgi?id=648<https://protect2.fireeye.com/v1/url?k=7eb6f901-212dc042-7eb6b99a-861fcb972bfc-dfa86a12f9703310=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=https%3A%2F%2Fbugs.dpdk.org%2Fshow_bug.cgi%3Fid%3D648>. Mechthild, you can check whether this is the issue you're experiencing by taking a packet trace on node 1 when both VRs are in the master state. If source pruning is causing the problem, you will not see any received advertisements from the peer in the trace because they will have been filtered by the hardware and never reach VPP. There is no supported way to disable source pruning when using the DPDK PMD, but if your packet trace indicates that this appears to be the issue I can give you a patch to try which should disable it. If not, please send the output from the packet trace anyway so I can try to diagnose what else might be going on. Thanks, -Matt On Fri, Jul 2, 2021 at 4:04 AM Neale Ranns mailto:ne...@graphiant.com>> wrote: Hi Mechthild, Core VRRP issues I can’t help with, I no next to nothing about VRRP. I’ll hand over to those who do. /neale From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> on behalf of Mechthild Buescher via lists.fd.io<https://protect2.fireeye.com/v1/url?k=2c6b4627-73f07f64-2c6b06bc-861fcb972bfc-cf791770fb17ceed=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=http%3A%2F%2Flists.fd.io%2F> mailto:ericsson@lists.fd.io>> Date: Thursday, 1 July 2021 at 22:55 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> Subject: Re: [vpp-dev] VRRP issue when using interface in a table Hi Neale, I did some deeper investigations on the vrrp issue. What I observed is as follows: On one node1 the VRRP config is: set interface state Ext-0 up set interface ip address Ext-0 192.168.61.52/25<https://protect2.fireeye.com/v1/url?k=9226f0ca-cdbdc989-9226b051-861fcb972bfc-987c9fef2ec84c30=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=http%3A%2F%2F192.168.61.52%2F25> vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode 192.168.61.50 On the other node2 the VRRP config is: set interface state Ext-0 up set interface ip address Ext-0 192.168.61.51/25<https://protect2.fireeye.com/v1/url?k=1f2f98b0-40b4a1f3-1f2fd82b-861fcb972bfc-90d03d09318514d8=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=http%3A%2F%2F192.168.61.51%2F25> vrrp vr add Ext-0 vr_id 61 priority 100 no_preempt accept_mode 192.168.61.50 When I start vpp and vrrp (vppctl vrrp proto start Ext-0 vr_id 61) on both nodes, everything looks fine: The node1 is master and has VIP: vppctl show int addr Ext-0 (up): L3 192.168.61.52/25<https://protect2.fireeye.com/v1/url?k=89e593eb-d67eaaa8-89e5d370-861fcb972bfc-37664a2997a749be=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=http%3A%2F%2F192.168.61.52%2F25> L3 192.168.61.50/25<https://protect2.fireeye.com/v1/url?k=98ac9667-c737af24-98acd6fc-861fcb972bfc-c2d2327744df6fee=1=a84fe3d4-3e6b-4dc3-be3e-b200c6641169=http%3A%2F%2F192.168.61.50%2F25> The node2 is backup: vppctl show int addr Ext-0 (up): L3 192.168.61.51/25<https://protect2.fireeye.com/v1/url?k=b91fc28a-e684fbc9-b91f8211-861fcb972bfc-ce656cb98dc38913=1=a84fe3d4-3e6b-4dc3-be3e-b2
Re: [vpp-dev] VRRP issue when using interface in a table
There could be an issue with the NIC: vpp# show hardware-interfaces > NameIdx Link Hardware > Ext-0 1 up Ext-0 > Link speed: 10 Gbps > Ethernet address e4:43:4b:ed:59:10 > Intel X710/XL710 Family With certain versions of firmware, these interfaces have a feature called "source pruning" enabled by default. When a MAC address is added on a X710/XL710 interface, packets which arrive with that address as their source MAC address are filtered by the NIC. Since VRRP uses a virtual MAC address as the source address of advertisements sent to peers, source pruning causes problems for it. A VRRP VR entering the master state will add the virtual MAC address to the NIC and henceforth the NIC will filter any higher priority advertisements that a peer might send because they are sourced from the virtual MAC address. I reported a bug to DPDK about it which has more details - https://bugs.dpdk.org/show_bug.cgi?id=648. Mechthild, you can check whether this is the issue you're experiencing by taking a packet trace on node 1 when both VRs are in the master state. If source pruning is causing the problem, you will not see any received advertisements from the peer in the trace because they will have been filtered by the hardware and never reach VPP. There is no supported way to disable source pruning when using the DPDK PMD, but if your packet trace indicates that this appears to be the issue I can give you a patch to try which should disable it. If not, please send the output from the packet trace anyway so I can try to diagnose what else might be going on. Thanks, -Matt On Fri, Jul 2, 2021 at 4:04 AM Neale Ranns wrote: > > > Hi Mechthild, > > > > Core VRRP issues I can’t help with, I no next to nothing about VRRP. I’ll > hand over to those who do. > > > > /neale > > > > > > *From: *vpp-dev@lists.fd.io on behalf of Mechthild > Buescher via lists.fd.io > *Date: *Thursday, 1 July 2021 at 22:55 > *To: *vpp-dev@lists.fd.io > *Subject: *Re: [vpp-dev] VRRP issue when using interface in a table > > Hi Neale, > > > > I did some deeper investigations on the vrrp issue. What I observed is as > follows: > > > > On one node1 the VRRP config is: > set interface state Ext-0 up > > set interface ip address Ext-0 192.168.61.52/25 > > vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode > 192.168.61.50 > > > > On the other node2 the VRRP config is: > > set interface state Ext-0 up > > set interface ip address Ext-0 192.168.61.51/25 > > vrrp vr add Ext-0 vr_id 61 priority 100 no_preempt accept_mode > 192.168.61.50 > > > > When I start vpp and vrrp (vppctl vrrp proto start Ext-0 vr_id 61) on both > nodes, everything looks fine: > > The node1 is master and has VIP: > vppctl show int addr > > Ext-0 (up): > > L3 192.168.61.52/25 > > L3 192.168.61.50/25 > > > > The node2 is backup: > > vppctl show int addr > > Ext-0 (up): > > L3 192.168.61.51/25 > > > > I can also swap the roles (master/backup) of the nodes by stopping and > starting vrrp on the node1: > > vppctl vrrp proto stop Ext-0 vr_id 61 > > vppctl vrrp proto start Ext-0 vr_id 61 > > > > But if node1 (master) goes down because the interface is flapping, > simulated with: > > vppctl set int state Ext-0 down; vppctl set int state Ext-0 up > > > > then node2 is getting master as expected but node1 is changing from state > ‘Interface Down’ to ‘Backup’ and then to ‘Master’. > > Now both nodes are master and both have the VIP. > > > > Is this another bug in VRRP? > > > > Your help is really appreciated. > > > > Thanks, BR/Mechthild > > > > > > *From:* Mechthild Buescher > *Sent:* Wednesday, 30 June 2021 17:40 > *To:* vpp-dev@lists.fd.io > *Subject:* RE: VRRP issue when using interface in a table > > > > Hi Neale, > > > > Thanks for your reply. The bugfix partly solved the issue – VRRP goes into > master/backup and keeps stable for a while. Unfortunately, it changes back > to master/master after some time (15 minutes – 1 hour). We are currently > trying to get more details and will come back to you. > > But thanks for your support so far, > > > > BR/Mechthild > > > > *From:* Neale Ranns > *Sent:* Thursday, 24 June 2021 12:33 > *To:* Mechthild Buescher ; > vpp-dev@lists.fd.io > *Subject:* Re: VRRP issue when using interface in a table > > > > Hi Mechthild, > > > > You’ll need to include: > > https://gerrit.fd.io/r/c/vpp/+/32298 > <https://protect2.fireeye.com/v1/url?k=43a4cf37-1c3ff635-43a48fac-869a14f4b08c
Re: [vpp-dev] VRRP issue when using interface in a table
Hi Mechthild, Core VRRP issues I can’t help with, I no next to nothing about VRRP. I’ll hand over to those who do. /neale From: vpp-dev@lists.fd.io on behalf of Mechthild Buescher via lists.fd.io Date: Thursday, 1 July 2021 at 22:55 To: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] VRRP issue when using interface in a table Hi Neale, I did some deeper investigations on the vrrp issue. What I observed is as follows: On one node1 the VRRP config is: set interface state Ext-0 up set interface ip address Ext-0 192.168.61.52/25 vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode 192.168.61.50 On the other node2 the VRRP config is: set interface state Ext-0 up set interface ip address Ext-0 192.168.61.51/25 vrrp vr add Ext-0 vr_id 61 priority 100 no_preempt accept_mode 192.168.61.50 When I start vpp and vrrp (vppctl vrrp proto start Ext-0 vr_id 61) on both nodes, everything looks fine: The node1 is master and has VIP: vppctl show int addr Ext-0 (up): L3 192.168.61.52/25 L3 192.168.61.50/25 The node2 is backup: vppctl show int addr Ext-0 (up): L3 192.168.61.51/25 I can also swap the roles (master/backup) of the nodes by stopping and starting vrrp on the node1: vppctl vrrp proto stop Ext-0 vr_id 61 vppctl vrrp proto start Ext-0 vr_id 61 But if node1 (master) goes down because the interface is flapping, simulated with: vppctl set int state Ext-0 down; vppctl set int state Ext-0 up then node2 is getting master as expected but node1 is changing from state ‘Interface Down’ to ‘Backup’ and then to ‘Master’. Now both nodes are master and both have the VIP. Is this another bug in VRRP? Your help is really appreciated. Thanks, BR/Mechthild From: Mechthild Buescher Sent: Wednesday, 30 June 2021 17:40 To: vpp-dev@lists.fd.io Subject: RE: VRRP issue when using interface in a table Hi Neale, Thanks for your reply. The bugfix partly solved the issue – VRRP goes into master/backup and keeps stable for a while. Unfortunately, it changes back to master/master after some time (15 minutes – 1 hour). We are currently trying to get more details and will come back to you. But thanks for your support so far, BR/Mechthild From: Neale Ranns mailto:ne...@graphiant.com>> Sent: Thursday, 24 June 2021 12:33 To: Mechthild Buescher mailto:mechthild.buesc...@ericsson.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: VRRP issue when using interface in a table Hi Mechthild, You’ll need to include: https://gerrit.fd.io/r/c/vpp/+/32298<https://protect2.fireeye.com/v1/url?k=43a4cf37-1c3ff635-43a48fac-869a14f4b08c-2b79380bed927e16=1=c3c90cbe-5ad1-4c2c-b574-111bb859ccb5=https%3A%2F%2Fgerrit.fd.io%2Fr%2Fc%2Fvpp%2F%2B%2F32298> /neale From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> on behalf of Mechthild Buescher via lists.fd.io mailto:mechthild.buescher=ericsson@lists.fd.io>> Date: Thursday, 24 June 2021 at 10:49 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> Subject: [vpp-dev] VRRP issue when using interface in a table Hi all, we are using VPP on two nodes where we would like to run VRRP. This works fine if the VRRP VR interface is in fib 0 but if we but the interface into FIB table 1 instead, VRRP is not working correctly anymore. Can you please help? Our setup: · 2 nodes with VPP on each node and one DPDK interface (we reduced the config to isolate the issue) connected to each VPP · a switch between the nodes which just forwards the traffic, so that it’s like a peer-2-peer connection The VPP version is (both nodes): vpp# show version vpp v21.01.0-6~gf70123b2c built by suse on SUSE at 2021-05-06T12:18:31 vpp# show version verbose Version: v21.01.0-6~gf70123b2c Compiled by: suse Compile host: SUSE Compile date: 2021-05-06T12:18:31 Compile location: /root/vpp-sp/vpp Compiler: GCC 7.5.0 Current PID: 6677 The VPP config uses the DPDK interface (both nodes): vpp# show hardware-interfaces NameIdx Link Hardware Ext-0 1 up Ext-0 Link speed: 10 Gbps Ethernet address e4:43:4b:ed:59:10 Intel X710/XL710 Family carrier up full duplex mtu 9206 flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum Devargs: rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32) tx: queues 3 (max 192), desc 1024 (min 64 max 4096 align 32) pci: device 8086:1572 subsystem 1028:1f9c address :17:00.00 numa 0 max rx packet len: 9728 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame scatter keep-crc rss-hash rx offload
Re: [vpp-dev] VRRP issue when using interface in a table
Hi Neale, I did some deeper investigations on the vrrp issue. What I observed is as follows: On one node1 the VRRP config is: set interface state Ext-0 up set interface ip address Ext-0 192.168.61.52/25 vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode 192.168.61.50 On the other node2 the VRRP config is: set interface state Ext-0 up set interface ip address Ext-0 192.168.61.51/25 vrrp vr add Ext-0 vr_id 61 priority 100 no_preempt accept_mode 192.168.61.50 When I start vpp and vrrp (vppctl vrrp proto start Ext-0 vr_id 61) on both nodes, everything looks fine: The node1 is master and has VIP: vppctl show int addr Ext-0 (up): L3 192.168.61.52/25 L3 192.168.61.50/25 The node2 is backup: vppctl show int addr Ext-0 (up): L3 192.168.61.51/25 I can also swap the roles (master/backup) of the nodes by stopping and starting vrrp on the node1: vppctl vrrp proto stop Ext-0 vr_id 61 vppctl vrrp proto start Ext-0 vr_id 61 But if node1 (master) goes down because the interface is flapping, simulated with: vppctl set int state Ext-0 down; vppctl set int state Ext-0 up then node2 is getting master as expected but node1 is changing from state 'Interface Down' to 'Backup' and then to 'Master'. Now both nodes are master and both have the VIP. Is this another bug in VRRP? Your help is really appreciated. Thanks, BR/Mechthild From: Mechthild Buescher Sent: Wednesday, 30 June 2021 17:40 To: vpp-dev@lists.fd.io Subject: RE: VRRP issue when using interface in a table Hi Neale, Thanks for your reply. The bugfix partly solved the issue - VRRP goes into master/backup and keeps stable for a while. Unfortunately, it changes back to master/master after some time (15 minutes - 1 hour). We are currently trying to get more details and will come back to you. But thanks for your support so far, BR/Mechthild From: Neale Ranns mailto:ne...@graphiant.com>> Sent: Thursday, 24 June 2021 12:33 To: Mechthild Buescher mailto:mechthild.buesc...@ericsson.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: VRRP issue when using interface in a table Hi Mechthild, You'll need to include: https://gerrit.fd.io/r/c/vpp/+/32298<https://protect2.fireeye.com/v1/url?k=43a4cf37-1c3ff635-43a48fac-869a14f4b08c-2b79380bed927e16=1=c3c90cbe-5ad1-4c2c-b574-111bb859ccb5=https%3A%2F%2Fgerrit.fd.io%2Fr%2Fc%2Fvpp%2F%2B%2F32298> /neale From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> on behalf of Mechthild Buescher via lists.fd.io mailto:mechthild.buescher=ericsson@lists.fd.io>> Date: Thursday, 24 June 2021 at 10:49 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> Subject: [vpp-dev] VRRP issue when using interface in a table Hi all, we are using VPP on two nodes where we would like to run VRRP. This works fine if the VRRP VR interface is in fib 0 but if we but the interface into FIB table 1 instead, VRRP is not working correctly anymore. Can you please help? Our setup: * 2 nodes with VPP on each node and one DPDK interface (we reduced the config to isolate the issue) connected to each VPP * a switch between the nodes which just forwards the traffic, so that it's like a peer-2-peer connection The VPP version is (both nodes): vpp# show version vpp v21.01.0-6~gf70123b2c built by suse on SUSE at 2021-05-06T12:18:31 vpp# show version verbose Version: v21.01.0-6~gf70123b2c Compiled by: suse Compile host: SUSE Compile date: 2021-05-06T12:18:31 Compile location: /root/vpp-sp/vpp Compiler: GCC 7.5.0 Current PID: 6677 The VPP config uses the DPDK interface (both nodes): vpp# show hardware-interfaces NameIdx Link Hardware Ext-0 1 up Ext-0 Link speed: 10 Gbps Ethernet address e4:43:4b:ed:59:10 Intel X710/XL710 Family carrier up full duplex mtu 9206 flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum Devargs: rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32) tx: queues 3 (max 192), desc 1024 (min 64 max 4096 align 32) pci: device 8086:1572 subsystem 1028:1f9c address :17:00.00 numa 0 max rx packet len: 9728 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame scatter keep-crc rss-hash rx offload active: ipv4-cksum jumbo-frame scatter tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs mbuf-fast-free tx offload active: udp-
Re: [vpp-dev] VRRP issue when using interface in a table
Hi Neale, Thanks for your reply. The bugfix partly solved the issue - VRRP goes into master/backup and keeps stable for a while. Unfortunately, it changes back to master/master after some time (15 minutes - 1 hour). We are currently trying to get more details and will come back to you. But thanks for your support so far, BR/Mechthild From: Neale Ranns Sent: Thursday, 24 June 2021 12:33 To: Mechthild Buescher ; vpp-dev@lists.fd.io Subject: Re: VRRP issue when using interface in a table Hi Mechthild, You'll need to include: https://gerrit.fd.io/r/c/vpp/+/32298<https://protect2.fireeye.com/v1/url?k=43a4cf37-1c3ff635-43a48fac-869a14f4b08c-2b79380bed927e16=1=c3c90cbe-5ad1-4c2c-b574-111bb859ccb5=https%3A%2F%2Fgerrit.fd.io%2Fr%2Fc%2Fvpp%2F%2B%2F32298> /neale From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> on behalf of Mechthild Buescher via lists.fd.io mailto:mechthild.buescher=ericsson@lists.fd.io>> Date: Thursday, 24 June 2021 at 10:49 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> Subject: [vpp-dev] VRRP issue when using interface in a table Hi all, we are using VPP on two nodes where we would like to run VRRP. This works fine if the VRRP VR interface is in fib 0 but if we but the interface into FIB table 1 instead, VRRP is not working correctly anymore. Can you please help? Our setup: * 2 nodes with VPP on each node and one DPDK interface (we reduced the config to isolate the issue) connected to each VPP * a switch between the nodes which just forwards the traffic, so that it's like a peer-2-peer connection The VPP version is (both nodes): vpp# show version vpp v21.01.0-6~gf70123b2c built by suse on SUSE at 2021-05-06T12:18:31 vpp# show version verbose Version: v21.01.0-6~gf70123b2c Compiled by: suse Compile host: SUSE Compile date: 2021-05-06T12:18:31 Compile location: /root/vpp-sp/vpp Compiler: GCC 7.5.0 Current PID: 6677 The VPP config uses the DPDK interface (both nodes): vpp# show hardware-interfaces NameIdx Link Hardware Ext-0 1 up Ext-0 Link speed: 10 Gbps Ethernet address e4:43:4b:ed:59:10 Intel X710/XL710 Family carrier up full duplex mtu 9206 flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum Devargs: rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32) tx: queues 3 (max 192), desc 1024 (min 64 max 4096 align 32) pci: device 8086:1572 subsystem 1028:1f9c address :17:00.00 numa 0 max rx packet len: 9728 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame scatter keep-crc rss-hash rx offload active: ipv4-cksum jumbo-frame scatter tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs mbuf-fast-free tx offload active: udp-cksum tcp-cksum multi-segs rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv6-frag ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload rss active:none tx burst mode: Scalar rx burst mode: Vector AVX2 Scattered The VRRP configs are (MASTER): set interface state Ext-0 up set interface ip address Ext-0 192.168.61.52/25 vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode 192.168.61.50 and on the system under test (SUT): ip table add 1 set interface ip table Ext-0 1 set interface state Ext-0 up set interface ip address Ext-0 192.168.61.51/25 vrrp vr add Ext-0 vr_id 61 priority 100 no_preempt accept_mode 192.168.61.50 On the MASTER, we started VRRP with: vrrp proto start Ext-0 vr_id 61 so that it has: vpp# show vrrp vr [0] sw_if_index 1 VR ID 61 IPv4 state Master flags: preempt no accept yes unicast no priority: configured 200 adjusted 200 timers: adv interval 100 master adv 100 skew 21 master down 321 virtual MAC 00:00:5e:00:01:3d addresses 192.168.61.50 peer addresses tracked interfaces On the SUT, we did not yet start VRRP, so we see: vpp# show vrrp vr [0] sw_if_index 1 VR ID 61 IPv4 state Initialize flags: preempt no accept yes unicast no priority: configured 100 adjusted 100 timers: adv interval 100 master adv 0 skew 0 master down 0 virtual MAC 00:00:5e:00:01:3d addresses 192.168.61.50 peer addresses tracked interfaces Here I see already that something is going wrong as the VRRP packets are not reaching vrrp4-input: vpp# show errors Count Node
Re: [vpp-dev] VRRP issue when using interface in a table
Hi Mechthild, You’ll need to include: https://gerrit.fd.io/r/c/vpp/+/32298 /neale From: vpp-dev@lists.fd.io on behalf of Mechthild Buescher via lists.fd.io Date: Thursday, 24 June 2021 at 10:49 To: vpp-dev@lists.fd.io Subject: [vpp-dev] VRRP issue when using interface in a table Hi all, we are using VPP on two nodes where we would like to run VRRP. This works fine if the VRRP VR interface is in fib 0 but if we but the interface into FIB table 1 instead, VRRP is not working correctly anymore. Can you please help? Our setup: · 2 nodes with VPP on each node and one DPDK interface (we reduced the config to isolate the issue) connected to each VPP · a switch between the nodes which just forwards the traffic, so that it’s like a peer-2-peer connection The VPP version is (both nodes): vpp# show version vpp v21.01.0-6~gf70123b2c built by suse on SUSE at 2021-05-06T12:18:31 vpp# show version verbose Version: v21.01.0-6~gf70123b2c Compiled by: suse Compile host: SUSE Compile date: 2021-05-06T12:18:31 Compile location: /root/vpp-sp/vpp Compiler: GCC 7.5.0 Current PID: 6677 The VPP config uses the DPDK interface (both nodes): vpp# show hardware-interfaces NameIdx Link Hardware Ext-0 1 up Ext-0 Link speed: 10 Gbps Ethernet address e4:43:4b:ed:59:10 Intel X710/XL710 Family carrier up full duplex mtu 9206 flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum Devargs: rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32) tx: queues 3 (max 192), desc 1024 (min 64 max 4096 align 32) pci: device 8086:1572 subsystem 1028:1f9c address :17:00.00 numa 0 max rx packet len: 9728 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame scatter keep-crc rss-hash rx offload active: ipv4-cksum jumbo-frame scatter tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs mbuf-fast-free tx offload active: udp-cksum tcp-cksum multi-segs rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv6-frag ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload rss active:none tx burst mode: Scalar rx burst mode: Vector AVX2 Scattered The VRRP configs are (MASTER): set interface state Ext-0 up set interface ip address Ext-0 192.168.61.52/25 vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode 192.168.61.50 and on the system under test (SUT): ip table add 1 set interface ip table Ext-0 1 set interface state Ext-0 up set interface ip address Ext-0 192.168.61.51/25 vrrp vr add Ext-0 vr_id 61 priority 100 no_preempt accept_mode 192.168.61.50 On the MASTER, we started VRRP with: vrrp proto start Ext-0 vr_id 61 so that it has: vpp# show vrrp vr [0] sw_if_index 1 VR ID 61 IPv4 state Master flags: preempt no accept yes unicast no priority: configured 200 adjusted 200 timers: adv interval 100 master adv 100 skew 21 master down 321 virtual MAC 00:00:5e:00:01:3d addresses 192.168.61.50 peer addresses tracked interfaces On the SUT, we did not yet start VRRP, so we see: vpp# show vrrp vr [0] sw_if_index 1 VR ID 61 IPv4 state Initialize flags: preempt no accept yes unicast no priority: configured 100 adjusted 100 timers: adv interval 100 master adv 0 skew 0 master down 0 virtual MAC 00:00:5e:00:01:3d addresses 192.168.61.50 peer addresses tracked interfaces Here I see already that something is going wrong as the VRRP packets are not reaching vrrp4-input: vpp# show errors Count Node Reason Severity 5 dpdk-input no error error 138 ip4-localip4 source lookup miss error (If we configure SUT similar to the MASTER, ie interface in FIB 0, I can see vrrp4-input at this point) The trace of dpdk-input gives: Packet 1 00:00:57:644818: dpdk-input Ext-0 rx queue 0 buffer 0x9b7ec: current data 0, length 60, buffer-pool 0, ref-count 1, totlen-nifb 0, trace handle 0x100 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 0, nb_segs 1, pkt_len 60 buf_len 2176, data_len 60, ol_flags 0x180, data_off 128, phys_addr 0x26dfb80 packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 Packet Offload Flags
[vpp-dev] VRRP issue when using interface in a table
Hi all, we are using VPP on two nodes where we would like to run VRRP. This works fine if the VRRP VR interface is in fib 0 but if we but the interface into FIB table 1 instead, VRRP is not working correctly anymore. Can you please help? Our setup: * 2 nodes with VPP on each node and one DPDK interface (we reduced the config to isolate the issue) connected to each VPP * a switch between the nodes which just forwards the traffic, so that it's like a peer-2-peer connection The VPP version is (both nodes): vpp# show version vpp v21.01.0-6~gf70123b2c built by suse on SUSE at 2021-05-06T12:18:31 vpp# show version verbose Version: v21.01.0-6~gf70123b2c Compiled by: suse Compile host: SUSE Compile date: 2021-05-06T12:18:31 Compile location: /root/vpp-sp/vpp Compiler: GCC 7.5.0 Current PID: 6677 The VPP config uses the DPDK interface (both nodes): vpp# show hardware-interfaces NameIdx Link Hardware Ext-0 1 up Ext-0 Link speed: 10 Gbps Ethernet address e4:43:4b:ed:59:10 Intel X710/XL710 Family carrier up full duplex mtu 9206 flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum Devargs: rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32) tx: queues 3 (max 192), desc 1024 (min 64 max 4096 align 32) pci: device 8086:1572 subsystem 1028:1f9c address :17:00.00 numa 0 max rx packet len: 9728 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame scatter keep-crc rss-hash rx offload active: ipv4-cksum jumbo-frame scatter tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs mbuf-fast-free tx offload active: udp-cksum tcp-cksum multi-segs rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv6-frag ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload rss active:none tx burst mode: Scalar rx burst mode: Vector AVX2 Scattered The VRRP configs are (MASTER): set interface state Ext-0 up set interface ip address Ext-0 192.168.61.52/25 vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode 192.168.61.50 and on the system under test (SUT): ip table add 1 set interface ip table Ext-0 1 set interface state Ext-0 up set interface ip address Ext-0 192.168.61.51/25 vrrp vr add Ext-0 vr_id 61 priority 100 no_preempt accept_mode 192.168.61.50 On the MASTER, we started VRRP with: vrrp proto start Ext-0 vr_id 61 so that it has: vpp# show vrrp vr [0] sw_if_index 1 VR ID 61 IPv4 state Master flags: preempt no accept yes unicast no priority: configured 200 adjusted 200 timers: adv interval 100 master adv 100 skew 21 master down 321 virtual MAC 00:00:5e:00:01:3d addresses 192.168.61.50 peer addresses tracked interfaces On the SUT, we did not yet start VRRP, so we see: vpp# show vrrp vr [0] sw_if_index 1 VR ID 61 IPv4 state Initialize flags: preempt no accept yes unicast no priority: configured 100 adjusted 100 timers: adv interval 100 master adv 0 skew 0 master down 0 virtual MAC 00:00:5e:00:01:3d addresses 192.168.61.50 peer addresses tracked interfaces Here I see already that something is going wrong as the VRRP packets are not reaching vrrp4-input: vpp# show errors Count Node Reason Severity 5 dpdk-input no error error 138 ip4-localip4 source lookup miss error (If we configure SUT similar to the MASTER, ie interface in FIB 0, I can see vrrp4-input at this point) The trace of dpdk-input gives: Packet 1 00:00:57:644818: dpdk-input Ext-0 rx queue 0 buffer 0x9b7ec: current data 0, length 60, buffer-pool 0, ref-count 1, totlen-nifb 0, trace handle 0x100 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 0, nb_segs 1, pkt_len 60 buf_len 2176, data_len 60, ol_flags 0x180, data_off 128, phys_addr 0x26dfb80 packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 Packet Offload Flags PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid Packet Types RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without extension headers
Re: [vpp-dev] VRRP issue
Hi Naveen, The trace file appears to show two VRRP advertisements which were received. There does not seem to be any indication that those packets ever reached the vrrp4-input node. For both packets the trace shows this at the end: 00:21:49:755122: error-drop rx:loop0 00:21:49:755122: drop ip4-local: ip4 source lookup miss The error counter for ip4 source lookup miss also appears to be increasing in the output of 'show err'. Does the backup have an IP address in the same subnet with 10.4.4.3 configured on loop0? Your original diagram showed addresses in that subnet configured on the loop0 interface on both VRs, but you mentioned that the configuration has changed. It is pretty hard for me to guess what's happening here without having the full configuration available. Can you give me a full set of commands I could run in vppctl to exactly replicate the configuration of both VRs? I don't just mean the settings related to VRRP, but all interface configuration, bridge domain, etc Thanks, -Matt On Fri, Aug 14, 2020 at 8:10 PM Naveen Joy (najoy) wrote: > Hi Matthew, > > > > Thanks! I have attached packet captures to this email. I have captured a > few more packets as the uplink was quite chatty. > > The VRRP speakers are on Vlan 110. > > The second file has the untruncated show err output run a few times. > > Also, apologies for not clarifying the IP address change. During > troubleshooting, I had switched the backup and master VRs – still seeing > the same behavior. > > The IP address of the master i.e. 10.4.4.3 shown below is correct > (Attached: updated diagram). > > > > Regards, > > Naveen > > > > *From: * on behalf of "Matthew Smith via lists.fd.io" > > *Reply-To: *"mgsm...@netgate.com" > *Date: *Friday, August 14, 2020 at 2:47 PM > *To: *"Naveen Joy (najoy)" > *Cc: *"vpp-dev@lists.fd.io" > *Subject: *Re: [vpp-dev] VRRP issue > > > > > > Hi Naveen, > > > > See replies inline below... > > > > On Fri, Aug 14, 2020 at 12:53 PM Naveen Joy (najoy) > wrote: > > Thanks, Matthew. I am seeing the same behavior with the default > advertisement interval of 1s. > > Tcpdump on a linux tap interface plugged into the same BD as the backup VR > shows VRRP advertisements arriving at the configured rate of 1s (100cs), > > So, there is no packet loss of advertisements or delays in sending > advertisements by the master VR. > > > > 10:37:19.991540 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:20.991619 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:21.991783 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:22.991792 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:23.991926 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:24.991976 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:25.992057 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:26.992131 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:27.992257 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:28.992311 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:29.992402 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:30.992513 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:31.992586 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > > > > > > > Ok, its good to know that the packets are all arriving on the backup. > > > > The diagram attached to the first message in this thread said that the > master has address 10.4.4.1 and the backup has 10.4.4.3. Is that diagram > still accurate? The packets in the capture have a source address of > 10.4.4.3, which is the address that is supposed to be configured on the > backup according to the diagram. If the diagram is still accurate, it seems > like those packets should be dropped by the backup as 'spoofed' packets > since their source address is configured locally on the backup. > > > > If the diagram is not accurate and the master VR i
Re: [vpp-dev] VRRP issue
Hi Naveen, See replies inline below... On Fri, Aug 14, 2020 at 12:53 PM Naveen Joy (najoy) wrote: > Thanks, Matthew. I am seeing the same behavior with the default > advertisement interval of 1s. > > Tcpdump on a linux tap interface plugged into the same BD as the backup VR > shows VRRP advertisements arriving at the configured rate of 1s (100cs), > > So, there is no packet loss of advertisements or delays in sending > advertisements by the master VR. > > > > 10:37:19.991540 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:20.991619 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:21.991783 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:22.991792 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:23.991926 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:24.991976 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:25.992057 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:26.992131 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:27.992257 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:28.992311 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:29.992402 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:30.992513 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > 10:37:31.992586 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid > 10, prio 110, intvl 100cs, length 12 > > > > > Ok, its good to know that the packets are all arriving on the backup. The diagram attached to the first message in this thread said that the master has address 10.4.4.1 and the backup has 10.4.4.3. Is that diagram still accurate? The packets in the capture have a source address of 10.4.4.3, which is the address that is supposed to be configured on the backup according to the diagram. If the diagram is still accurate, it seems like those packets should be dropped by the backup as 'spoofed' packets since their source address is configured locally on the backup. If the diagram is not accurate and the master VR is truly supposed to be advertising with a source address of 10.4.4.3, can you please use vppctl to generate a packet capture on the backup VR? You could run something like 'vppctl trace add dpdk-input 50; sleep 10; vppctl show trace'. Depending on how noisy your network is, that ought to capture a few inbound advertisements. > However, it appears that there is a delay in VRRP packet processing at the > backup VR resulting in frequent state transitions. > > > > On the backup VR: > > vpp# show err > >CountNode Reason > > 120347 vrrp4-input VRRP packets processed > > > > vpp# show err (after 1 sec) > >CountNode Reason > > 120347 vrrp4-input VRRP packets processed > > > Is that the only output from 'show err' or did you clean up the output to only include the counters which looked like they are related to VRRP? If there are other counters displayed by 'show err' that you omitted from the output, it would be helpful to see the full output. > Also, log on the backup VR shows that VRRP advertisements from master are > received every 4s > > > > Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received > advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4 > > Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0] > sw_if_index 14 VR ID 10 IPv4 transitioning to Backup > > Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: > Deleting VR addresses on sw_if_index 14 > > Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: > Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 13 > > Aug 14 10:44:00 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0] > sw_if_index 14 VR ID 10 IPv4 transitioning to Master > > Aug 14 10:44:00 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Adding > VR addresses on sw_if_index 14 > > Aug 14 10:44:00 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Adding > virtual MAC address 00:00:5e:00:01:0a on hardware interface 13 > > Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received > advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4 > > Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0] > sw_if_index 14 VR ID 10 IPv4 transitioning to Backup > > Aug
Re: [vpp-dev] VRRP issue
Thanks, Matthew. I am seeing the same behavior with the default advertisement interval of 1s. Tcpdump on a linux tap interface plugged into the same BD as the backup VR shows VRRP advertisements arriving at the configured rate of 1s (100cs), So, there is no packet loss of advertisements or delays in sending advertisements by the master VR. 10:37:19.991540 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:20.991619 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:21.991783 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:22.991792 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:23.991926 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:24.991976 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:25.992057 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:26.992131 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:27.992257 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:28.992311 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:29.992402 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:30.992513 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 10:37:31.992586 IP 10.4.4.3 > vrrp.mcast.net: VRRPv3, Advertisement, vrid 10, prio 110, intvl 100cs, length 12 However, it appears that there is a delay in VRRP packet processing at the backup VR resulting in frequent state transitions. On the backup VR: vpp# show err CountNode Reason 120347 vrrp4-input VRRP packets processed vpp# show err (after 1 sec) CountNode Reason 120347 vrrp4-input VRRP packets processed Also, log on the backup VR shows that VRRP advertisements from master are received every 4s Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4 Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0] sw_if_index 14 VR ID 10 IPv4 transitioning to Backup Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Deleting VR addresses on sw_if_index 14 Aug 14 10:43:57 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 13 Aug 14 10:44:00 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0] sw_if_index 14 VR ID 10 IPv4 transitioning to Master Aug 14 10:44:00 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Adding VR addresses on sw_if_index 14 Aug 14 10:44:00 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Adding virtual MAC address 00:00:5e:00:01:0a on hardware interface 13 Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4 Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0] sw_if_index 14 VR ID 10 IPv4 transitioning to Backup Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Deleting VR addresses on sw_if_index 14 Aug 14 10:44:01 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 13 Aug 14 10:44:04 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0] sw_if_index 14 VR ID 10 IPv4 transitioning to Master Aug 14 10:44:04 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Adding VR addresses on sw_if_index 14 Aug 14 10:44:04 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Adding virtual MAC address 00:00:5e:00:01:0a on hardware interface 13 Aug 14 10:44:05 ml-ucs-01 vnet[5504]: vrrp_input_process:223: Received advertisement for master VR [0] sw_if_index 14 VR ID 10 IPv4 Aug 14 10:44:05 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0] sw_if_index 14 VR ID 10 IPv4 transitioning to Backup Aug 14 10:44:05 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Deleting VR addresses on sw_if_index 14 Aug 14 10:44:05 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 13 Aug 14 10:44:08 ml-ucs-01 vnet[5504]: vrrp_vr_transition:283: VR [0] sw_if_index 14 VR ID 10 IPv4 transitioning to Master Aug 14 10:44:08 ml-ucs-01 vnet[5504]: vrrp_vr_transition_addrs:238: Adding VR addresses on sw_if_index 14 Aug 14 10:44:08 ml-ucs-01 vnet[5504]: vrrp_vr_transition_vmac:123: Adding virtual MAC address 00:00:5e:00:01:0a on hardware interface 13 Aug 14 10:44:09 ml-ucs-01
Re: [vpp-dev] VRRP issue
Hi Naveen, Generally a transition from backup to master occurs if the master down timer expires and no advertisement has been received. So it seems like some advertisement packets from the higher priority VR are not being received or are not being processed before the timer expires. Since the lower priority VR keeps switching from master to backup after receiving an advertisement from the higher priority VR, at least some of the advertisements from the higher priority VR are clearly being received. Packet loss of advertisements is the first thing that I suggest you investigate. A packet trace or capture on the lower priority VR would probably be helpful in determining whether the advertisements are arriving. It also might be helpful to see what 'vppctl show errors' says. Another possibility is that there are delays in sending advertisements by the higher priority VR or in receiving/processing them on the lower priority VR. The advertisement interval appears to be set to 0.3s and the master down timer is 1.08s. It seems unlikely that with packets being sent every 0.3s that either sending or receiving could be delayed by enough that the receiving side would not process one within 1.08s. Just to rule out that possibility, you could try increasing the advertisement interval to a higher value and see if the situation improves. Do you still see the same behavior if you configure the default advertisement interval of 1s (100cs) on both VRs? Thanks, -Matt On Thu, Aug 13, 2020 at 5:34 PM Naveen Joy (najoy) wrote: > Hi Matthew/All, > > > > I am facing an issue with VRRP in VPP and would appreciate your help. > > > > (Attached - architecture diagram) > > > >1. I have 2 nodes with VPP & in each node, VRRP is configured to back >up a router BVI interface in a bridge domain. >2. The VRRP VRs are speaking VRRP (multicast) over an uplink VLAN >interface connected to an external switch. >3. The active router has a VR priority of 110 and is set to preempt. > > The backup router has a VR priority of 100 and is not in preempt. > > > >1. The issue is that VRRP in the backup router is unstable and keeps >transitioning between the master and backup states every second. > > However, the VRRP in the master node is stable. > > > > I am running the latest VPP release installed from master this week. > > > > vpp# show version verbose > > Version: v20.09-rc0~283-g40c07ce7a~b1542 > > Compiled by: root > > Compile host: 1f7cd9b19229 > > Compile date: 2020-08-11T20:40:47 > > Compile location: /w/workspace/vpp-merge-master-ubuntu1804 > > Compiler: Clang/LLVM 9.0.0 (tags/RELEASE_900/final) > > Current PID: 5504 > > > > *On the backup node –* > > > > Aug 13 12:59:48 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: > Deleting VR addresses on sw_if_index 11 > > Aug 13 12:59:48 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: > Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 > > Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] > sw_if_index 11 VR ID 10 IPv4 transitioning to Master > > Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Adding > VR addresses on sw_if_index 11 > > Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Adding > virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 > > Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_input_process:223: Received > advertisement for master VR [0] sw_if_index 11 VR ID 10 IPv4 > > Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] > sw_if_index 11 VR ID 10 IPv4 transitioning to Backup > > Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: > Deleting VR addresses on sw_if_index 11 > > Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: > Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 > > Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] > sw_if_index 11 VR ID 10 IPv4 transitioning to Master > > Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Adding > VR addresses on sw_if_index 11 > > Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Adding > virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 > > Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_input_process:223: Received > advertisement for master VR [0] sw_if_index 11 VR ID 10 IPv4 > > Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] > sw_if_index 11 VR ID 10 IPv4 transitioning to Backup > > Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: > Deleting VR addresses on sw_if_index 11 > > Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: > Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 > > Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] > sw_if_index 11 VR ID 10 IPv4 transitioning to Master > >
Re: [vpp-dev] VRRP issue
Hi Naveen, Could you try a shut/no shut on the vlan segment between the VRs? Sometimes unreliable communication between the switches can cause the hellos to not be seen at other end.. Interface counters on the interface in question should help Thanks! -- Regards, Balaji. From: on behalf of "Naveen Joy via lists.fd.io" Reply-To: "Naveen Joy (najoy)" Date: Thursday, August 13, 2020 at 3:34 PM To: "mgsm...@netgate.com" Cc: "vpp-dev@lists.fd.io" Subject: [vpp-dev] VRRP issue Hi Matthew/All, I am facing an issue with VRRP in VPP and would appreciate your help. (Attached - architecture diagram) 1. I have 2 nodes with VPP & in each node, VRRP is configured to back up a router BVI interface in a bridge domain. 2. The VRRP VRs are speaking VRRP (multicast) over an uplink VLAN interface connected to an external switch. 3. The active router has a VR priority of 110 and is set to preempt. The backup router has a VR priority of 100 and is not in preempt. 1. The issue is that VRRP in the backup router is unstable and keeps transitioning between the master and backup states every second. However, the VRRP in the master node is stable. I am running the latest VPP release installed from master this week. vpp# show version verbose Version: v20.09-rc0~283-g40c07ce7a~b1542 Compiled by: root Compile host: 1f7cd9b19229 Compile date: 2020-08-11T20:40:47 Compile location: /w/workspace/vpp-merge-master-ubuntu1804 Compiler: Clang/LLVM 9.0.0 (tags/RELEASE_900/final) Current PID: 5504 On the backup node – Aug 13 12:59:48 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Deleting VR addresses on sw_if_index 11 Aug 13 12:59:48 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] sw_if_index 11 VR ID 10 IPv4 transitioning to Master Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Adding VR addresses on sw_if_index 11 Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Adding virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_input_process:223: Received advertisement for master VR [0] sw_if_index 11 VR ID 10 IPv4 Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] sw_if_index 11 VR ID 10 IPv4 transitioning to Backup Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Deleting VR addresses on sw_if_index 11 Aug 13 12:59:50 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] sw_if_index 11 VR ID 10 IPv4 transitioning to Master Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Adding VR addresses on sw_if_index 11 Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Adding virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_input_process:223: Received advertisement for master VR [0] sw_if_index 11 VR ID 10 IPv4 Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] sw_if_index 11 VR ID 10 IPv4 transitioning to Backup Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Deleting VR addresses on sw_if_index 11 Aug 13 12:59:51 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] sw_if_index 11 VR ID 10 IPv4 transitioning to Master Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Adding VR addresses on sw_if_index 11 Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Adding virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_input_process:223: Received advertisement for master VR [0] sw_if_index 11 VR ID 10 IPv4 Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition:283: VR [0] sw_if_index 11 VR ID 10 IPv4 transitioning to Backup Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition_addrs:238: Deleting VR addresses on sw_if_index 11 Aug 13 12:59:52 ml-ucs-03 vnet[4023]: vrrp_vr_transition_vmac:123: Deleting virtual MAC address 00:00:5e:00:01:0a on hardware interface 10 vpp# show vrrp vr [0] sw_if_index 11 VR ID 10 IPv4 state Backup flags: preempt no accept yes unicast no priority: configured 100 adjusted 100 timers: adv interval 30 master adv 30 skew 18 master down 108 virtual MAC 00:00:5e:00:01:0a addresses 10.4.4.5 peer addresses tracked interfaces vpp# show vrrp vr [0] sw_if_index 11 VR ID 10 IPv4 state Master flags: preempt no accept yes unicast no priority: configured 100 adjust
