Hi Matt, Thanks for the patch. I can confirm that the patch solves the vrrp issue and from what we see until now, it doesn’t break other traffic handled by that interface. So everything is fine 😊
Fyi: We applied the patch on VPP 21.06.1 Thank you very much for your help! BR/Mechthild From: Matthew Smith <mgsm...@netgate.com> Sent: Friday, 2 July 2021 23:01 To: Mechthild Buescher <mechthild.buesc...@ericsson.com> Cc: Neale Ranns <ne...@graphiant.com>; vpp-dev@lists.fd.io Subject: Re: [vpp-dev] VRRP issue when using interface in a table Hi Mechthild, Here is a patch you can try - https://gerrit.fd.io/r/c/vpp/+/32999<https://protect2.fireeye.com/v1/url?k=7a0bd14a-2590e9a8-7a0b91d1-86073b36ea28-8df48bb092ba60d9&q=1&e=b9334abf-d165-4a83-8357-4b2df662477f&u=https%3A%2F%2Fgerrit.fd.io%2Fr%2Fc%2Fvpp%2F%2B%2F32999>. Let me know if it works for you. I don't have an i40e readily available to test it on and my local build is still using DPDK 21.01 while the current gerrit master branch uses DPDK 21.05 by default. So I could not test it against the current gerrit master branch. The exact same patch works against DPDK 21.01 and the function it modifies was not changed between 21.01 and 21.05 so I expect it ought to work, but..... YMMV. I'll set the review score to -2 until I receive confirmation from you that it is working. -Matt On Fri, Jul 2, 2021 at 11:48 AM Mechthild Buescher <mechthild.buesc...@ericsson.com<mailto:mechthild.buesc...@ericsson.com>> wrote: Hi Matt, Thanks for your fast reply. Yes, it seems to be the “source pruning” issue on X710/XL710. When both VRs are in the master state, I can’t see any VRRP messages in the dpdk-input trace. Furthermore, I tried VRRP with Intel e1000 NICs where it behaves correctly: When the interface of the VRRP master goes down, the VRRP backup changes to state master and when the VRRP master recovers (ie. Interface is up), the peer node changes back to state backup. It would be nice if you can tell me how to disable source pruning with DPDK PMD. Thank you, BR/Mechthild From: Matthew Smith <mgsm...@netgate.com<mailto:mgsm...@netgate.com>> Sent: Friday, 2 July 2021 16:06 To: Neale Ranns <ne...@graphiant.com<mailto:ne...@graphiant.com>> Cc: Mechthild Buescher <mechthild.buesc...@ericsson.com<mailto:mechthild.buesc...@ericsson.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] VRRP issue when using interface in a table There could be an issue with the NIC: vpp# show hardware-interfaces Name Idx Link Hardware Ext-0 1 up Ext-0 Link speed: 10 Gbps Ethernet address e4:43:4b:ed:59:10 Intel X710/XL710 Family With certain versions of firmware, these interfaces have a feature called "source pruning" enabled by default. When a MAC address is added on a X710/XL710 interface, packets which arrive with that address as their source MAC address are filtered by the NIC. Since VRRP uses a virtual MAC address as the source address of advertisements sent to peers, source pruning causes problems for it. A VRRP VR entering the master state will add the virtual MAC address to the NIC and henceforth the NIC will filter any higher priority advertisements that a peer might send because they are sourced from the virtual MAC address. I reported a bug to DPDK about it which has more details - https://bugs.dpdk.org/show_bug.cgi?id=648<https://protect2.fireeye.com/v1/url?k=7eb6f901-212dc042-7eb6b99a-861fcb972bfc-dfa86a12f9703310&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=https%3A%2F%2Fbugs.dpdk.org%2Fshow_bug.cgi%3Fid%3D648>. Mechthild, you can check whether this is the issue you're experiencing by taking a packet trace on node 1 when both VRs are in the master state. If source pruning is causing the problem, you will not see any received advertisements from the peer in the trace because they will have been filtered by the hardware and never reach VPP. There is no supported way to disable source pruning when using the DPDK PMD, but if your packet trace indicates that this appears to be the issue I can give you a patch to try which should disable it. If not, please send the output from the packet trace anyway so I can try to diagnose what else might be going on. Thanks, -Matt On Fri, Jul 2, 2021 at 4:04 AM Neale Ranns <ne...@graphiant.com<mailto:ne...@graphiant.com>> wrote: Hi Mechthild, Core VRRP issues I can’t help with, I no next to nothing about VRRP. I’ll hand over to those who do. /neale From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> on behalf of Mechthild Buescher via lists.fd.io<https://protect2.fireeye.com/v1/url?k=2c6b4627-73f07f64-2c6b06bc-861fcb972bfc-cf791770fb17ceed&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2Flists.fd.io%2F> <mechthild.buescher=ericsson....@lists.fd.io<mailto:ericsson....@lists.fd.io>> Date: Thursday, 1 July 2021 at 22:55 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> Subject: Re: [vpp-dev] VRRP issue when using interface in a table Hi Neale, I did some deeper investigations on the vrrp issue. What I observed is as follows: On one node1 the VRRP config is: set interface state Ext-0 up set interface ip address Ext-0 192.168.61.52/25<https://protect2.fireeye.com/v1/url?k=9226f0ca-cdbdc989-9226b051-861fcb972bfc-987c9fef2ec84c30&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.52%2F25> vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode 192.168.61.50 On the other node2 the VRRP config is: set interface state Ext-0 up set interface ip address Ext-0 192.168.61.51/25<https://protect2.fireeye.com/v1/url?k=1f2f98b0-40b4a1f3-1f2fd82b-861fcb972bfc-90d03d09318514d8&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.51%2F25> vrrp vr add Ext-0 vr_id 61 priority 100 no_preempt accept_mode 192.168.61.50 When I start vpp and vrrp (vppctl vrrp proto start Ext-0 vr_id 61) on both nodes, everything looks fine: The node1 is master and has VIP: vppctl show int addr Ext-0 (up): L3 192.168.61.52/25<https://protect2.fireeye.com/v1/url?k=89e593eb-d67eaaa8-89e5d370-861fcb972bfc-37664a2997a749be&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.52%2F25> L3 192.168.61.50/25<https://protect2.fireeye.com/v1/url?k=98ac9667-c737af24-98acd6fc-861fcb972bfc-c2d2327744df6fee&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.50%2F25> The node2 is backup: vppctl show int addr Ext-0 (up): L3 192.168.61.51/25<https://protect2.fireeye.com/v1/url?k=b91fc28a-e684fbc9-b91f8211-861fcb972bfc-ce656cb98dc38913&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.51%2F25> I can also swap the roles (master/backup) of the nodes by stopping and starting vrrp on the node1: vppctl vrrp proto stop Ext-0 vr_id 61 vppctl vrrp proto start Ext-0 vr_id 61 But if node1 (master) goes down because the interface is flapping, simulated with: vppctl set int state Ext-0 down; vppctl set int state Ext-0 up then node2 is getting master as expected but node1 is changing from state ‘Interface Down’ to ‘Backup’ and then to ‘Master’. Now both nodes are master and both have the VIP. Is this another bug in VRRP? Your help is really appreciated. Thanks, BR/Mechthild From: Mechthild Buescher Sent: Wednesday, 30 June 2021 17:40 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: VRRP issue when using interface in a table Hi Neale, Thanks for your reply. The bugfix partly solved the issue – VRRP goes into master/backup and keeps stable for a while. Unfortunately, it changes back to master/master after some time (15 minutes – 1 hour). We are currently trying to get more details and will come back to you. But thanks for your support so far, BR/Mechthild From: Neale Ranns <ne...@graphiant.com<mailto:ne...@graphiant.com>> Sent: Thursday, 24 June 2021 12:33 To: Mechthild Buescher <mechthild.buesc...@ericsson.com<mailto:mechthild.buesc...@ericsson.com>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: VRRP issue when using interface in a table Hi Mechthild, You’ll need to include: https://gerrit.fd.io/r/c/vpp/+/32298<https://protect2.fireeye.com/v1/url?k=43a4cf37-1c3ff635-43a48fac-869a14f4b08c-2b79380bed927e16&q=1&e=c3c90cbe-5ad1-4c2c-b574-111bb859ccb5&u=https%3A%2F%2Fgerrit.fd.io%2Fr%2Fc%2Fvpp%2F%2B%2F32298> /neale From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> on behalf of Mechthild Buescher via lists.fd.io<https://protect2.fireeye.com/v1/url?k=468e2c88-191515cb-468e6c13-861fcb972bfc-0eee98e2c4e12bfd&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2Flists.fd.io%2F> <mechthild.buescher=ericsson....@lists.fd.io<mailto:mechthild.buescher=ericsson....@lists.fd.io>> Date: Thursday, 24 June 2021 at 10:49 To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> Subject: [vpp-dev] VRRP issue when using interface in a table Hi all, we are using VPP on two nodes where we would like to run VRRP. This works fine if the VRRP VR interface is in fib 0 but if we but the interface into FIB table 1 instead, VRRP is not working correctly anymore. Can you please help? Our setup: • 2 nodes with VPP on each node and one DPDK interface (we reduced the config to isolate the issue) connected to each VPP • a switch between the nodes which just forwards the traffic, so that it’s like a peer-2-peer connection The VPP version is (both nodes): vpp# show version vpp v21.01.0-6~gf70123b2c built by suse on SUSE at 2021-05-06T12:18:31 vpp# show version verbose Version: v21.01.0-6~gf70123b2c Compiled by: suse Compile host: SUSE Compile date: 2021-05-06T12:18:31 Compile location: /root/vpp-sp/vpp Compiler: GCC 7.5.0 Current PID: 6677 The VPP config uses the DPDK interface (both nodes): vpp# show hardware-interfaces Name Idx Link Hardware Ext-0 1 up Ext-0 Link speed: 10 Gbps Ethernet address e4:43:4b:ed:59:10 Intel X710/XL710 Family carrier up full duplex mtu 9206 flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum Devargs: rx: queues 1 (max 192), desc 1024 (min 64 max 4096 align 32) tx: queues 3 (max 192), desc 1024 (min 64 max 4096 align 32) pci: device 8086:1572 subsystem 1028:1f9c address 0000:17:00.00 numa 0 max rx packet len: 9728 promiscuous: unicast off all-multicast on vlan offload: strip off filter off qinq off rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum qinq-strip outer-ipv4-cksum vlan-filter vlan-extend jumbo-frame scatter keep-crc rss-hash rx offload active: ipv4-cksum jumbo-frame scatter tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum tcp-tso outer-ipv4-cksum qinq-insert vxlan-tnl-tso gre-tnl-tso ipip-tnl-tso geneve-tnl-tso multi-segs mbuf-fast-free tx offload active: udp-cksum tcp-cksum multi-segs rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv6-frag ipv6-tcp ipv6-udp ipv6-sctp ipv6-other l2-payload rss active: none tx burst mode: Scalar rx burst mode: Vector AVX2 Scattered The VRRP configs are (MASTER): set interface state Ext-0 up set interface ip address Ext-0 192.168.61.52/25<https://protect2.fireeye.com/v1/url?k=efa88263-b033bb20-efa8c2f8-861fcb972bfc-550986599dc3ddee&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.52%2F25> vrrp vr add Ext-0 vr_id 61 priority 200 no_preempt accept_mode 192.168.61.50 and on the system under test (SUT): ip table add 1 set interface ip table Ext-0 1 set interface state Ext-0 up set interface ip address Ext-0 192.168.61.51/25<https://protect2.fireeye.com/v1/url?k=bf18bf02-e0838641-bf18ff99-861fcb972bfc-e230c27f8888cb49&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.51%2F25> vrrp vr add Ext-0 vr_id 61 priority 100 no_preempt accept_mode 192.168.61.50 On the MASTER, we started VRRP with: vrrp proto start Ext-0 vr_id 61 so that it has: vpp# show vrrp vr [0] sw_if_index 1 VR ID 61 IPv4 state Master flags: preempt no accept yes unicast no priority: configured 200 adjusted 200 timers: adv interval 100 master adv 100 skew 21 master down 321 virtual MAC 00:00:5e:00:01:3d addresses 192.168.61.50 peer addresses tracked interfaces On the SUT, we did not yet start VRRP, so we see: vpp# show vrrp vr [0] sw_if_index 1 VR ID 61 IPv4 state Initialize flags: preempt no accept yes unicast no priority: configured 100 adjusted 100 timers: adv interval 100 master adv 0 skew 0 master down 0 virtual MAC 00:00:5e:00:01:3d addresses 192.168.61.50 peer addresses tracked interfaces Here I see already that something is going wrong as the VRRP packets are not reaching vrrp4-input: vpp# show errors Count Node Reason Severity 5 dpdk-input no error error 138 ip4-local ip4 source lookup miss error (If we configure SUT similar to the MASTER, ie interface in FIB 0, I can see vrrp4-input at this point) The trace of dpdk-input gives: Packet 1 00:00:57:644818: dpdk-input Ext-0 rx queue 0 buffer 0x9b7ec: current data 0, length 60, buffer-pool 0, ref-count 1, totlen-nifb 0, trace handle 0x1000000 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 0, nb_segs 1, pkt_len 60 buf_len 2176, data_len 60, ol_flags 0x180, data_off 128, phys_addr 0x26dfb80 packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 Packet Offload Flags PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid Packet Types RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without extension headers RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet IP4: 00:00:5e:00:01:3d -> 01:00:5e:00:00:12 VRRP: 192.168.61.50 -> 224.0.0.18 tos 0x00, ttl 255, length 32, checksum 0xdd80 dscp CS0 ecn NON_ECN fragment id 0x0000 00:00:57:644832: ethernet-input frame: flags 0x3, hw-if-index 1, sw-if-index 1 IP4: 00:00:5e:00:01:3d -> 01:00:5e:00:00:12 00:00:57:644840: ip4-input-no-checksum VRRP: 192.168.61.50 -> 224.0.0.18 tos 0x00, ttl 255, length 32, checksum 0xdd80 dscp CS0 ecn NON_ECN fragment id 0x0000 00:00:57:644843: ip4-mfib-forward-lookup fib 1 entry 11 00:00:57:644846: ip4-mfib-forward-rpf entry 11 itf 1 flags Accept, 00:00:57:644848: ip4-replicate replicate: 8 via [@1]: dpo-receive 00:00:57:644850: ip4-local VRRP: 192.168.61.50 -> 224.0.0.18 tos 0x00, ttl 255, length 32, checksum 0xdd80 dscp CS0 ecn NON_ECN fragment id 0x0000 00:00:57:644854: ip4-drop VRRP: 192.168.61.50 -> 224.0.0.18 tos 0x00, ttl 255, length 32, checksum 0xdd80 dscp CS0 ecn NON_ECN fragment id 0x0000 00:00:57:644855: error-drop rx:Ext-0 00:00:57:644856: drop ip4-local: ip4 source lookup miss And the FIB table 1: vpp# show ip fib table 1 ipv4-VRF:1, fib_index:1, flow hash:[src dst sport dport proto ] epoch:0 flags:none locks:[CLI:2, ] 0.0.0.0/0<https://protect2.fireeye.com/v1/url?k=210a7193-7e9148d0-210a3108-861fcb972bfc-4508d6b3f86ae02e&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F0.0.0.0%2F0> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:8 buckets:1 uRPF:7 to:[0:0]] [0] [@0]: dpo-drop ip4 0.0.0.0/32<https://protect2.fireeye.com/v1/url?k=ce6e296c-91f5102f-ce6e69f7-861fcb972bfc-ab128b8eb6a4eda9&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F0.0.0.0%2F32> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:8 to:[0:0]] [0] [@0]: dpo-drop ip4 192.168.61.0/32<https://protect2.fireeye.com/v1/url?k=7e26a205-21bd9b46-7e26e29e-861fcb972bfc-b0a298ab67402bbe&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.0%2F32> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:14 buckets:1 uRPF:14 to:[0:0]] [0] [@0]: dpo-drop ip4 192.168.61.0/25<https://protect2.fireeye.com/v1/url?k=39f2a14f-6669980c-39f2e1d4-861fcb972bfc-0648016663148a5b&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.0%2F25> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:13 to:[0:0]] [0] [@4]: ipv4-glean: [src:192.168.61.0/25<https://protect2.fireeye.com/v1/url?k=474daa98-18d693db-474dea03-861fcb972bfc-b5e61bbc266b71d4&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.0%2F25>] Ext-0: mtu:9000 next:1 ffffffffffffe4434be52a100806 192.168.61.51/32<https://protect2.fireeye.com/v1/url?k=fb0f88f1-a494b1b2-fb0fc86a-861fcb972bfc-4b3a4016c7b2bcbe&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.51%2F32> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:16 buckets:1 uRPF:18 to:[0:0]] [0] [@2]: dpo-receive: 192.168.61.51 on Ext-0 192.168.61.127/32<https://protect2.fireeye.com/v1/url?k=de8801b7-811338f4-de88412c-861fcb972bfc-d1429965e56b72b7&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.127%2F32> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:15 buckets:1 uRPF:16 to:[0:0]] [0] [@0]: dpo-drop ip4 224.0.0.0/4<https://protect2.fireeye.com/v1/url?k=07be7d3f-5825447c-07be3da4-861fcb972bfc-78c5371f1b10ddad&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F224.0.0.0%2F4> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:10 to:[0:0]] [0] [@0]: dpo-drop ip4 240.0.0.0/4<https://protect2.fireeye.com/v1/url?k=8da2178e-d2392ecd-8da25715-861fcb972bfc-3fb783c1d6d13a20&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F240.0.0.0%2F4> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:9 to:[0:0]] [0] [@0]: dpo-drop ip4 255.255.255.255/32<https://protect2.fireeye.com/v1/url?k=e21bea85-bd80d3c6-e21baa1e-861fcb972bfc-4c3d6cb628d71f5f&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F255.255.255.255%2F32> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:11 to:[0:0]] [0] [@0]: dpo-drop ip4 If I read the trace correctly, ip4-mfib-forward-lookup goes into index 11 instead of index 13 and therefore is not reaching local, vrrp4-input. (Again, if I configure SUT using FIB 0, I can see that ip4-mfib-forward-lookup is pointing to the FIB entry 192.168.61.0/25<https://protect2.fireeye.com/v1/url?k=335376a4-6cc84fe7-3353363f-861fcb972bfc-cd2e8d948d75f3f2&q=1&e=a84fe3d4-3e6b-4dc3-be3e-b200c6641169&u=http%3A%2F%2F192.168.61.0%2F25> and then reaches local, vrrp4-input) If I start VR at this point in time (vrrp proto start Ext-0 vr_id 61), VRRP is changing to BACKUP, then immediately to MASTER while the peer stays on MASTER. So, we have MASTER/MASTER. And ‘show errors’ still doesn’t list vrrp4-input. Is this a bug in VPP or is VRRP not supported with FIBs other than 0? Do you have a suggestion how to fix/solve this? Thank you and best regards, Mechthild Buescher
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19712): https://lists.fd.io/g/vpp-dev/message/19712 Mute This Topic: https://lists.fd.io/mt/83756732/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
