Thanks Dave.
I did this, and as expected, the L2-forwarding by container is always reaching
only memif1/0 and never on memif2/0. I have attached the trace for 3 packets.
We see memif-input only on memif1/0.
That is, the packets from slave net_memif2 in container, is reaching the master
memif1/0 in host instead of master memif2/0 in host – Which is NOT correct.
I have checked the parameters of dpdk l2fwd application in container;
--vdev=net_memif1,socket=/run/memif-13a02ae0641b-net1.sock,role=slave
--vdev=net_memif2,socket=/run/memif-13a02ae0641b-net2.sock
And corresponding memif info on vpp in host. ( details pasted below)
memif1/0 - /run/memif-13a02ae0641b-net1.sock
memif2/0 - /run/memif-13a02ae0641b-net2.sock
I suppose this setup of 2 memif (slaves) interfaces in container using 2 memif
(master) interfaces in host is supported.
Any suggestion on what I should try next, please?
Regards
Sridhar
--- memif output---
vpp# show memif
sockets
id listener filename
2 yes (1) /run/memif-13a02ae0641b-net2.sock
0 no /run/vpp/memif.sock
1 yes (1) /run/memif-13a02ae0641b-net1.sock
interface memif1/0
remote-name "DPDK 19.08.0"
remote-interface "(null)"
socket-id 1 id 0 mode ethernet
flags admin-up connected
listener-fd 43 conn-fd 42
num-s2m-rings 1 num-m2s-rings 1 buffer-size 0 num-regions 1
region 0 size 4227328 fd 46
master-to-slave ring 0:
region 0 offset 16512 ring-size 1024 int-fd 50
head 15106 tail 14082 flags 0x0001 interrupts 0
slave-to-master ring 0:
region 0 offset 0 ring-size 1024 int-fd 48
head 34023 tail 34023 flags 0x0001 interrupts 0
interface memif2/0
remote-name "DPDK 19.08.0"
remote-interface "(null)"
socket-id 2 id 0 mode ethernet
flags admin-up connected
listener-fd 44 conn-fd 45
num-s2m-rings 1 num-m2s-rings 1 buffer-size 0 num-regions 1
region 0 size 4227328 fd 47
master-to-slave ring 0:
region 0 offset 16512 ring-size 1024 int-fd 51
head 12201 tail 11177 flags 0x0001 interrupts 0
slave-to-master ring 0:
region 0 offset 0 ring-size 1024 int-fd 49
head 0 tail 0 flags 0x0001 interrupts 0
From: Dave Barach (dbarach) <[email protected]>
Sent: 28 June 2020 18:54
To: Rao, Sridhar <[email protected]>; [email protected]
Subject: RE: [vpp-dev] Userspace CNI - VPP Memif Forwarding Issue.
[Warning] This email comes from an external source. Be careful of any embedded
links and attachments.
“trace add [dpdk-input|memif-input] <nnn>” ... run failing case ... “show
trace”. Should explain what’s happening.
Dave
From: [email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>> On Behalf Of Sridhar K. N. Rao
Sent: Saturday, June 27, 2020 9:16 AM
To: [email protected]<mailto:[email protected]>
Subject: [vpp-dev] Userspace CNI - VPP Memif Forwarding Issue.
Hello,
We (OPNFV-VSPERF) are working container networking performance testing. As part
of this we are testing Userspace CNI with VPP. The Pod runs a dpdk l2fwd
application. We are stuck with a forwarding issue, and any suggestion would be
extremely helpful. I checked with CNI team, and they see no issues.
I have provided the summary here. All the details (vpp configuration, version,
l2fwd run in pod, patch-info, interface info, statistics, pod-interface-stats,
failed-scenarios, etc) can be found here -
https://github.com/opensource-tnbt/cnb/blob/master/vpp-forwarding-issue.md
We have a very simple 2-node setup. The 2 nodes are connect directly by 2
10-gig interfaces for data traffic.
On one node we have T-Rex Running, whereas on the other node we have VPP and
DPDK-pod running.
So, the traffic will flow like this:
Direction1:
Trex0-phy ---> TenGigabitEthernet6/0/0 ---> memif1/0 ---> net_memif1
DPDK l2fwd app in pod: net_memif1 ---> net_memif2
Trex1-phy <--- TenGigabitEthernet6/0/1 <--- memif2/0 <--- net_memif2
Direction2:
Trex0-phy ---> TenGigabitEthernet6/0/0 ---> memif2/0 ---> net_memif2
DPDK l2fwd app in pod: net_memif2 ---> net_memif1
Trex1-phy <--- TenGigabitEthernet6/0/1 <--- memif1/0 <--- net_memif1
L2pathes are configured
Direction:1
sudo vppctl test l2patch rx TenGigabitEthernet6/0/0 tx memif1/0
sudo vppctl test l2patch rx memif2/0 tx TenGigabitEthernet6/0/1
Direction:2
sudo vppctl test l2patch rx TenGigabitEthernet6/0/1 tx memif2/0
sudo vppctl test l2patch rx memif1/0 tx TenGigabitEthernet6/0/0
When The traffic runs, All the traffic somehow ends up only on memif1/0 – like
this:
vpp# show interface
Name Idx State MTU (L3/IP4/IP6/MPLS)
Counter Count
TenGigabitEthernet6/0/0 1 up 9000/0/0/0 rx packets
70598102
rx bytes
79993565352
tx packets
616631302
tx bytes
696194649320
tx-error
458997285
TenGigabitEthernet6/0/1 2 up 9000/0/0/0 rx packets
70587607
rx bytes
79981831588
local0 0 down 0/0/0/0
memif1/0 3 up 9000/0/0/0 rx packets
616631302
rx bytes
696194649320
tx packets
70598102
tx bytes
79993565352
memif2/0 4 up 9000/0/0/0 tx packets
70587607
tx bytes
79981831588
We have tried the following (The outputs of all these are in the link above).
1. Testing 1-Direction at a time.
2. Swapping the Phy -> memif mapping in the patch
3. Using Xconnect instead of l2patch.
4. Changing the mode from Interface to bridge in CNI.
None of these helped to debug the issue.
Looking forward for an advice.
Cheers,
Sridhar
vpp# show interface
Name Idx State MTU (L3/IP4/IP6/MPLS)
Counter Count
TenGigabitEthernet6/0/0 1 up 9000/0/0/0
TenGigabitEthernet6/0/1 2 up 9000/0/0/0
local0 0 down 0/0/0/0
memif1/0 3 up 9000/0/0/0
memif2/0 4 up 9000/0/0/0
vpp# show trace
------------------- Start of thread 0 vpp_main -------------------
No packets in trace buffer
------------------- Start of thread 1 vpp_wk_0 -------------------
Packet 1
00:07:52:959683: memif-input
memif: hw_if_index 3 next-index 6
slot: ring 0
00:07:52:959716: l2-patch
L2_PATCH: rx 3 tx 1
00:07:52:959718: TenGigabitEthernet6/0/0-output
TenGigabitEthernet6/0/0
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x01, ttl 64, length 1006, checksum 0xc048
fragment id 0xffff
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
00:07:52:959720: TenGigabitEthernet6/0/0-tx
TenGigabitEthernet6/0/0 tx queue 1
buffer 0xfb1afb: current data 0, length 1020, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace 0x0
PKT MBUF: port 65535, nb_segs 1, pkt_len 1020
buf_len 2176, data_len 1020, ol_flags 0x0, data_off 128, phys_addr
0xbec6bf40
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x01, ttl 64, length 1006, checksum 0xc048
fragment id 0xffff
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
Packet 2
00:07:52:960602: memif-input
memif: hw_if_index 3 next-index 6
slot: ring 0
00:07:52:960614: l2-patch
L2_PATCH: rx 3 tx 1
00:07:52:960615: TenGigabitEthernet6/0/0-output
TenGigabitEthernet6/0/0
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
00:07:52:960616: TenGigabitEthernet6/0/0-tx
TenGigabitEthernet6/0/0 tx queue 1
buffer 0xfb1900: current data 0, length 1020, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace 0x1
PKT MBUF: port 65535, nb_segs 1, pkt_len 1020
buf_len 2176, data_len 1020, ol_flags 0x0, data_off 128, phys_addr
0xbec64080
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
Packet 3
00:07:52:960602: memif-input
memif: hw_if_index 3 next-index 6
slot: ring 0
00:07:52:960614: l2-patch
L2_PATCH: rx 3 tx 1
00:07:52:960615: TenGigabitEthernet6/0/0-output
TenGigabitEthernet6/0/0
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
00:07:52:960616: TenGigabitEthernet6/0/0-tx
TenGigabitEthernet6/0/0 tx queue 1
buffer 0xfb1927: current data 0, length 1020, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace 0x2
PKT MBUF: port 65535, nb_segs 1, pkt_len 1020
buf_len 2176, data_len 1020, ol_flags 0x0, data_off 128, phys_addr
0xbec64a40
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
------------------- Start of thread 2 vpp_wk_1 -------------------
Packet 1
00:07:52:978568: dpdk-input
TenGigabitEthernet6/0/1 rx queue 1
buffer 0xfd3e81: current data 0, length 1020, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace 0x0
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 1, nb_segs 1, pkt_len 1020
buf_len 2176, data_len 1020, ol_flags 0x8a, data_off 128, phys_addr
0xbf4fa0c0
packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x2afd4f2d fdir.hi 0x0 fdir.lo 0x2afd4f2d
Packet Offload Flags
PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
PKT_RX_L4_CKSUM_BAD (0x0008) L4 cksum of RX pkt. is not OK
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
Packet Types
RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
RTE_PTYPE_L4_UDP (0x0200) UDP packet
IP4: 00:1e:67:fd:f8:39 -> 00:1e:67:fd:f8:38
UDP: 90.90.90.90 -> 1.1.1.1
tos 0x01, ttl 64, length 1006, checksum 0xc048
fragment id 0xffff
UDP: 3001 -> 3000
length 986, checksum 0x0bf5
00:07:52:978573: l2-patch
L2_PATCH: rx 2 tx 4
00:07:52:978576: memif2/0-output
memif2/0 l4-cksum-computed l4-cksum-correct
IP4: 00:1e:67:fd:f8:39 -> 00:1e:67:fd:f8:38
UDP: 90.90.90.90 -> 1.1.1.1
tos 0x01, ttl 64, length 1006, checksum 0xc048
fragment id 0xffff
UDP: 3001 -> 3000
length 986, checksum 0x0bf5
Packet 2
00:07:52:979468: dpdk-input
TenGigabitEthernet6/0/1 rx queue 1
buffer 0xfd3e5a: current data 0, length 1020, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace 0x1
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 1, nb_segs 1, pkt_len 1020
buf_len 2176, data_len 1020, ol_flags 0x182, data_off 128, phys_addr
0xbf4f9700
packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x2afd4f2d fdir.hi 0x0 fdir.lo 0x2afd4f2d
Packet Offload Flags
PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
Packet Types
RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
RTE_PTYPE_L4_UDP (0x0200) UDP packet
IP4: 00:1e:67:fd:f8:39 -> 00:1e:67:fd:f8:38
UDP: 90.90.90.90 -> 1.1.1.1
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3001 -> 3000
length 986, checksum 0x0bf5
00:07:52:979469: l2-patch
L2_PATCH: rx 2 tx 4
00:07:52:979470: memif2/0-output
memif2/0 l4-cksum-computed l4-cksum-correct
IP4: 00:1e:67:fd:f8:39 -> 00:1e:67:fd:f8:38
UDP: 90.90.90.90 -> 1.1.1.1
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3001 -> 3000
length 986, checksum 0x0bf5
Packet 3
00:07:52:979471: dpdk-input
TenGigabitEthernet6/0/1 rx queue 1
buffer 0xfd3e33: current data 0, length 1020, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace 0x2
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 1, nb_segs 1, pkt_len 1020
buf_len 2176, data_len 1020, ol_flags 0x182, data_off 128, phys_addr
0xbf4f8d40
packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x2afd4f2d fdir.hi 0x0 fdir.lo 0x2afd4f2d
Packet Offload Flags
PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
Packet Types
RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
RTE_PTYPE_L4_UDP (0x0200) UDP packet
IP4: 00:1e:67:fd:f8:39 -> 00:1e:67:fd:f8:38
UDP: 90.90.90.90 -> 1.1.1.1
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3001 -> 3000
length 986, checksum 0x0bf5
00:07:52:979472: l2-patch
L2_PATCH: rx 2 tx 4
00:07:52:979472: memif2/0-output
memif2/0 l4-cksum-computed l4-cksum-correct
IP4: 00:1e:67:fd:f8:39 -> 00:1e:67:fd:f8:38
UDP: 90.90.90.90 -> 1.1.1.1
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3001 -> 3000
length 986, checksum 0x0bf5
------------------- Start of thread 3 vpp_wk_2 -------------------
Packet 1
00:07:52:959554: dpdk-input
TenGigabitEthernet6/0/0 rx queue 2
buffer 0xff1197: current data 0, length 1020, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace 0x0
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 0, nb_segs 1, pkt_len 1020
buf_len 2176, data_len 1020, ol_flags 0x8a, data_off 128, phys_addr
0xbfc46640
packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0xb4b7332 fdir.hi 0x0 fdir.lo 0xb4b7332
Packet Offload Flags
PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
PKT_RX_L4_CKSUM_BAD (0x0008) L4 cksum of RX pkt. is not OK
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
Packet Types
RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
RTE_PTYPE_L4_UDP (0x0200) UDP packet
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x01, ttl 64, length 1006, checksum 0xc048
fragment id 0xffff
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
00:07:52:959574: l2-patch
L2_PATCH: rx 1 tx 3
00:07:52:959583: memif1/0-output
memif1/0 l4-cksum-computed l4-cksum-correct
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x01, ttl 64, length 1006, checksum 0xc048
fragment id 0xffff
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
Packet 2
00:07:52:960545: dpdk-input
TenGigabitEthernet6/0/0 rx queue 2
buffer 0xff1170: current data 0, length 1020, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace 0x1
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 0, nb_segs 1, pkt_len 1020
buf_len 2176, data_len 1020, ol_flags 0x182, data_off 128, phys_addr
0xbfc45c80
packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0xb4b7332 fdir.hi 0x0 fdir.lo 0xb4b7332
Packet Offload Flags
PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
Packet Types
RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
RTE_PTYPE_L4_UDP (0x0200) UDP packet
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
00:07:52:960546: l2-patch
L2_PATCH: rx 1 tx 3
00:07:52:960547: memif1/0-output
memif1/0 l4-cksum-computed l4-cksum-correct
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
Packet 3
00:07:52:960548: dpdk-input
TenGigabitEthernet6/0/0 rx queue 2
buffer 0xff1149: current data 0, length 1020, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace 0x2
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 0, nb_segs 1, pkt_len 1020
buf_len 2176, data_len 1020, ol_flags 0x182, data_off 128, phys_addr
0xbfc452c0
packet_type 0x211 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0xb4b7332 fdir.hi 0x0 fdir.lo 0xb4b7332
Packet Offload Flags
PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
Packet Types
RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
RTE_PTYPE_L4_UDP (0x0200) UDP packet
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
00:07:52:960549: l2-patch
L2_PATCH: rx 1 tx 3
00:07:52:960550: memif1/0-output
memif1/0 l4-cksum-computed l4-cksum-correct
IP4: 00:1e:67:fd:f8:38 -> 00:1e:67:fd:f8:39
UDP: 1.1.1.1 -> 90.90.90.90
tos 0x00, ttl 64, length 1006, checksum 0xc048
fragment id 0x0001
UDP: 3000 -> 3001
length 986, checksum 0x0bf5
------------------- Start of thread 4 vpp_wk_3 -------------------
No packets in trace buffer
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16838): https://lists.fd.io/g/vpp-dev/message/16838
Mute This Topic: https://lists.fd.io/mt/75152055/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
