Hi Florin,
By using VCL library in an UDP receiver application, I am able to receive
only 2 Mbps traffic. On increasing the traffic, I see Rx FIFO full error
and application stopped receiving the traffic from the session layer.
Whereas, with TCP I can easily achieve 15Gbps throughput without tuning any
DPDK parameter. UDP tx also looks fine. From an host application I can
send ~5Gbps without any issue.
I am running VPP( stable/2001 code) on RHEL8 server using Mellanox 100G
(MLNX5) adapters.
Please advise if I can use VCL library to receive high throughput UDP
traffic ( in Gbps). I would be running multiple instances of host
application to receive data ( ~50-60 Gbps).
I also tried by increasing the Rx FIFO size to 16MB but did not help much.
The host application is just throwing the received packets , it is not
doing any packet processing.
[root@orc01 vcl_test]# VCL_DEBUG=2 ./udp6_server_vcl
VCL<20201>: configured VCL debug level (2) from VCL_DEBUG!
VCL<20201>: allocated VCL heap = 0x7f39a17ab010, size 268435456 (0x10000000)
VCL<20201>: configured rx_fifo_size 4000000 (0x3d0900)
VCL<20201>: configured tx_fifo_size 4000000 (0x3d0900)
VCL<20201>: configured app_scope_local (1)
VCL<20201>: configured app_scope_global (1)
VCL<20201>: configured api-socket-name (/tmp/vpp-api.sock)
VCL<20201>: completed parsing vppcom config!
vppcom_connect_to_vpp:480: vcl<20201:0>: app (udp6_server) is connected to
VPP!
vppcom_app_create:1104: vcl<20201:0>: sending session enable
vppcom_app_create:1112: vcl<20201:0>: sending app attach
vppcom_app_create:1121: vcl<20201:0>: app_name 'udp6_server',
my_client_index 256 (0x100)
vppcom_epoll_create:2439: vcl<20201:0>: Created vep_idx 0
vppcom_session_create:1179: vcl<20201:0>: created session 1
vppcom_session_bind:1317: vcl<20201:0>: session 1 handle 1: binding to
local IPv6 address fd0d:edc4:ffff:2001::203 port 8092, proto UDP
vppcom_session_listen:1349: vcl<20201:0>: session 1: sending vpp listen
request...
vcl_session_bound_handler:604: vcl<20201:0>: session 1 [0x1]: listen
succeeded!
vppcom_epoll_ctl:2541: vcl<20201:0>: EPOLL_CTL_ADD: vep_sh 0, sh 1, events
0x1, data 0x1!
vppcom_session_create:1179: vcl<20201:0>: created session 2
vppcom_session_bind:1317: vcl<20201:0>: session 2 handle 2: binding to
local IPv6 address fd0d:edc4:ffff:2001::203 port 8093, proto UDP
vppcom_session_listen:1349: vcl<20201:0>: session 2: sending vpp listen
request...
vcl_session_app_add_segment_handler:765: vcl<20201:0>: mapped new segment
'20190-2' size 134217728
vcl_session_bound_handler:604: vcl<20201:0>: session 2 [0x2]: listen
succeeded!
vppcom_epoll_ctl:2541: vcl<20201:0>: EPOLL_CTL_ADD: vep_sh 0, sh 2, events
0x1, data 0x2!
vpp# sh session verbose 2
[#0][U] fd0d:edc4:ffff:2001::203:8092->:::0
Rx fifo: cursize 3999125 nitems 3999999 has_event 1
head 2554045 tail 2553170 segment manager 1
vpp session 0 thread 0 app session 1 thread 0
ooo pool 0 active elts newest 4294967295
Tx fifo: cursize 0 nitems 3999999 has_event 0
head 0 tail 0 segment manager 1
vpp session 0 thread 0 app session 1 thread 0
ooo pool 0 active elts newest 0
[#0][U] fd0d:edc4:ffff:2001::203:8093->:::0
Rx fifo: cursize 0 nitems 3999999 has_event 0
head 0 tail 0 segment manager 2
vpp session 1 thread 0 app session 2 thread 0
ooo pool 0 active elts newest 0
Tx fifo: cursize 0 nitems 3999999 has_event 0
head 0 tail 0 segment manager 2
vpp session 1 thread 0 app session 2 thread 0
ooo pool 0 active elts newest 0
Thread 0: active sessions 2
[root@orc01 vcl_test]# cat /etc/vpp/vcl.conf
vcl {
rx-fifo-size 4000000
tx-fifo-size 4000000
app-scope-local
app-scope-global
api-socket-name /tmp/vpp-api.sock
}
[root@orc01 vcl_test]#
------------------- Start of thread 0 vpp_main -------------------
Packet 1
00:09:53:445025: dpdk-input
HundredGigabitEthernet12/0/0 rx queue 0
buffer 0x88078: current data 0, length 1516, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace handle 0x0
ext-hdr-valid
l4-cksum-computed l4-cksum-correct
PKT MBUF: port 0, nb_segs 1, pkt_len 1516
buf_len 2176, data_len 1516, ol_flags 0x180, data_off 128, phys_addr
0x75601e80
packet_type 0x2e1 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
Packet Offload Flags
PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
Packet Types
RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
RTE_PTYPE_L3_IPV6_EXT_UNKNOWN (0x00e0) IPv6 packet with or without
extension headers
RTE_PTYPE_L4_UDP (0x0200) UDP packet
IP6: b8:83:03:79:9f:e4 -> b8:83:03:79:af:8c 802.1q vlan 2001
UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203
tos 0x00, flow label 0x0, hop limit 64, payload length 1458
UDP: 56944 -> 8092
length 1458, checksum 0xb22d
00:09:53:445028: ethernet-input
frame: flags 0x3, hw-if-index 2, sw-if-index 2
IP6: b8:83:03:79:9f:e4 -> b8:83:03:79:af:8c 802.1q vlan 2001
00:09:53:445029: ip6-input
UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203
tos 0x00, flow label 0x0, hop limit 64, payload length 1458
UDP: 56944 -> 8092
length 1458, checksum 0xb22d
00:09:53:445031: ip6-lookup
fib 0 dpo-idx 6 flow hash: 0x00000000
UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203
tos 0x00, flow label 0x0, hop limit 64, payload length 1458
UDP: 56944 -> 8092
length 1458, checksum 0xb22d
00:09:53:445032: ip6-local
UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203
tos 0x00, flow label 0x0, hop limit 64, payload length 1458
UDP: 56944 -> 8092
length 1458, checksum 0xb22d
00:09:53:445032: ip6-udp-lookup
UDP: src-port 56944 dst-port 8092
00:09:53:445033: udp6-input
UDP_INPUT: connection 0, disposition 5, thread 0
thanks,
-Raj
On Wed, Jan 15, 2020 at 4:09 PM Raj Kumar via Lists.Fd.Io <raj.gautam25=
gmail....@lists.fd.io> wrote:
> Hi Florin,
> Yes, [2] patch resolved the IPv6/UDP receiver issue.
> Thanks! for your help.
>
> thanks,
> -Raj
>
> On Tue, Jan 14, 2020 at 9:35 PM Florin Coras <fcoras.li...@gmail.com>
> wrote:
>
>> Hi Raj,
>>
>> First of all, with this [1], the vcl test app/client can establish a udpc
>> connection. Note that udp will most probably lose packets, so large
>> exchanges with those apps may not work.
>>
>> As for the second issue, does [2] solve it?
>>
>> Regards,
>> Florin
>>
>> [1] https://gerrit.fd.io/r/c/vpp/+/24332
>> [2] https://gerrit.fd.io/r/c/vpp/+/24334
>>
>> On Jan 14, 2020, at 12:59 PM, Raj Kumar <raj.gauta...@gmail.com> wrote:
>>
>> Hi Florin,
>> Thanks! for the reply.
>>
>> I realized the issue with the non-connected case. For receiving
>> datagrams, I was using recvfrom() with DONOT_WAIT flag because of
>> that vppcom_session_recvfrom() api was failing. It expects either 0 or
>> MSG_PEEK flag.
>> if (flags == 0)
>> rv = vppcom_session_read (session_handle, buffer, buflen);
>> else if (flags & MSG_PEEK) 0x2
>> rv = vppcom_session_peek (session_handle, buffer, buflen);
>> else
>> {
>> VDBG (0, "Unsupport flags for recvfrom %d", flags);
>> return VPPCOM_EAFNOSUPPORT;
>> }
>>
>> I changed the flag to 0 in recvfrom() , after that UDP rx is working
>> fine but only for IPv4.
>>
>> I am facing a different issue with IPv6/UDP receiver. I am getting "no
>> listener for dst port" error.
>>
>> Please let me know if I am doing something wrong.
>> Here are the traces : -
>>
>> [root@orc01 testcode]# VCL_DEBUG=2 LDP_DEBUG=2
>> LD_PRELOAD=/opt/vpp/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so
>> VCL_CONFIG=/etc/vpp/vcl.cfg ./udp6_rx
>> VCL<1164>: configured VCL debug level (2) from VCL_DEBUG!
>> VCL<1164>: allocated VCL heap = 0x7ff877439010, size 268435456
>> (0x10000000)
>> VCL<1164>: configured rx_fifo_size 4000000 (0x3d0900)
>> VCL<1164>: configured tx_fifo_size 4000000 (0x3d0900)
>> VCL<1164>: configured app_scope_local (1)
>> VCL<1164>: configured app_scope_global (1)
>> VCL<1164>: configured api-socket-name (/tmp/vpp-api.sock)
>> VCL<1164>: completed parsing vppcom config!
>> vppcom_connect_to_vpp:549: vcl<1164:0>: app (ldp-1164-app) is connected
>> to VPP!
>> vppcom_app_create:1067: vcl<1164:0>: sending session enable
>> vppcom_app_create:1075: vcl<1164:0>: sending app attach
>> vppcom_app_create:1084: vcl<1164:0>: app_name 'ldp-1164-app',
>> my_client_index 0 (0x0)
>> ldp_init:209: ldp<1164>: configured LDP debug level (2) from env var
>> LDP_DEBUG!
>> ldp_init:282: ldp<1164>: LDP initialization: done!
>> ldp_constructor:2490: LDP<1164>: LDP constructor: done!
>> socket:974: ldp<1164>: calling vls_create: proto 1 (UDP), is_nonblocking 0
>> vppcom_session_create:1142: vcl<1164:0>: created session 0
>> bind:1086: ldp<1164>: fd 32: calling vls_bind: vlsh 0, addr
>> 0x7fff9a93efe0, len 28
>> vppcom_session_bind:1280: vcl<1164:0>: session 0 handle 0: binding to
>> local IPv6 address :: port 8092, proto UDP
>> vppcom_session_listen:1312: vcl<1164:0>: session 0: sending vpp listen
>> request...
>> vcl_session_bound_handler:610: vcl<1164:0>: session 0 [0x1]: listen
>> succeeded!
>> bind:1102: ldp<1164>: fd 32: returning 0
>>
>> vpp# sh app server
>> Connection App Wrk
>> [0:0][CT:U] :::8092->:::0 ldp-1164-app[shm] 0
>> [#0][U] :::8092->:::0 ldp-1164-app[shm] 0
>>
>> vpp# sh err
>> Count Node Reason
>> 7 dpdk-input no error
>> 2606 ip6-udp-lookup no listener for dst port
>> 8 arp-reply ARP replies sent
>> 1 arp-disabled ARP Disabled on this
>> interface
>> 13 ip6-glean neighbor solicitations
>> sent
>> 2606 ip6-input valid ip6 packets
>> 4 ip6-local-hop-by-hop Unknown protocol ip6
>> local h-b-h packets dropped
>> 2606 ip6-icmp-error destination unreachable
>> response sent
>> 40 ip6-icmp-input valid packets
>> 1 ip6-icmp-input neighbor solicitations
>> from source not on link
>> 12 ip6-icmp-input neighbor solicitations
>> for unknown targets
>> 1 ip6-icmp-input neighbor advertisements
>> sent
>> 1 ip6-icmp-input neighbor advertisements
>> received
>> 40 ip6-icmp-input router advertisements
>> sent
>> 40 ip6-icmp-input router advertisements
>> received
>> 1 ip4-icmp-input echo replies sent
>> 89 lldp-input lldp packets received
>> on disabled interfaces
>> 1328 llc-input unknown llc ssap/dsap
>> vpp#
>>
>> vpp# show trace
>> ------------------- Start of thread 0 vpp_main -------------------
>> Packet 1
>>
>> 00:23:39:401354: dpdk-input
>> HundredGigabitEthernet12/0/0 rx queue 0
>> buffer 0x8894e: current data 0, length 1516, buffer-pool 0, ref-count
>> 1, totlen-nifb 0, trace handle 0x0
>> ext-hdr-valid
>> l4-cksum-computed l4-cksum-correct
>> PKT MBUF: port 0, nb_segs 1, pkt_len 1516
>> buf_len 2176, data_len 1516, ol_flags 0x180, data_off 128, phys_addr
>> 0x75025400
>> packet_type 0x2e1 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>> rss 0x0 fdir.hi 0x0 fdir.lo 0x0
>> Packet Offload Flags
>> PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
>> PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
>> Packet Types
>> RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
>> RTE_PTYPE_L3_IPV6_EXT_UNKNOWN (0x00e0) IPv6 packet with or without
>> extension headers
>> RTE_PTYPE_L4_UDP (0x0200) UDP packet
>> IP6: b8:83:03:79:9f:e4 -> b8:83:03:79:af:8c 802.1q vlan 2001
>> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203
>> tos 0x00, flow label 0x0, hop limit 64, payload length 1458
>> UDP: 60593 -> 8092
>> length 1458, checksum 0x0964
>> 00:23:39:401355: ethernet-input
>> frame: flags 0x3, hw-if-index 2, sw-if-index 2
>> IP6: b8:83:03:79:9f:e4 -> b8:83:03:79:af:8c 802.1q vlan 2001
>> 00:23:39:401356: ip6-input
>> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203
>> tos 0x00, flow label 0x0, hop limit 64, payload length 1458
>> UDP: 60593 -> 8092
>> length 1458, checksum 0x0964
>> 00:23:39:401357: ip6-lookup
>> fib 0 dpo-idx 5 flow hash: 0x00000000
>> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203
>> tos 0x00, flow label 0x0, hop limit 64, payload length 1458
>> UDP: 60593 -> 8092
>> length 1458, checksum 0x0964
>> 00:23:39:401361: ip6-local
>> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203
>> tos 0x00, flow label 0x0, hop limit 64, payload length 1458
>> UDP: 60593 -> 8092
>> length 1458, checksum 0x0964
>> 00:23:39:401362: ip6-udp-lookup
>> UDP: src-port 60593 dst-port 8092 (no listener)
>> 00:23:39:401362: ip6-icmp-error
>> UDP: fd0d:edc4:ffff:2001::201 -> fd0d:edc4:ffff:2001::203
>> tos 0x00, flow label 0x0, hop limit 64, payload length 1458
>> UDP: 60593 -> 8092
>> length 1458, checksum 0x0964
>> 00:23:39:401363: error-drop
>> rx:HundredGigabitEthernet12/0/0.2001
>> 00:23:39:401364: drop
>> ip6-input: valid ip6 packets
>>
>> vpp#
>>
>>
>> Thanks,
>> -Raj
>>
>>
>> On Tue, Jan 14, 2020 at 1:44 PM Florin Coras <fcoras.li...@gmail.com>
>> wrote:
>>
>>> Hi Raj,
>>>
>>> Session layer does support connection-less transports but udp does not
>>> raise accept notifications to vcl. UDPC might, but we haven’t tested udpc
>>> with vcl in a long time so it might not work properly.
>>>
>>> What was the problem you were hitting in the non-connected case?
>>>
>>> Regards,
>>> Florin
>>>
>>> > On Jan 14, 2020, at 7:13 AM, raj.gauta...@gmail.com wrote:
>>> >
>>> > Hi ,
>>> > I am trying some host application tests ( using LD_PRELOAD) . TCP rx
>>> and tx both work fine. UDP tx also works fine.
>>> > The issue is only with UDP rx . In some discussion it was mentioned
>>> that session layer does not support connection-less transports so protocols
>>> like udp still need to accept connections and only afterwards read from the
>>> fifos.
>>> > So, I changed the UDP receiver application to use listen() and
>>> accept() before read() . But , I am still having issue to make it run.
>>> > After I started, udp traffic from other server it seems to accept the
>>> connection but never returns from the vppcom_session_accept() function.
>>> > VPP release is 19.08.
>>> >
>>> > vpp# sh app server
>>> > Connection App
>>> Wrk
>>> > [0:0][CT:U] 0.0.0.0:8090->0.0.0.0:0 ldp-36646-app[shm]
>>> 0
>>> > [#0][U] 0.0.0.0:8090->0.0.0.0:0 ldp-36646-app[shm]
>>> 0
>>> > vpp#
>>> >
>>> >
>>> > [root@orc01 testcode]# VCL_DEBUG=2 LDP_DEBUG=2
>>> LD_PRELOAD=/opt/vpp/build-root/install-vpp-native/vpp/lib/libvcl_ldpreload.so
>>> VCL_CONFIG=/etc/vpp/vcl.cfg ./udp_rx
>>> > VCL<36646>: configured VCL debug level (2) from VCL_DEBUG!
>>> > VCL<36646>: allocated VCL heap = 0x7f77e5309010, size 268435456
>>> (0x10000000)
>>> > VCL<36646>: configured rx_fifo_size 4000000 (0x3d0900)
>>> > VCL<36646>: configured tx_fifo_size 4000000 (0x3d0900)
>>> > VCL<36646>: configured app_scope_local (1)
>>> > VCL<36646>: configured app_scope_global (1)
>>> > VCL<36646>: configured api-socket-name (/tmp/vpp-api.sock)
>>> > VCL<36646>: completed parsing vppcom config!
>>> > vppcom_connect_to_vpp:549: vcl<36646:0>: app (ldp-36646-app) is
>>> connected to VPP!
>>> > vppcom_app_create:1067: vcl<36646:0>: sending session enable
>>> > vppcom_app_create:1075: vcl<36646:0>: sending app attach
>>> > vppcom_app_create:1084: vcl<36646:0>: app_name 'ldp-36646-app',
>>> my_client_index 0 (0x0)
>>> > ldp_init:209: ldp<36646>: configured LDP debug level (2) from env var
>>> LDP_DEBUG!
>>> > ldp_init:282: ldp<36646>: LDP initialization: done!
>>> > ldp_constructor:2490: LDP<36646>: LDP constructor: done!
>>> > socket:974: ldp<36646>: calling vls_create: proto 1 (UDP),
>>> is_nonblocking 0
>>> > vppcom_session_create:1142: vcl<36646:0>: created session 0
>>> > Socket successfully created..
>>> > bind:1086: ldp<36646>: fd 32: calling vls_bind: vlsh 0, addr
>>> 0x7fff3f3c1040, len 16
>>> > vppcom_session_bind:1280: vcl<36646:0>: session 0 handle 0: binding to
>>> local IPv4 address 0.0.0.0 port 8090, proto UDP
>>> > vppcom_session_listen:1312: vcl<36646:0>: session 0: sending vpp
>>> listen request...
>>> > vcl_session_bound_handler:610: vcl<36646:0>: session 0 [0x1]: listen
>>> succeeded!
>>> > bind:1102: ldp<36646>: fd 32: returning 0
>>> > Socket successfully binded..
>>> > listen:2005: ldp<36646>: fd 32: calling vls_listen: vlsh 0, n 5
>>> > vppcom_session_listen:1308: vcl<36646:0>: session 0 [0x1]: already in
>>> listen state!
>>> > listen:2020: ldp<36646>: fd 32: returning 0
>>> > Server listening..
>>> > ldp_accept4:2043: ldp<36646>: listen fd 32: calling
>>> vppcom_session_accept: listen sid 0, ep 0x0, flags 0x3f3c0fc0
>>> > vppcom_session_accept:1478: vcl<36646:0>: discarded event: 0
>>> >
>>> >
>>>
>>>
>>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15199): https://lists.fd.io/g/vpp-dev/message/15199
Mute This Topic: https://lists.fd.io/mt/69694900/21656
Mute #vpp-hoststack: https://lists.fd.io/mk?hashtag=vpp-hoststack&subid=1480452
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
