Re: Private: Re: [vpp-dev] increase nginx performace using vpp host stack#nginx #vpp-hoststack

[Florin Coras]

As mentioned previously, that is not supported for connects currently.  

Regards,
Florin

> On Mar 31, 2022, at 9:44 PM, weizhen9...@163.com wrote:
> 
> Hi,
> I describe our scene in detail. We use nginx in vpp host stack as a proxy. 
> And we add some features in nginx. For example, nginx close upstream tcp 
> links actively and this causes a lot of TIME_WAIT states in nginx proxy when 
> we test the performance of nginx using vpp host stack. So we configure kernel 
> parameters to multiplex tcp port and recycle port quickly. But I can't find 
> vpp host stack parameters which is similar with kernel parameters.
> How can we configure vpp parameters to reuse tcp port and recycle port 
> quickly?
> Thanks.  
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21177): https://lists.fd.io/g/vpp-dev/message/21177
Mute This Topic: https://lists.fd.io/mt/90157992/21656
Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: Private: Re: [vpp-dev] increase nginx performace using vpp host stack#nginx #vpp-hoststack

[weizhen9612]

Hi,
I describe our scene in detail. We use nginx in vpp host stack as a proxy. And 
we add some features in nginx. For example, nginx close upstream tcp links 
actively and this causes a lot of TIME_WAIT states in nginx proxy when we test 
the performance of nginx using vpp host stack. So we configure kernel 
parameters to multiplex tcp port and recycle port quickly. But I can't find vpp 
host stack parameters which is similar with kernel parameters.
How can we configure vpp parameters to reuse tcp port and recycle port quickly?
Thanks.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21176): https://lists.fd.io/g/vpp-dev/message/21176
Mute This Topic: https://lists.fd.io/mt/90157992/21656
Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: Private: Re: [vpp-dev] increase nginx performace using vpp host stack#nginx #vpp-hoststack

[Florin Coras]

Hi,

Given that 20k connections are being actively opened, those on main thread, and 
40k are established, those on the workers, suggests that tcp runs out of ports 
for connects. If possible either increase the number of destination IPs for 
nginx or try “tcp src-address ip1-ip2” and pass in a range of consecutive ips 
that could be used the vpp as source ips.   

The two workers seem to be pretty lightly loaded since loops/2 is over 2M, 
i.e., they’re mostly spinning empty, without work. 

Regards,
Florin

> On Mar 31, 2022, at 8:21 PM, weizhen9...@163.com wrote:
> 
> Hi,
> 
> From this test results, can you see some errors? Why is the performance of 
> nginx using vpp low?
> 
> Thanks. 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21175): https://lists.fd.io/g/vpp-dev/message/21175
Mute This Topic: https://lists.fd.io/mt/90157992/21656
Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: Private: Re: [vpp-dev] increase nginx performace using vpp host stack#nginx #vpp-hoststack

[weizhen9612]

Hi,

>From this test results, can you see some errors? Why is the performance of 
>nginx using vpp low?

Thanks.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21174): https://lists.fd.io/g/vpp-dev/message/21174
Mute This Topic: https://lists.fd.io/mt/90157992/21656
Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: Private: Re: [vpp-dev] increase nginx performace using vpp host stack#nginx #vpp-hoststack

[Florin Coras]

TCP accepts connections in time-wait (see here [1]) but won’t reuse ports of 
connection in time-wait for connects. If you expect lots of active opens from 
nginx to only one destination ip and you have multiple source ips, you could 
try to use "tcp src-address” cli. 

Regards,
Florin

[1] https://git.fd.io/vpp/tree/src/vnet/tcp/tcp_input.c#n2667

> On Mar 31, 2022, at 7:00 PM, weizhen9...@163.com wrote:
> 
> Hi, 
> 
> When we test the performance of nginx using vpp host stack, we execute the 
> following command.
> #show run 
> 
> 
> 
> 
> 
> #show session
> 
> 
> 
> Besides, we add some new features in nginx and this feature will cause a lot 
> of time_wait states. In order to reuse TIME_WAIT state tcp_tw_reuse and 
> tcp_tw_recycle. Besides, to make the port unrestricted, we modify other 
> parameters.
> So how can we modify the vpp host stack parameters?
> Thanks.
> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21173): https://lists.fd.io/g/vpp-dev/message/21173
Mute This Topic: https://lists.fd.io/mt/90157992/21656
Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] #vpp-dev AF_XDP interface create and delete problem

[Smith Beirvin]

Hi VPP fellows:
I got problems when I tried to create and delete AF_XDP interface. Pls give me 
some suggestions to figure out the problem. The issue description is as below:

Step 1: create af_xdp interface based on host if wlan0: create interface af_xdp 
host-if wlan0 num-rx-queues 1
Step 2: delete the af_xdp interface: delete interface af_xdp wlan0/0
Step 3: create af_xdp interface based on host if wlan0 again: create interface 
af_xdp host-if wlan0 num-rx-queues 1
Here get excetion info as below
DBGvpp# create interface af_xdp host-if wlan0 num-rx-queues 1
libbpf: can't get next link: Invalid argument
create interface af_xdp: xsk_socket__create() failed (is linux netdev wlan0 
up?): Device or resource busy

if I try to create af_xdp interface by using command: create interface af_xdp 
host-if wlan0 num-rx-queues all
Step 1: create af_xdp interface based on host if wlan0: create interface af_xdp 
host-if wlan0 num-rx-queues all
Step 2: delete the af_xdp interface: delete interface af_xdp wlan0/0
Step 3: create af_xdp interface based on host if wlan0 again: create interface 
af_xdp host-if wlan0 num-rx-queues all
Here I find that af_xdp_create_queue() is faild, as result, there is no queue 
bond to the interface because of bind() function failed.
af_xdp_create_queue()
->xsk_socket__create()
->xsk_socket__create_shared
->err = bind(xsk->fd, (struct sockaddr *), sizeof(sxdp));
if (err) {
err = -errno;
goto out_mmap_tx;
}

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21171): https://lists.fd.io/g/vpp-dev/message/21171
Mute This Topic: https://lists.fd.io/mt/90169229/21656
Mute #vpp-dev:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-dev
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[Mohsin Kazmi via lists.fd.io]

virtio device is using Linux Kernel driver “virtio-pci”. You need to rebind it 
to ‘vfio-pci’ be able to use it in user-space.

There is a rebinding script in vpp:
https://git.fd.io/vpp/tree/extras/scripts/pci-nic-bind


-br

mohsin
From:  on behalf of "long...@gmail.com" 
Date: Thursday, March 31, 2022 at 5:50 PM
To: "vpp-dev@lists.fd.io" 
Subject: Re: [vpp-dev] Fastest way to connect application in user space to VPP 
#vpp

Thank Mohsin,

I create interface without DPDK like below:
create int virtio :00:08.0 gso-enabl
but I got error:
create interface virtio: device not bound to 'vfio-pci' or 'uio_pci_generic' 
kernel module
I tried modprobe virtio_pci but still got it.
Do you know how to fix that?

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21170): https://lists.fd.io/g/vpp-dev/message/21170
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[longtrb]

[Edited Message Follows]

Thank Mohsin,

I create interface without DPDK like below:
> 
> create int virtio :00:08.0 gso-enabl
> 

but I got *error* :
*create interface virtio: device not bound to 'vfio-pci' or 'uio_pci_generic' 
kernel module
* I tried modprobe virtio_pci but still got it.

List pci in my server:
root@ubuntu2004:/home/ubuntu# vppctl show pci
Address      Sock VID:PID     Link Speed    Driver          Product Name        
            Vital Product Data
:00:03.0   0  8086:100e   unknown       e1000
:00:08.0   0  1af4:1000   unknown       virtio-pci
:00:09.0   0  1af4:1000   unknown       virtio-pci

Do you know how to fix that?

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21169): https://lists.fd.io/g/vpp-dev/message/21169
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[longtrb]

Thank Mohsin,

I create interface without DPDK like below:
> 
> create int virtio :00:08.0 gso-enabl
> 

but I got *error* :
*create interface virtio: device not bound to 'vfio-pci' or 'uio_pci_generic' 
kernel module
* I tried modprobe virtio_pci but still got it.
Do you know how to fix that?

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21169): https://lists.fd.io/g/vpp-dev/message/21169
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: Private: Re: [vpp-dev] increase nginx performace using vpp host stack#nginx #vpp-hoststack

[Florin Coras]

I didn’t mean you should switch to envoy, just that throughput is pretty low 
probably because of some configuration. What that configuration is is not 
obvious unfortunately. 

Regarding the kernel parameters, we have time wait reuse enabled (equivalent to 
tcp_tw_reuse) but that should not matter unless nginx establishes a new 
connection for each request. Does it, as part of the features you’ve added?

Could you do: 
- clear run; show run - in vpp when under load
- show sessions - to see the number of session in vpp 

Also, could you reduce the number of workers to maybe 1-2 in nginx and 1 in 
vpp, to get some base performance readings?

Regards,
Florin 

[1] https://wiki.fd.io/images/0/08/Using_vpp_as_envoys_network_stack.pdf 


> On Mar 31, 2022, at 8:56 AM, weizhen9...@163.com wrote:
> 
> Hi,
> 
> I'm doing proxying with nginx. And we develop some new functions in nginx. 
> The performance of nginx in kernel host stack is higher than nginx using vpp 
> host stack. When testing the nginx in kernel host stack, we modify the kernel 
> parameters in kernel host stack. When using vpp host stack, how can we modify 
> the similar parameters?
> 
> Besides, we don't use the envoy. We develop some functions in nginx.
>> 
>> Hi, 
>> 
>> Spoke too soon. Noticed you’re doing proxying with nginx. 
>> 
>> What does clear run; show run report in vpp when under load?
>> 
>> Side note, with envoy I’m seeing much better numbers. See for instance slide 
>> 12 here [1]. So I suspect this is a configuration issue but can’t tell 
>> upfront what it is.
>> 
>> Regards,
>> Florin

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21168): https://lists.fd.io/g/vpp-dev/message/21168
Mute This Topic: https://lists.fd.io/mt/90157992/21656
Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[Mohsin Kazmi via lists.fd.io]

Forgot to mention, your config have enabled gso on tap interface while is not 
enabled on dpdk interface.
Either enable it on dpdk interface if the underneath vhost device supports it. 
or, disable it from tap interface too.



In case of vpp native virtio:

create int virtio :00:08.0 gso-enabled


From:  on behalf of "Mohsin Kazmi via lists.fd.io" 

Reply-To: "Mohsin Kazmi (sykazmi)" 
Date: Thursday, March 31, 2022 at 4:54 PM
To: "long...@gmail.com" , "vpp-dev@lists.fd.io" 

Subject: Re: [vpp-dev] Fastest way to connect application in user space to VPP 
#vpp


Please add following to dpdk stanza in startup.conf



“DPDK { dev :00:08.0 { tso on } enable-tcp-udp-checksum }”



Or else try VPP native virtio without using dpdk:

create int virtio :00:08.0

From:  on behalf of "long...@gmail.com" 
Date: Thursday, March 31, 2022 at 4:40 PM
To: "vpp-dev@lists.fd.io" 
Subject: Re: [vpp-dev] Fastest way to connect application in user space to VPP 
#vpp

Hi Mohsin,
Please see below:

Can you please share the numbers you are getting?

 Without VPP, I get 5Gbps. With VPP I get ~2Gbps.

And also please share the output of:

sh hardware

sh tap

sh threads

sh cpu
Please see in the attachments.
Thank you.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21167): https://lists.fd.io/g/vpp-dev/message/21167
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[Mohsin Kazmi via lists.fd.io]

Please add following to dpdk stanza in startup.conf



“DPDK { dev :00:08.0 { tso on } enable-tcp-udp-checksum }”



Or else try VPP native virtio without using dpdk:

create int virtio :00:08.0

From:  on behalf of "long...@gmail.com" 
Date: Thursday, March 31, 2022 at 4:40 PM
To: "vpp-dev@lists.fd.io" 
Subject: Re: [vpp-dev] Fastest way to connect application in user space to VPP 
#vpp

Hi Mohsin,
Please see below:

Can you please share the numbers you are getting?

 Without VPP, I get 5Gbps. With VPP I get ~2Gbps.

And also please share the output of:

sh hardware

sh tap

sh threads

sh cpu
Please see in the attachments.
Thank you.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21166): https://lists.fd.io/g/vpp-dev/message/21166
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[longtrb]

Hi Mohsin,
Please see below:

> 
> 
> 
> Can you please share the numbers you are getting?
> 
> 

*Without VPP, I get 5Gbps. With VPP I get ~2Gbps.*

> 
> 
> 
> And also please share the output of:
> 
> 
> 
> sh hardware
> 
> 
> 
> sh tap
> 
> 
> 
> sh threads
> 
> 
> 
> sh cpu
> 
> 

Please see in the attachments.
Thank you.
root@ubuntu2004:/home/ubuntu# vppctl sh hardware
  NameIdx   Link  Hardware
enp0s8 1 up   enp0s8
  Link speed: 4294 Gbps
  RX Queues:
queue thread mode  
0 vpp_wk_0 (1)   polling   
  Ethernet address 08:00:27:76:f4:06
  Red Hat Virtio
carrier up full duplex mtu 9206  promisc
flags: admin-up promisc pmd maybe-multiseg
Devargs: 
rx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1)
tx: queues 1 (max 1), desc 256 (min 0 max 65535 align 1)
pci: device 1af4:1000 subsystem 1af4:0001 address :00:08.00 numa 0
max rx packet len: 9728
promiscuous: unicast on all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail:  vlan-strip udp-cksum tcp-cksum tcp-lro vlan-filter 
   jumbo-frame scatter 
rx offload active: jumbo-frame scatter 
tx offload avail:  vlan-insert udp-cksum tcp-cksum tcp-tso multi-segs 
tx offload active: multi-segs 
rss avail: none
rss active:none
tx burst function: virtio_xmit_pkts
rx burst function: virtio_recv_mergeable_pkts

tx frames ok  47
tx bytes ok 5720
rx frames ok   9
rx bytes ok  770
extended stats:
  rx_good_packets  9
  tx_good_packets 47
  rx_good_bytes  770
  tx_good_bytes 5720
  rx_q0_packets9
  rx_q0_bytes770
  tx_q0_packets   47
  tx_q0_bytes   5720
  rx_q0_good_packets   9
  rx_q0_good_bytes   770
  rx_q0_undersize_packets  2
  rx_q0_size_65_127_packets7
  tx_q0_good_packets  47
  tx_q0_good_bytes  5720
  tx_q0_multicast_packets 38
  tx_q0_broadcast_packets  1
  tx_q0_undersize_packets  3
  tx_q0_size_65_127_packets   29
  tx_q0_size_128_255_packets  15
local0 0down  local0
  Link speed: unknown
  local
tap0   2 up   tap0
  Link speed: unknown
  RX Queues:
queue thread mode  
0 vpp_wk_1 (2)   polling   
  Ethernet address 02:fe:f6:35:06:33
  VIRTIO interface
 instance 0
root@ubuntu2004:/home/ubuntu# vppctl sh cpu
Model name:   AMD Ryzen 7 4800H with Radeon Graphics 
Microarch model (family): unknown (family 0x0f model 0x60)
Flags:sse3 pclmulqdq ssse3 sse41 sse42 avx rdrand avx2 
rdseed aes invariant_tsc 
Base frequency:   2.89 GHz
oot@ubuntu2004:/home/ubuntu# vppctl sh tap
Interface: tap0 (ifindex 2)
  name "tap0"
  host-ns "(nil)"
  host-mac-addr: 02:fe:fa:c8:b3:ca
  host-carrier-up: 1
  vhost-fds 38 39 40 41 42
  tap-fds 37
  gso-enabled 1
  csum-enabled 0
  packet-coalesce 0
  packet-buffering 0
  Mac Address: 02:fe:f6:35:06:33
  Device instance: 0
  flags 0x1
admin-up (0)
  features 0x110008000
VIRTIO_NET_F_MRG_RXBUF (15)
VIRTIO_RING_F_INDIRECT_DESC (28)
VIRTIO_F_VERSION_1 (32)
  remote-features 0x33d008000
VIRTIO_NET_F_MRG_RXBUF (15)
VIRTIO_F_NOTIFY_ON_EMPTY (24)
VHOST_F_LOG_ALL (26)
VIRTIO_F_ANY_LAYOUT (27)
VIRTIO_RING_F_INDIRECT_DESC (28)
VIRTIO_RING_F_EVENT_IDX (29)
VIRTIO_F_VERSION_1 (32)
VIRTIO_F_IOMMU_PLATFORM (33)
  Number of RX Virtqueue  1
  Number of TX Virtqueue  5
  Virtqueue (RX) 0
qsz 256, last_used_idx 60, desc_next 32, desc_in_use 228
avail.flags 0x1 avail.idx 288 used.flags 0x1 used.idx 60
kickfd 44, callfd 43
  Virtqueue (TX) 1
qsz 256, last_used_idx 0, desc_next 0, desc_in_use 0
avail.flags 0x1 avail.idx 0 used.flags 0x0 used.idx 0
kickfd 45, callfd -1
  Virtqueue (TX) 3
qsz 256, last_used_idx 20, desc_next 21, desc_in_use 1
avail.flags 0x1 avail.idx 21 used.flags 0x0 used.idx 21
kickfd 46, callfd -1
  Virtqueue (TX) 5
qsz 256, last_used_idx 0, desc_next 0, desc_in_use 0
avail.flags 0x1 avail.idx 0 used.flags 0x0 used.idx 0
kickfd 47, callfd 

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[longtrb]

Hi Jerome,
> 
> you enabled gso but didn't turn gro on. IOW, in this mode, VPP accepts GSO
> packets sent by linux but won't coalesce packets to reduce the load on
> linux side.
> You may be interested in this article 
> https://medium.com/fd-io-vpp/getting-to-40g-encrypted-container-networking-with-calico-vpp-on-commodity-hardware-d7144e52659a
> even if it's not super recent. Anyway, if you application can support VCL,
> it will certainly be much faster.

I have read article you shared, It is very useful but I don't know to configure 
vpp to support virtio backend or enable GRO and how to enable multiple queue 
for tap interface.
Do you know how to do that? Please share me, many thanks tp you

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21164): https://lists.fd.io/g/vpp-dev/message/21164
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] increase nginx performace using vpp host stack#nginx #vpp-hoststack

[weizhen9612]

Hi,
The version of vpp is 22.06. The configs are in the attachment.
The nginx using vpp host stack is low.

The most important test indicators is RPS.
How can we increase the performance of nginx using vpp host stack?


nginx.conf
Description: Binary data


startup.conf
Description: Binary data


vcl.conf
Description: Binary data

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21163): https://lists.fd.io/g/vpp-dev/message/21163
Mute This Topic: https://lists.fd.io/mt/90154397/21656
Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] increase nginx performace using vpp host stack#nginx #vpp-hoststack

[Florin Coras]

Hi, 

Could you provide a bit more info about the numbers you are seeing and the type 
of test you are doing? Also some details about your configs and vpp version? 

As for tcp_tw_recycle, I believe that was removed in Linux 4.12 because it was 
causing issues for nat-ed connections. Did you mean tcp_tw_reuse? If yes, are 
you testing CPS with only one remote client IP? 

Regards, 
Florin

> On Mar 31, 2022, at 6:35 AM, weizhen9...@163.com wrote:
> 
> Now, we develop some functions in nginx and test nginx performance in kernel 
> host stack and vpp host stack. We found the performance of nginx in vpp is 
> lower than nginx in kernel.
> So I want to ask how to debug to increase the performance of nginx in vpp. 
> Besides, we modify some parameter in kernel. For example, in file 
> /etc/sysctl.conf, we add the following parameters:
> net.ipv4.tcp_max_tw_buckets = 3000
> net.ipv4.tcp_tw_reuse = 1
> net.ipv4.tcp_tw_recycle = 1
> In file /etc/security/limited.conf, we add the following parameters:
> * soft nofile 102400
> * hard nofile 102400
> So how can we add those parameters in vpp host stack? And how can we increase 
> the performance of nginx in vpp?
> Thanks.
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21162): https://lists.fd.io/g/vpp-dev/message/21162
Mute This Topic: https://lists.fd.io/mt/90154397/21656
Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] increase nginx performace using vpp host stack#nginx #vpp-hoststack

[weizhen9612]

Now, we develop some functions in nginx and test nginx performance in kernel 
host stack and vpp host stack. We found the performance of nginx in vpp is 
lower than nginx in kernel.
So I want to ask how to debug to increase the performance of nginx in vpp. 
Besides, we modify some parameter in kernel. For example, in file 
/etc/sysctl.conf, we add the following parameters:
net.ipv4.tcp_max_tw_buckets = 3000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
In file /etc/security/limited.conf, we add the following parameters:
* soft nofile 102400
* hard nofile 102400
So how can we add those parameters in vpp host stack? And how can we increase 
the performance of nginx in vpp?
Thanks.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21161): https://lists.fd.io/g/vpp-dev/message/21161
Mute This Topic: https://lists.fd.io/mt/90154397/21656
Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[Mohsin Kazmi via lists.fd.io]

Hello,

Can you please share the numbers you are getting?

And also please share the output of:
sh hardware
sh tap
sh threads
sh cpu

-br
Mohsin

From:  on behalf of "long...@gmail.com" 
Date: Thursday, March 31, 2022 at 10:20 AM
To: "vpp-dev@lists.fd.io" 
Subject: Re: [vpp-dev] Fastest way to connect application in user space to VPP 
#vpp

Thank you Ole,
NIC - VPP - BSD socket app (using LD_PRELOAD 
https://s3-docs.fd.io/vpp/22.02/developer/extras/vcl_ldpreload.html)
==> Can any my application (Java, Python,...) run on vlc ld preload like NIC - 
VPP - vcl ld_preload socket - application?

But the answer here is very much "it depends".

==> What is it?

Thank you very much for helping me!

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21160): https://lists.fd.io/g/vpp-dev/message/21160
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[Jerome Tollet via lists.fd.io]

Hello,
In your config you enabled gso but didn't turn gro on. IOW, in this mode, VPP 
accepts GSO packets sent by linux but won't coalesce packets to reduce the load 
on linux side.
You may be interested in this article 
https://medium.com/fd-io-vpp/getting-to-40g-encrypted-container-networking-with-calico-vpp-on-commodity-hardware-d7144e52659a
 even if it's not super recent. Anyway, if you application can support VCL, it 
will certainly be much faster.
Jerome

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21159): https://lists.fd.io/g/vpp-dev/message/21159
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[longtrb]

Thank you Ole,

> 
> NIC - VPP - BSD socket app (using LD_PRELOAD 
> https://s3-docs.fd.io/vpp/22.02/developer/extras/vcl_ldpreload.html
> )

==> Can any my application (Java, Python,...) run on vlc ld preload like NIC - 
VPP - vcl ld_preload socket - application?

> 
> But the answer here is very much "it depends".
> 

==> What is it?

Thank you very much for helping me!

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21158): https://lists.fd.io/g/vpp-dev/message/21158
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] #vpp-dev: VRF route leak not working between two VRF's

[Pim van Pelt]

Hoi Suresh,

As a datapoint to this -  route synchronization from VPP -> Linux is not
implemented. The other way around (setting routes in Linux, using Netlink,
will be propagated into the VPP dataplane).


On Thu, Mar 31, 2022 at 5:29 AM suresh vuppala  wrote:

> I have two loopback interfaces in two different VRF's. I have now
> configured route leak between two VRF's as shown below but route leak is
> not happening and also linux is not learning these leaked routes
>
>
>
>
> ip table add 1
>
> set interface ip address loop9997 192.168.97.9/24
>
> set interface state loop9997 up
>
> set interface ip table loop9997 1
>
> lcp create loop9997 host-if data9997
>
> ip table add 2
>
> set interface ip address loop9998 192.168.98.9/24
>
> set interface state loop9998 up
>
> set interface ip table loop9998 2
>
> lcp create loop9998 host-if data9998
>
>
>
> ip route add 192.168.98.0/24 table 1 via loop9998
>
> ip route add 192.168.97.0/24 table 2 via loop9997
>
> ON linux no routes are learned
> =
>
> $ sudo ip netns exec dataplane ip route show 192.168.98.0/24 vrf vrf1
>
> $
>
> $ sudo ip netns exec dataplane ip route show 192.168.97.0/24 vrf vrf2
>
>
>
> $
>
> 
>
>

-- 
Pim van Pelt 
PBVP1-RIPE - http://www.ipng.nl/

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21157): https://lists.fd.io/g/vpp-dev/message/21157
Mute This Topic: https://lists.fd.io/mt/90148412/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Fastest way to connect application in user space to VPP #vpp

[Ole Troan]

Dear Mr Longtrb,

> I am a newbie in VPP. Many thanks to you all for making great Vpp.
> I have many applications in user space (written by java, C, python,...) and I 
> want to connect those apps to VPP so what is the fastest way that VPP can 
> support without  changing the application?
> 
> I tried with tap create by VPP and cross connect it to a physical NIC, NIC is 
> bound by DPDK ( data path: NIC VPP - tap - application):
> 
> vppctl create tap id 0 host-ip4-addr 192.168.56.110/24 gso
> vppctl set interface tap0 ip address 192.168.56.110/24
> vppctl set interface l2 xconnect tap0 enp0s8
> vppctl set interface l2 xconnect enp0s8 tap0
> vppctl set interface state enp0s8 up
> vppctl set interface state tap0 up
> 
> But the bandwidth are lower and latency is higher than when I use real NIC ( 
> without VPP). 
> VPP is version 21.10, I use 4 workers.  Server is ubuntu 20.04, tool test is 
> Iperf3.

If you compare:

NIC - Linux kernel - BSD socket app

and 

NIC - VPP - Linux kernel - BSD socket app

Then I would indeed expect the latter to be slower.

NIC - VPP - BSD socket app (using LD_PRELOAD 
https://s3-docs.fd.io/vpp/22.02/developer/extras/vcl_ldpreload.html)

is one option.

But the answer here is very much "it depends".

Best regards,
Ole


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21156): https://lists.fd.io/g/vpp-dev/message/21156
Mute This Topic: https://lists.fd.io/mt/90135014/21656
Mute #vpp:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] issue with ethernet-input (or not)

[Stanislav Zaikin]

I put an assert just to be sure that virtio-input got a frame with the
ETH_INPUT_FRAME_F_SINGLE_SW_IF_IDX
flag.

Attaching to process 324596
[New LWP 324612]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x7fc5c21cf1dc in virtio_device_input_inline (vm=0x7fc580e0b740,
node=0x7fc581491f40, frame=0x0, vif=0x7fc58175f900,
qid=0, type=VIRTIO_IF_TYPE_TAP) at
/home/zstas/vpp/src/vnet/devices/virtio/node.c:447
447 /home/zstas/vpp/src/vnet/devices/virtio/node.c: No such file or
directory.
Continuing.

Thread 1 "vpp_main" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x7fc5c0f33859 in __GI_abort () at abort.c:79
#2  0x004072f3 in os_panic () at
/home/zstas/vpp/src/vpp/vnet/main.c:413
#3  0x7fc5c1283bc9 in debugger () at
/home/zstas/vpp/src/vppinfra/error.c:84
#4  0x7fc5c128392d in _clib_error (how_to_die=2, function_name=0x0,
line_number=0,
fmt=0x7fc5c22051a7 "%s:%d (%s) assertion `%s' fails") at
/home/zstas/vpp/src/vppinfra/error.c:143
#5  0x7fc5c21cf4ae in virtio_device_input_gso_inline
(vm=0x7fc580e0b740, node=0x7fc581491f40, frame=0x0,
vif=0x7fc58175ef00, vring=0x7fc5866aedc0, type=VIRTIO_IF_TYPE_TAP,
gso_enabled=0, checksum_offload_enabled=0, packed=0)
at /home/zstas/vpp/src/vnet/devices/virtio/node.c:266
#6  0x7fc5c21cf1e4 in virtio_device_input_inline (vm=0x7fc580e0b740,
node=0x7fc581491f40, frame=0x0, vif=0x7fc58175ef00,
qid=0, type=VIRTIO_IF_TYPE_TAP) at
/home/zstas/vpp/src/vnet/devices/virtio/node.c:447
#7  0x7fc5c21cedb1 in virtio_input_node_fn_icl (vm=0x7fc580e0b740,
node=0x7fc581491f40, frame=0x0)
at /home/zstas/vpp/src/vnet/devices/virtio/node.c:472
#8  0x7fc5c13f3f48 in dispatch_node (vm=0x7fc580e0b740,
node=0x7fc581491f40, type=VLIB_NODE_TYPE_INPUT,
dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0,
last_time_stamp=388854079070548) at /home/zstas/vpp/src/vlib/main.c:961
#9  0x7fc5c13ef276 in vlib_main_or_worker_loop (vm=0x7fc580e0b740,
is_main=1) at /home/zstas/vpp/src/vlib/main.c:1556
#10 0x7fc5c13f14ba in vlib_main_loop (vm=0x7fc580e0b740) at
/home/zstas/vpp/src/vlib/main.c:1715
#11 0x7fc5c13f129e in vlib_main (vm=0x7fc580e0b740,
input=0x7fc57c8b9fa8) at /home/zstas/vpp/src/vlib/main.c:2009
#12 0x7fc5c145cd5e in thread0 (arg=140486247495488) at
/home/zstas/vpp/src/vlib/unix/main.c:656
#13 0x7fc5c129cb48 in clib_calljmp () at
/home/zstas/vpp/src/vppinfra/longjmp.S:123
#14 0x7ffcb8196770 in ?? ()
#15 0x7fc5c145c94c in vlib_unix_main (argc=82, argv=0x1873450) at
/home/zstas/vpp/src/vlib/unix/main.c:736
#16 0x004066bf in main (argc=82, argv=0x1873450) at
/home/zstas/vpp/src/vpp/vnet/main.c:342
(gdb) select 5
(gdb) info locals
_f = 0x7fc5867ad040
_n = 25
n_left_to_next = 0
next0 = 4
vnm = 0x7fc5c24a06e0 
thread_index = 0
n_trace = 0
next_index = 4
hdr_sz = 12
to_next = 0x0
n_rx_packets = 0
n_rx_bytes = 0
mask = 255
last = 27
n_left = 2
bt = {{cacheline0 = 0x7fc57c8b96c0 "", current_data = 0, current_length =
0, flags = 0, flow_id = 0, ref_count = 0 '\000',
buffer_pool_index = 0 '\000', error = 0, next_buffer = 0,
{current_config_index = 0, punt_reason = 0}, opaque = {0, 0, 0,
  0, 0, 0, 0, 0, 0, 0}, template_end = 0x7fc57c8b9700 "", second_half =
0x7fc57c8b9700 "", trace_handle = 0,
total_length_not_including_first_buffer = 0, opaque2 = {0 }, headroom = 0x7fc57c8b9740 "",
pre_data = '\000' , data = 0x7fc57c8b97c0
"`\230\213|\305\177"}, as_u8x16 = {'\000' ,
'\000' , '\000' , '\000' }, as_u8x32 = {'\000' ,
'\000' }, as_u8x64 = {'\000' }}
(gdb) p *_f
$1 = {frame_flags = 6, flags = 1, scalar_offset = 16, vector_offset = 32,
aux_offset = 0, n_vectors = 25,
  arguments = 0x7fc5867ad04c "\376\376\376\376\003"}
(gdb)

On Wed, 30 Mar 2022 at 19:35, Stanislav Zaikin via lists.fd.io  wrote:

> Looks like I found the issue:
>
> it happens when ethernet-node gets a frame with flag
> ETH_INPUT_FRAME_F_SINGLE_SW_IF_IDX but they're actually packets from
> different interfaces (from virtio-node but some of them are l2 and some of
> them are l3). virtio-input doesn't set this flag, but I thought maybe it
> just wasn't cleared.
>
> After I've added a piece of code which clears the flag in frames - the
> issue was gone:
> diff --git a/src/vlib/main.c b/src/vlib/main.c
> index fd0c1fc5f..e88a0724a 100644
> --- a/src/vlib/main.c
> +++ b/src/vlib/main.c
> @@ -208,6 +208,7 @@ vlib_frame_free (vlib_main_t * vm, vlib_node_runtime_t
> * r, vlib_frame_t * f)
>  }
>
>f->frame_flags &= ~(VLIB_FRAME_IS_ALLOCATED | VLIB_FRAME_NO_APPEND);
> +  f->flags = 0;
>
>vec_add1 (fs->free_frames, f);
>ASSERT (fs->n_alloc_frames > 0);
> @@ -1163,6 +1164,7 @@ dispatch_pending_node (vlib_main_t * vm, uword
>