Hi Alan,

How are you starting the VM? QEMU or Libvirt?



The dpdk vhost ports are associated with the numa node the virtqueue memory has 
been allocated on initially. So if running the VM you may want to use taskset 
–c with QEMU to allocate cores associated with numa node 1 to run the VM. If 
using libvirt try to ensure vcupin corresponds to numa 1 cores also in the xml.



In my testing to ensure that the vhost port used memory from the same socket as 
the core it’s PMD is running on I had to compile DPDK with 
CONFIG_RTE_LIBRTE_VHOST_NUMA=y. This will avoid the warning altogether 
regardless if memory is allocated to both sockets.



If you’re interested in how to test this there is a blog for using vhost numa 
aware that could be of use:



https://software.intel.com/en-us/articles/vhost-user-numa-awareness-in-open-vswitch-with-dpdk



Hope this helps.

Ian

From: Alan Kayahan [mailto:hsy...@gmail.com]
Sent: Friday, April 13, 2018 8:05 AM
To: Stokes, Ian <ian.sto...@intel.com>
Cc: ovs-discuss@openvswitch.org
Subject: Re: [ovs-discuss] There's no available (non-isolated) pmd thread on 
numa node 0, Expect reduced performance.

Hi Ian,

> As you are setting all lcore and pmd core to node 1 why are you giving 1024 
> memory to node 0?
> When processing packets for this port it means cpu is accessing data across 
> the numa nodes which causes a performance penalty
I am benchmarking performance in different settings and trying to understand 
the roles of OVS and DPDK in mediating core affinity of pmds and hugepage 
utilization. Your answer helps a lot!

> try using ‘other_config:dpdk-socket-mem=0,4096’ and see if you still see the 
> issue.
But this warning should appear regardless the socket-mem allocation right? If 
my understanding is correct, when OVS pmd's are pinned to 10-19 and the VM 
tespmd app is pinned to core 2; the OVSpmd thread running on node 1 is having 
to access a huge-page on node 0 which VMtestpmd happens to access as well.

Thanks,
Alan


2018-04-12 18:28 GMT+02:00 Stokes, Ian 
<ian.sto...@intel.com<mailto:ian.sto...@intel.com>>:
Hi,

I was able to reproduce the issue on my system.

As you are setting all lcore and pmd core to node 1 why are you giving 1024 
memory to node 0?

I saw the same issue on my system but the warning did not appear once memory 
was allocated to node 1 only.

I would think the VM being launched is using memory for the vhost port from 
node 0, however the queue for the vhost port is assigned to core 14 which is on 
node1. When processing packets for this port it means cpu is accessing data 
across the numa nodes which causes a performance penalty, hence the warning.

To avoid you should ensure all memory and cores operate on the same node where 
possible, try using ‘other_config:dpdk-socket-mem=0,4096’ and see if you still 
see the issue.

Thanks
Ian

From: 
ovs-discuss-boun...@openvswitch.org<mailto:ovs-discuss-boun...@openvswitch.org> 
[mailto:ovs-discuss-boun...@openvswitch.org<mailto:ovs-discuss-boun...@openvswitch.org>]
 On Behalf Of Alan Kayahan
Sent: Thursday, April 12, 2018 2:27 AM
To: ovs-discuss@openvswitch.org<mailto:ovs-discuss@openvswitch.org>
Subject: [ovs-discuss] There's no available (non-isolated) pmd thread on numa 
node 0, Expect reduced performance.

Hello,

On the following setup, where all cores but 0 are isolated,

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9
node 1 cpus: 10 11 12 13 14 15 16 17 18 19

I am trying to start OVS entirely on numa node 1 as following

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true 
other_config:dpdk-lcore-mask=0x00400 other_config:pmd-cpu-mask=0xffc00 
other_config:dpdk-socket-mem=1024,4096

However when I create a vhost port SRC to attach a VNF(via virtio) on node 0, I 
get the following

dpif_netdev|WARN|There's no available (non-isolated) pmd thread on numa node 0. 
Queue 0 on port 'SRC' will be assigned to the pmd on core 14 (numa node 1). 
Expect reduced performance.

Any ideas?

Thanks

_______________________________________________
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Reply via email to