Hi Alan,
How are you starting the VM? QEMU or Libvirt? The dpdk vhost ports are associated with the numa node the virtqueue memory has been allocated on initially. So if running the VM you may want to use taskset –c with QEMU to allocate cores associated with numa node 1 to run the VM. If using libvirt try to ensure vcupin corresponds to numa 1 cores also in the xml. In my testing to ensure that the vhost port used memory from the same socket as the core it’s PMD is running on I had to compile DPDK with CONFIG_RTE_LIBRTE_VHOST_NUMA=y. This will avoid the warning altogether regardless if memory is allocated to both sockets. If you’re interested in how to test this there is a blog for using vhost numa aware that could be of use: https://software.intel.com/en-us/articles/vhost-user-numa-awareness-in-open-vswitch-with-dpdk Hope this helps. Ian From: Alan Kayahan [mailto:[email protected]] Sent: Friday, April 13, 2018 8:05 AM To: Stokes, Ian <[email protected]> Cc: [email protected] Subject: Re: [ovs-discuss] There's no available (non-isolated) pmd thread on numa node 0, Expect reduced performance. Hi Ian, > As you are setting all lcore and pmd core to node 1 why are you giving 1024 > memory to node 0? > When processing packets for this port it means cpu is accessing data across > the numa nodes which causes a performance penalty I am benchmarking performance in different settings and trying to understand the roles of OVS and DPDK in mediating core affinity of pmds and hugepage utilization. Your answer helps a lot! > try using ‘other_config:dpdk-socket-mem=0,4096’ and see if you still see the > issue. But this warning should appear regardless the socket-mem allocation right? If my understanding is correct, when OVS pmd's are pinned to 10-19 and the VM tespmd app is pinned to core 2; the OVSpmd thread running on node 1 is having to access a huge-page on node 0 which VMtestpmd happens to access as well. Thanks, Alan 2018-04-12 18:28 GMT+02:00 Stokes, Ian <[email protected]<mailto:[email protected]>>: Hi, I was able to reproduce the issue on my system. As you are setting all lcore and pmd core to node 1 why are you giving 1024 memory to node 0? I saw the same issue on my system but the warning did not appear once memory was allocated to node 1 only. I would think the VM being launched is using memory for the vhost port from node 0, however the queue for the vhost port is assigned to core 14 which is on node1. When processing packets for this port it means cpu is accessing data across the numa nodes which causes a performance penalty, hence the warning. To avoid you should ensure all memory and cores operate on the same node where possible, try using ‘other_config:dpdk-socket-mem=0,4096’ and see if you still see the issue. Thanks Ian From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Alan Kayahan Sent: Thursday, April 12, 2018 2:27 AM To: [email protected]<mailto:[email protected]> Subject: [ovs-discuss] There's no available (non-isolated) pmd thread on numa node 0, Expect reduced performance. Hello, On the following setup, where all cores but 0 are isolated, available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 node 1 cpus: 10 11 12 13 14 15 16 17 18 19 I am trying to start OVS entirely on numa node 1 as following ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true other_config:dpdk-lcore-mask=0x00400 other_config:pmd-cpu-mask=0xffc00 other_config:dpdk-socket-mem=1024,4096 However when I create a vhost port SRC to attach a VNF(via virtio) on node 0, I get the following dpif_netdev|WARN|There's no available (non-isolated) pmd thread on numa node 0. Queue 0 on port 'SRC' will be assigned to the pmd on core 14 (numa node 1). Expect reduced performance. Any ideas? Thanks
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
