Hello, DPDK version: 21.11.1 OVS version: 2.17.2 OS: ubuntu 18.04 Kernel: 4.15.0-188-generic
DPDK and OVS were compiled from source. I am running into a number of issues trying to get OVS to work with DPDK / Mellanox ConnectX6, can someone assist with the following problems below? 1) After restarting the openvswitch-switch service I see the following warning: ``` 2022-07-15T14:41:39.548Z|00023|dpdk|WARN|EAL: DPDK is running on a NUMA system, but is compiled without NUMA support. 2022-07-15T14:41:39.548Z|00024|dpdk|WARN|EAL: This will have adverse consequences for performance and usability. 2022-07-15T14:41:39.548Z|00025|dpdk|WARN|EAL: Please use --legacy-mem option, or recompile with NUMA support. ``` This is a DPDK warning, but it's not clear on how I can fix this. I can't find any documentation on how to compile with the "--legacy-mem" option. Is this a meson or ninja parameter? Or how do I "recompile with NUMA support", this is not clear and the DPDK documentation is not clear either. Or can I just ignore this? 2) I am trying to follow these instructions on how to deploy on Ubuntu ( https://ubuntu.com/server/docs/openvswitch-dpdk), however the Open_vSwitch table settings presented cause OVS to crash when starting up. ``` # Allocate 2G huge pages (not Numa node aware) ovs-vsctl set Open_vSwitch . "other_config:dpdk-alloc-mem=2048" # limit to one whitelisted device ovs-vsctl set Open_vSwitch . "other_config:dpdk-extra=--pci-whitelist=0000:04:00.0" ``` 2.1) Trying to run the first command (ovs-vsctl set Open_vSwitch . "other_config:dpdk-alloc-mem=2048") gives me this error: ``` 2022-07-15T15:14:05.549Z|00026|dpdk|ERR|EAL: Not enough memory available! Requested: 2048MB, available: 118MB ``` But the system has lots of memory: ``` $ free -g total used free shared buff/cache available Mem: 251 89 113 0 47 159 Swap: 7 0 7 ``` Why am I getting this error? I have 113GB of memory free. Why does DPDK show I only have 118MB? 2.2) Trying to run the second command (ovs-vsctl set Open_vSwitch . "other_config:dpdk-extra=--pci-whitelist=0000:41:01.6") I figured out this command is wrong and used the following which did take: ` sudo ovs-vsctl set Open_vSwitch . "other_config:dpdk-whitelist-pci-ids="0000:41:01.6,0000:41:00.6""` instead. However How can I tell if these ports are actually white listed? The log shows the following output when I start the server, are the warnings a problem? ``` 2022-07-15T15:21:16.632Z|00059|dpdk|INFO|EAL: Probe PCI driver: mlx5_pci (15b3:101e) device: 0000:41:01.6 (socket 0) 2022-07-15T15:21:16.642Z|00060|dpdk|WARN|mlx5_net: No available register for sampler. 2022-07-15T15:21:16.317Z|00046|dpdk|INFO|EAL: Probe PCI driver: mlx5_pci (15b3:101e) device: 0000:41:00.6 (socket 0) 2022-07-15T15:21:16.328Z|00047|dpdk|WARN|mlx5_net: No available register for sampler. ``` I tried unbinding the ports from the kernel with dpdk-devbind but this caused errors to happen on these PCI slots, I put them back as control of the kernel. 3) How can I tell if DPDK is actually working? I would think that I would need to unbind the ports from the kernel before I can add them to OVS/DPDK. Here is my config for adding a dpdk bridge and ports ``` ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev ovs-vsctl add-port br0 dpdk-enp65s0f6 -- set Interface dpdk-enp65s0f6 type=dpdk options:dpdk-devargs=0000:41:00.6 ovs-vsctl add-port br0 dpdk-enp65s1f6 -- set Interface dpdk-enp65s1f6 type=dpdk options:dpdk-devargs=0000:41:01.6 ``` Here is the show: ``` e2a3fa00-23c9-4c3d-b9b6-e37df4f00dd7 Bridge br0 datapath_type: netdev Port br0 Interface br0 type: internal Port dpdk-enp65s1f6 Interface dpdk-enp65s1f6 type: dpdk options: {dpdk-devargs="0000:41:01.6"} Port dpdk-enp65s0f6 Interface dpdk-enp65s0f6 type: dpdk options: {dpdk-devargs="0000:41:00.6"} ovs_version: "2.17.2" ``` It appears they are in place, here is log output ``` 2022-07-15T15:28:36.385Z|00190|netdev_dpdk|INFO|Port 8: 52:f4:8d:db:ed:b9 2022-07-15T15:28:36.385Z|00191|dpif_netdev|INFO|Performing pmd to rx queue assignment using cycles algorithm. 2022-07-15T15:28:36.385Z|00192|dpif_netdev|INFO|Core 21 on numa node 0 assigned port 'dpdk-enp65s0f6' rx queue 0 (measured processing cycles 296240). 2022-07-15T15:28:36.385Z|00193|dpif_netdev|INFO|Core 21 on numa node 0 assigned port 'dpdk-enp65s1f6' rx queue 0 (measured processing cycles 0). 2022-07-15T15:28:36.386Z|00194|bridge|INFO|bridge br0: added interface dpdk-enp65s1f6 on port 2 ``` But the interfaces still show up in the kernel `ip -br a` shows as much. Is this expected? Is RDMA being used or something that uses DPDK based abstraction? Should I be unbinding and adding in a different way? Hoping someone can help with my queries, thanks. - Dave
_______________________________________________ discuss mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
