Hi Richard,
I also got to cross check this bug now.
I can not reproduce your issue on my current setup.
Here some logs, and at the end the recommended next steps.
# prep
ovs-vsctl set Open_vSwitch . "other_config:dpdk-init=true"
ovs-vsctl set Open_vSwitch . "other_config:dpdk-alloc-mem=2048"
ovs-vsctl set Open_vSwitch . "other_config:dpdk-extra=--vhost-owner
libvirt-qemu:kvm --vhost-perm 0666"
# restart
update-alternatives --set ovs-vswitchd
/usr/lib/openvswitch-switch-dpdk/ovs-vswitchd-dpdk
systemctl restart openvswitch-switch
# add bridge and dev
ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
options:dpdk-devargs=0000:04:00.1
# check defaults
$ sudo pidstat -p 3670 -u -t 5
Average: UID TGID TID %usr %system %guest %wait %CPU
CPU Command
Average: 0 3670 - 100,00 0,13 0,00 0,00 100,00
- ovs-vswitchd
Average: 0 - 3670 0,07 0,07 0,00 0,00 0,13
- |__ovs-vswitchd
[...]
Average: 0 - 3868 0,07 0,07 0,00 0,00 0,13
- |__revalidator27
Average: 0 - 3916 100,00 0,00 0,00 0,00 100,00
- |__pmd28
1 PMD thread, nothing unexpected yet (I have one numa node, so default
pmd mask is 1).
# Note n-dpdk-rxqs is no more since OVS>2.5
# ovs-vsctl set Open_vSwitch . "other_config:n-dpdk-rxqs=10"
# Instead the config is per device like
# ovs-vsctl set Interface dpdk0 options:n_rxq=8
# iterate (re)-configure queues and mask
# MASK = CPU Mask, NUM = expected threadss, so set queues accordingly
MASK="1011"; HEX=$(printf '0x%x\n' $((2#${MASK}))); NUM=0; val=$((2#${MASK}));
while [ ${val} -ne 0 ]; do if [ $((${val} & 1)) -eq 1 ]; then ((NUM++)); fi;
val=$((val>>1)); done; echo "Mask: $MASK => Hex: $HEX => Num: $NUM";
ovs-vsctl set Interface dpdk0 options:n_rxq=${NUM}
ovs-vsctl set Open_vSwitch . "other_config:pmd-cpu-mask=${HEX}"
I see it correctly e.g.:
- adding a queue when I go from 11 -> 1011
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 3 created.
- removeing a queue when going from 1011 -> 1001
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 1 destroyed.
- I can fill up all 12 Cores I have atm
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 9 created.
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 11 created.
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 8 created.
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 7 created.
dpif_netdev|INFO|PMD thread on numa_id: 0, core id: 10 created.
dpif_netdev|INFO|There are 12 pmd threads on numa node 0
You can have less queues than your mask opens for PMDs.
But this will likely be in efficient. So e.g. the following works but isn't
recommended:
$ ovs-vsctl set Interface dpdk0 options:n_rxq=6
$ ovs-vsctl set Open_vSwitch . "other_config:pmd-cpu-mask=0xdef"
If I set your example of 1 -> 111111110000 (more than 6) things still work fine:
Also changing the mask is rather robust to mistakes, if I add more CPUs than I
actually have those are just ignored.
So on a 12 CPU system 111111110000 (12) -> 11111111000000 (14) just ignores the
cpus that are too high.
I wondered first if this might be as in bug 1719392 related to your Numa node
setup?
I have only one node and can use all cores, you have issues to use more than a
few and hang.
The alternative reason I realized is the PMD itself having issues. I currently
use net_ixgbe.
Given your report you have thunderX, so I picked a system with those.
I only have a 1 Node 48 Core system with those, but let me try this PMD there.
I can reproduce my good results at first with smaller masks, I was even able to
add more than 6 (7) in my case.
But I found that when going further as soon as I pass 7 (in my case) I see a
crash:
dpdk|EMER|Cannot assign sufficient number of secondary queues to primary VF3
dpdk|ERR|1:
[/usr/lib/aarch64-linux-gnu/librte_eal.so.17.05(rte_dump_stack+0x38)
[0xffffb9a5d778]]
daemon_unix(monitor)|ERR|1 crashes: pid 4281 died, killed (Aborted), core
dumped, restarting
To the command that was issued this appears as a hang, as the service
crashed.
That is not an Ubuntu issue, it is three issues:
1. a HW limitation of the thunderX PMD (primary vs secondary queues)
2. a thunderx PMD issue to hard fail in this case instead of spawning just fewer
3. a design issue to crash OVS in such cases (this is being worked on upstream
already)
This primary/secondary queue constraint [1] has driven me nuts as well,
but there is nothing Ubuntu can do about the HW specifics.
I'll mark this bug as invalid for now.
If you still think this an Ubuntu issue, please attach the following extra Data:
Please include:
1. $ lscpu -e
2. $ numactl -H
3. $ for i in /sys/bus/pci/devices/*; do printf "%s => %2s\n" $(basename $i)
$(cat $i/numa_node); done
4. Before triggering the hang start
$ sudo tail -f /var/log/openvswitch/ovs-vswitchd.log
Then attach all the log that gets added by the call to further open up the
CPU mask
[1]: http://dpdk.org/doc/guides/nics/thunderx.html#multiple-queue-set-
per-dpdk-port-configuration
** Changed in: dpdk (Ubuntu)
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1719387
Title:
DPDK with OVS cannot allocate more than 6 CPU
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/dpdk/+bug/1719387/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
