When I try to set other_config:n-dpdk-rxqs to 64 with two intel niantic dpdk
ports on a single dpdk bridge, the bridge instance will 'crash' - I
can't access
it's flows through ovs-ofctl commands. I am running an OVS from the 2.5
branch,
specifically commit ID b3e263929a7a00c96a1329f93f1b8fce58b726e4, DPDK 16.04.
If I set other_config:n-dpdk-rxqs to 32, everything works:
Sep 16 10:18:35 bl460gen9-05 ovs-vsctl: ovs|00001|vsctl|INFO|Called as
ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=32
Sep 16 10:18:35 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_dev_tx_queue_setup(): sw_ring=0x7fceda7954c0
hw_ring=0x7fceda79d500 dma_addr=0xfda79d500
Sep 16 10:18:35 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_set_tx_function(): Using simple tx code path
Sep 16 10:18:35 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_set_tx_function(): Vector tx enabled.
...(lots of the same three lines)
If I set n-dpdk-rxqs to 64 with a single dpdk port, everything works.
But if I try for 64 with _two_ ports, which should be a value the
niantic PMD supports, I get:
Sep 16 10:21:47 bl460gen9-05 ovs-vsctl: ovs|00001|vsctl|INFO|Called as
ovs-vsctl set Open_vSwitch . other_config:n-dpdk-rxqs=64
Sep 16 10:21:47 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_dev_tx_queue_setup(): sw_ring=0x7fceda7954c0
hw_ring=0x7fceda79d500 dma_addr=0xfda79d500
Sep 16 10:21:47 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_set_tx_function(): Using simple tx code path
Sep 16 10:21:47 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_set_tx_function(): Vector tx enabled.
...
Sep 16 10:21:47 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_dev_rx_queue_setup(): sw_ring=0x7fced8f32f40
sw_sc_ring=0x7fced8f2ee00 hw_ring=0x7fced8f37080 dma_addr=0xfd8f37080
Sep 16 10:21:47 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_dev_rx_queue_setup(): sw_ring=0x7fced8f1a8c0
sw_sc_ring=0x7fced8f16780 hw_ring=0x7fced8f1ea00 dma_addr=0xfd8f1ea00
Sep 16 10:21:47 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_set_rx_function(): Vector rx enabled, please make sure RX burst
size no less than 4 (port=1).
Sep 16 10:21:48 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_alloc_rx_queue_mbufs(): RX mbuf alloc failed queue_id=60
Sep 16 10:21:48 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_dev_rx_queue_start(): Could not alloc mbuf for queue:60
Sep 16 10:21:48 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_dev_start(): Unable to start rxtx queues
Sep 16 10:21:48 bl460gen9-05 ovs-vswitchd[24697]: PMD:
ixgbe_dev_start(): failure in ixgbe_dev_start(): -1
Sep 16 10:21:48 bl460gen9-05 ovs-vswitchd[24697]:
ovs|00070|dpdk|ERR|Interface dpdk1 start error: Input/output error
Sep 16 10:21:48 bl460gen9-05 ovs-vswitchd[24697]:
ovs|00071|dpif_netdev|ERR|Failed to set dpdk interface dpdk1 rx_queue to: 64
Sep 16 10:21:48 bl460gen9-05 kernel: [156923.315906]
ovs-vswitchd[24697]: segfault at 410100 ip 0000000000460192 sp
00007ffeba4587b0 error 7 in ovs-vswitchd-dpdk[400000+1bc000]
Sep 16 10:21:49 bl460gen9-05 ovs-vswitchd:
ovs|00002|daemon_unix(monitor)|ERR|1 crashes: pid 24697 died, killed
(Segmentation fault), core dumped, restarting
The coredump shows a bunch of threads doing poll() epoll_wait(), select() or
nanosleep() except one:
(gdb) bt
#0 0x0000000000460192 in dp_netdev_process_rxq_port.isra ()
#1 0x00007fe6ffda7620 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00000000004d4b70 in shash_add_nocopy__ ()
#3 0x0000000002218de0 in ?? ()
#4 0x00000000004e1ac5 in xmalloc ()
#5 0x0000000000000000 in ?? ()
(gdb)
This is with 12G hugepages on node0. Despite the error messages, ps
shows that
vswitchd is still running:
john@bl460gen9-05:~$ ps -ef | grep -i ovs-vswitchd
root 24696 1 0 10:18 ? 00:00:00 ovs-vswit... --dpdk -c
0x1 -n 2 --socket-mem 12288,0 -- unix:/var/run/openvswitch/db.sock
-vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir
--log-file=/var/log/openvswitch/ovs-vswitchd.log
--pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
root 25802 24696 79 10:23 ? 00:00:00 ovs-vswitchd --dpdk -c
0x1 -n 2 --socket-mem 12288 0 -- unix:/var/run/openvswitch/db.sock
-vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir
--log-file=/var/log/openvswitch/ovs-vswitchd.log
--pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
john 25804 24200 0 10:23 pts/3 00:00:00 grep -i ovs-vswitchd
It won't respond, however to /etc/init.d/openvswitch-switch stop, which
gives
the message:
root@bl460gen9-05:/home/john# /etc/init.d/openvswitch-switch stop
Exiting ovs-vswitchd (28055).
ovsdb-server is not running.
But is apparently lying because I still see ovs-vswitchd running. I have
to kill
it and my hugepages are therefore not de-allocated.
But I can't show flows, nor can I add a bridge and look at the flows
later. I
can run testpmd with 64 rxq and 64 txq with no issue:
# testpmd -c 0x3 -d
/usr/lib/x86_64-linux-gnu/dpdk/librte_pmd_ixgbe.so.1.1 -- --rxq=64
--txq=64 -i
I can succesfully forward in this way, with two ports. Thinking that it
was simply a memory
allocation issue, I tried allocating less memory using --socket-mem, and was
able to succesfully run testpmd with only 1G of hugepage memory and 64
rxq and
txq. I have not yet tried this with OVS master and DPDK 16.07.
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
