I finished writing the environment setup script following instructions from
https://raw.githubusercontent.com/openvswitch/ovs/master/INSTALL.DPDK and
the results are:
- ovs still not building against dpdk head (version 1.7.1) because: const
struct rte_mbuf' has no member named 'pkt'
- ovs builds against dpdk v1.7.0
- ovs-vswitchd starts and creates dpdk ports without crashing
- able to run iperf through the dpdk port between the ovs with dpdk and
another Ubuntu machine where the NIC is handled by the kernel (ixgbe
module). Results: 550 Mbits/sec on a 10G link
So I was not able to reproduce the crash after following the new set of
instructions. I was probably repeatedly doing something wrong, but can't
pinpoint the problem. I thought it was an issue caused by the number of
cores or memory channels but it's not, I've tried with multiple combinations
and everything is ok. Sorry for the false alarm.
What is the expected release date for OVS 2.4? (I'm interested in the dpdk
vhost support)
Thanks,
Daniel
Here's the output of ovs-vsctl show after starting ovs-vswitchd:
ovs-vsctl show
#12f96467-8814-4814-a482-6fad87e8281b
# Bridge "br0"
# Port "br0"
# Interface "br0"
# type: internal
# Port "dpdk0"
# Interface "dpdk0"
# type: dpdk
# Port "dpdk1"
# Interface "dpdk1"
# type: dpdk
and here's the setup script I'm using to build dpdk and ovs, `a la DevStack:
- creates a screen session and runs commands in parallel in multiple
windows
- uses my specific NIC PCI configuration
#!/bin/bash
SELF=$(cd $( dirname ${BASH_SOURCE[0]} ) && pwd)/$(basename
${BASH_SOURCE[0]})
TMP=/tmp/run-ovs-dpdk
ROOT=/opt/ovs-dpdk
NIC_PCI="0000:47:00.0 0000:47:00.1"
HUGETLBFS_MOUNTPOINT=/mnt/huge
env_setup() {
export DPDK_DIR=${ROOT}/dpdk
export RTE_SDK=${DPDK_DIR}
export RTE_TARGET=x86_64-native-linuxapp-gcc
export DPDK_BUILD=${DPDK_DIR}/${RTE_TARGET}
export OVS_DIR=${ROOT}/ovs
export OVS_DB_SOCK=/usr/local/var/run/openvswitch/db.sock
}
setup() {
screen -S dpdk -X quit
rm -r ${TMP}
mkdir -p ${TMP}
screen -dmS dpdk
screen -S dpdk -X screen
screen -S dpdk -X screen
screen -S dpdk -X screen
screen -S dpdk -X caption always '%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%<'
screen -S dpdk -p 0 -X title shell
screen -S dpdk -p 1 -X title dpdk_build
screen -S dpdk -p 1 -X stuff "${SELF} dpdk-build"
screen -S dpdk -p 1 -X stuff $'\r'
screen -S dpdk -p 2 -X title ovsdb-server
screen -S dpdk -p 2 -X stuff "${SELF} ovsdb-server"
screen -S dpdk -p 2 -X stuff $'\r'
screen -S dpdk -p 3 -X title ovs-vswitchd
screen -S dpdk -p 3 -X stuff "${SELF} ovs-vswitchd"
screen -S dpdk -p 3 -X stuff $'\r'
}
dpdk_build() {
rm ${TMP}/dpdk.*
echo "Remove existing DPDK folder ${DPDK_DIR}"
rm -rf ${DPDK_DIR}
git clone git://dpdk.org/dpdk ${DPDK_DIR}
cd ${DPDK_DIR}
git checkout tags/v1.7.0
patch -p1 <<EOF
diff --git a/config/common_linuxapp b/config/common_linuxapp
index 5bee910..4413620 100644
--- a/config/common_linuxapp
+++ b/config/common_linuxapp
@@ -81,7 +81,7 @@ CONFIG_RTE_BUILD_SHARED_LIB=n
#
# Combine to one single library
#
-CONFIG_RTE_BUILD_COMBINE_LIBS=n
+CONFIG_RTE_BUILD_COMBINE_LIBS=y
CONFIG_RTE_LIBNAME="intel_dpdk"
#
diff --git a/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
b/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
index 5a06383..66ae116 100644
--- a/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
+++ b/lib/librte_eal/linuxapp/kni/ethtool/igb/kcompat.h
@@ -3859,7 +3859,7 @@ static inline struct sk_buff
*__kc__vlan_hwaccel_put_tag(struct sk_buff *skb,
#define HAVE_ENCAP_TSO_OFFLOAD
#endif /* >= 3.10.0 */
-#if ( LINUX_VERSION_CODE < KERNEL_VERSION(3,14,0) )
+#if ( LINUX_VERSION_CODE < KERNEL_VERSION(3,13,0) )
#if (!(RHEL_RELEASE_CODE && RHEL_RELEASE_CODE >=
RHEL_RELEASE_VERSION(7,0)))
#if (!(UBUNTU_RELEASE_CODE == UBUNTU_RELEASE_VERSION(14,4) &&
UBUNTU_KERNEL_CODE >= UBUNTU_KERNEL_VERSION(3,13,0,3
#ifdef NETIF_F_RXHASH
EOF
make T=${RTE_TARGET} config
make T=${RTE_TARGET} install -j 16 || return
touch ${TMP}/dpdk.build
eval $(sed 's/ /\n/g' /proc/cmdline | grep hugepage)
if [[ ${hugepagesz} != "1G" ]] || (( hugepages < 1 )); then
echo "Error: unsupported huge pages configuration"
return
fi
rmmod igb_uio
rmmod uio
modprobe uio
insmod ${DPDK_BUILD}/kmod/igb_uio.ko
$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio ${NIC_PCI}
$DPDK_DIR/tools/dpdk_nic_bind.py --status
umount ${HUGETLBFS_MOUNTPOINT}
mkdir -p ${HUGETLBFS_MOUNTPOINT}
mount -t hugetlbfs -o pagesize=1G none ${HUGETLBFS_MOUNTPOINT}
touch ${TMP}/dpdk.ready
}
ovs_build() {
rm ${TMP}/ovs.build
pkill -9 ovsdb-server
pkill -9 ovs-vswitchd
echo "Remove existing OVS folder"
rm -rf ${OVS_DIR}
git clone https://github.com/openvswitch/ovs.git ${OVS_DIR}
echo "wait for dpdk.build"
while [[ ! -f ${TMP}/dpdk.build ]]; do
sleep 1
done
cd ${OVS_DIR}
./boot.sh
./configure --with-dpdk=${DPDK_BUILD}
make -j 16
if [[ $? -eq 0 ]]; then
touch ${TMP}/ovs.build
return 0
else
return 1
fi
}
ovsdb_server() {
ovs_build || return
rm ${TMP}/ovsdb-server.*
pkill ovsdb-server
sleep 1
pkill -9 ovsdb-server
mkdir -p /usr/local/etc/openvswitch
mkdir -p /usr/local/var/run/openvswitch
rm /usr/local/etc/openvswitch/conf.db
cd ${OVS_DIR}
./ovsdb/ovsdb-tool create \
/usr/local/etc/openvswitch/conf.db \
./vswitchd/vswitch.ovsschema
./ovsdb/ovsdb-server --remote=punix:${OVS_DB_SOCK} \
--remote=db:Open_vSwitch,Open_vSwitch,manager_options \
--private-key=db:Open_vSwitch,SSL,private_key \
--certificate=Open_vSwitch,SSL,certificate \
--bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \
--pidfile 2>&1 \
| tee >(awk "(found==0 && /peak resident set size/){system(\"touch
${TMP}/ovsdb-server.running\");found=1;}") &
while [[ ! -f ${TMP}/ovsdb-server.running ]]; do
sleep 1
done
./utilities/ovs-vsctl --no-wait init
touch ${TMP}/ovsdb-server.ready
echo "ovsdb-server running in background"
}
ovs_vswitchd() {
rm ${TMP}/ovs-vswitchd.*
pkill ovs-vswitchd
sleep 1
pkill -9 ovs-vswitchd
echo "wait for dpdk.ready"
while [[ ! -f ${TMP}/dpdk.ready ]]; do
sleep 1
done
while [[ ! -f ${TMP}/ovsdb-server.ready ]]; do
sleep 1
done
cd ${OVS_DIR}
rmmod openvswitch
modprobe openvswitch
./vswitchd/ovs-vswitchd --dpdk -c 0x3 -n 4 -- unix:${OVS_DB_SOCK}
--pidfile 2>&1 \
| tee >(awk "(found==0 && /ovs-vswitchd \(Open
vSwitch\)/){system(\"touch ${TMP}/ovs-vswitchd.running\");found=1;}") &
while [[ ! -f ${TMP}/ovs-vswitchd.running ]]; do
sleep 1
done
./utilities/ovs-vsctl add-br br0
./utilities/ovs-vsctl set bridge br0 datapath_type=netdev
./utilities/ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0
type=dpdk
./utilities/ovs-vsctl add-port br0 dpdk1 -- set Interface dpdk1
type=dpdk
./utilities/ovs-vsctl show
touch ${TMP}/ovs-vswitchd.ready
echo "ovsdb-server running in background"
}
env_setup
if [[ $# -eq 0 ]]; then
setup
screen -x dpdk -p 1
else
if [[ $1 =~ ^(dpdk-build|ovsdb-server|ovs-vswitchd)$ ]]; then
cmd=${1/-/_}
shift
${cmd} $*
else
echo "error: unknown command"
fi
fi
-----Original Message-----
From: Daniel Badea [mailto:daniel.ba...@windriver.com]
Sent: Friday, September 26, 2014 1:40 PM
To: 'Daniele Di Proietto'
Cc: 'discuss@openvswitch.org'; 'Alex Wang'
Subject: RE: [ovs-discuss] ovs-vswitchd with crashes when adding a dpdk port
Here the output of building OVS master vs DPDK master (make -j 16).
I'm working on an environment setup script to build everything from scratch
in a reproducible way and will post it when it's done.
Thanks,
Daniel
cd dpdk
grep url .git/config
# url = git://dpdk.org/dpdk
git log -n1 --pretty=format:%H%d
# 23fcffe8ffaccf8a2901050e7daa4979597141ed (HEAD, origin/master,
origin/HEAD, master)
cd ../ovs
grep url .git/config
# url = https://github.com/openvswitch/ovs.git
git log -n1 --pretty=format:%H%d
# 260f76b7e3cf6bd52a2a80d5186f2a68b0123017 (HEAD, origin/master,
origin/HEAD, master)
...
lib/ofpbuf.h:363:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_data':
lib/ofpbuf.h:368:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h: In function 'ofpbuf_size':
lib/ofpbuf.h:383:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_size':
lib/ofpbuf.h:388:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h:389:12: error: 'struct rte_mbuf' has no member named 'pkt'
In file included from
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_ring.h:98:0,
from
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:74,
from
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_mbuf.h:56,
from
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_ethdev.h:178,
from lib/netdev-dpdk.h:13,
from lib/ofpbuf.h:25,
from lib/dpif.h:392,
from lib/dpif-provider.h:26,
from lib/dpif.c:18:
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_common.h: In
function 'rte_is_aligned':
/opt/ovs-dpdk/dpdk/x86_64-native-linuxapp-gcc/include/rte_common.h:176:9:
warning: cast from function call of type 'uintptr_t' to non-matching type
'void *' [-Wbad-function-cast] In file included from lib/dpif.h:392:0,
from lib/dpctl.c:33:
lib/ofpbuf.h: In function 'ofpbuf_data':
lib/ofpbuf.h:363:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_data':
lib/ofpbuf.h:368:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h: In function 'ofpbuf_size':
lib/ofpbuf.h:383:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_size':
lib/ofpbuf.h:388:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h:389:12: error: 'struct rte_mbuf' has no member named 'pkt'
In file included from lib/dpif.h:392:0,
from lib/dpif-provider.h:26,
from lib/dpif.c:18:
lib/ofpbuf.h: In function 'ofpbuf_data':
lib/ofpbuf.h:363:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_data':
lib/ofpbuf.h:368:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h: In function 'ofpbuf_size':
lib/ofpbuf.h:383:19: error: 'const struct rte_mbuf' has no member named
'pkt'
lib/ofpbuf.h: In function 'ofpbuf_set_size':
lib/ofpbuf.h:388:12: error: 'struct rte_mbuf' has no member named 'pkt'
lib/ofpbuf.h:389:12: error: 'struct rte_mbuf' has no member named 'pkt'
In file included from lib/dpif-netdev.c:57:0:
lib/packet-dpif.h: In function 'dpif_packet_get_dp_hash':
lib/packet-dpif.h:52:26: error: 'struct rte_mbuf' has no member named 'pkt'
lib/packet-dpif.h: In function 'dpif_packet_set_dp_hash':
lib/packet-dpif.h:62:19: error: 'struct rte_mbuf' has no member named 'pkt'
lib/dpctl.c: In function 'ofpbuf_size':
lib/ofpbuf.h:384:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpctl.c: In function 'ofpbuf_data':
lib/ofpbuf.h:364:1: warning: control reaches end of non-void function
[-Wreturn-type]
make[2]: *** [lib/dpctl.lo] Error 1
In file included from lib/dpif.c:38:0:
lib/packet-dpif.h: In function 'dpif_packet_get_dp_hash':
lib/packet-dpif.h:52:26: error: 'struct rte_mbuf' has no member named 'pkt'
lib/packet-dpif.h: In function 'dpif_packet_set_dp_hash':
lib/packet-dpif.h:62:19: error: 'struct rte_mbuf' has no member named 'pkt'
lib/dpif-netdev.c: In function 'ofpbuf_size':
lib/ofpbuf.h:384:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpif-netdev.c: In function 'dpif_packet_get_dp_hash':
lib/packet-dpif.h:56:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpif-netdev.c: In function 'ofpbuf_data':
lib/ofpbuf.h:364:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpif.c: In function 'ofpbuf_size':
lib/ofpbuf.h:384:1: warning: control reaches end of non-void function
[-Wreturn-type]
lib/dpif.c: In function 'ofpbuf_data':
lib/ofpbuf.h:364:1: warning: control reaches end of non-void function
[-Wreturn-type]
make[2]: *** [lib/dpif.lo] Error 1
make[2]: *** [lib/dpif-netdev.lo] Error 1
make[2]: Leaving directory `/opt/ovs-dpdk/ovs'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/opt/ovs-dpdk/ovs'
make: *** [all] Error 2
-----Original Message-----
From: Daniele Di Proietto [mailto:ddiproie...@vmware.com]
Sent: Thursday, September 25, 2014 11:14 PM
To: Daniel Badea
Cc: discuss@openvswitch.org; Alex Wang
Subject: Re: [ovs-discuss] ovs-vswitchd with crashes when adding a dpdk port
Hi Daniel,
On 9/25/14, 10:34 AM, "Daniel Badea" <daniel.ba...@windriver.com> wrote:
>I encountered the following issue: ovs-vswitchd with crashes when
>adding a dpdk port.
>
>Building ovs with dpdk support is not straightforward because:
> - ovs doest not build with the latest dpdk version 1.7.1
We're regularly building with dpdk 1.7.1 (dpdk 1.7.0 is also supported).
Can you please tell us the error that you're experiencing, so that we can
try to fix it?
> - the document here
>http://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob_plai
>n;f =INSTALL.DPDK;hb=HEAD refers to dpdk 1.6.0r2 but that doesn't work
>either.
That's an old version of the file. Please read INSTALL.DPDK in the source
tree (or
https://raw.githubusercontent.com/openvswitch/ovs/master/INSTALL.DPDK)
>
>So I checked out dpdk 1.7.0 did some minor changes (combine libs and
>kcompat fix for kernel 3.13.0) Now ovs builds ok but:
> - if the database is empty then ovs-vswitchd starts but crashes on
>ovs-vsctl add-port ... type=dpdk
> - if the database already contains dpdk port configuration then
>ovs-vswitchd crashes at startup
>
>Logs seem to indicate a numa socket = -1 problem.
>
>Any ideea how to fix his?
We've managed to reproduce the crash on a single node numa machine. Alex is
about to post a fix to the mailing list.
Thank you for the detailed report!
Daniele
_______________________________________________
discuss mailing list
discuss@openvswitch.org
http://openvswitch.org/mailman/listinfo/discuss
