This patchset, targetted for 16.11, introduces the support of rx and tx offload in virtio pmd. To achieve this, some new mbuf flags must be introduced, as discussed in [1].
It applies on master + a patch fixing the testpmd csum engine: http://dpdk.org/dev/patchwork/patch/16538/ The new mbuf checksum flags are backward compatible for current applications that assume that unknown_csum = good_cum (since there was only a bad_csum flag). But it the patchset is integrated, we should consider updating the PMDs to match the new API for 16.11. [1] http://dpdk.org/ml/archives/dev/2016-May/039920.html changes v2 -> v3 - fix typo in release note - add unlikely() in cksum calculation error case - add likely() in virtio rx function when cksum != 0xffff - return an error code instead of the cksum in rte_raw_cksum_mbuf() - do not access to the virtio header if no offload is negotiated (rx and tx) - return an error if offload cannot be negotiated - use offsetof() instead of magic hardcoded values for cksum offsets - changefix some commit titles changes v1 -> v2 - change mbuf checksum calculation static inline - fix checksum calculation for protocol where csum=0 means no csum - move mbuf checksum calculation in librte_net - use RTE_MIN() to set max rx/tx queue - rebase on top of head Olivier Matz (12): virtio: move device initialization in a function virtio: setup and start cq in configure callback virtio: reinitialize the device in configure callback net: add function to calculate a checksum in a mbuf mbuf: add new Rx checksum mbuf flags app/testpmd: fix checksum stats in csum engine mbuf: new flag for LRO app/testpmd: display lro segment size virtio: add Rx checksum offload support virtio: add Tx checksum offload support virtio: add Lro support virtio: add Tso support app/test-pmd/csumonly.c | 8 +- doc/guides/rel_notes/release_16_11.rst | 16 ++ drivers/net/virtio/virtio_ethdev.c | 197 ++++++++++++++-------- drivers/net/virtio/virtio_ethdev.h | 18 +- drivers/net/virtio/virtio_pci.h | 4 +- drivers/net/virtio/virtio_rxtx.c | 298 ++++++++++++++++++++++++++++++--- drivers/net/virtio/virtqueue.h | 1 + lib/librte_mbuf/rte_mbuf.c | 18 +- lib/librte_mbuf/rte_mbuf.h | 58 ++++++- lib/librte_net/rte_ip.h | 71 ++++++++ 10 files changed, 580 insertions(+), 109 deletions(-) Test plan ========= (replayed on v3) Platform description -------------------- guest (dpdk) +----------------+ | | | | | port0 +-----<---+ | ixgbe / | | | directio | | | | | | port1 | ^ flow1 +----------------+ | (flow2 is the reverse) | | | virtio | v | +----------------+ | | tap0 / | | |1.1.1.1 / | | |ns-tap / | | | / | | | / ixgbe2 +------>--+ | / 1.1.1.2 | |/ ns-ixgbe | +----------------+ host (linux, vhost-net) flow1: host -(ixgbe)-> guest -(virtio)-> host 1.1.1.2 -> 1.1.1.1 flow2: host -(virtio)-> guest -(ixgbe)-> host 1.1.1.2 -> 1.1.1.1 Host configuration ------------------ Start qemu with: - a ne2k management interface to avoi any conflict with dpdk - 2 ixgbe interfaces given to with vm through vfio - a virtio net device, connected to a tap interface through vhost-net /usr/bin/qemu-system-x86_64 -k fr -daemonize --enable-kvm -m 1G -cpu host \ -smp 3 -serial telnet::40564,server,nowait -serial null \ -qmp tcp::44340,server,nowait -monitor telnet::49229,server,nowait \ -device ne2k_pci,mac=de:ad:de:01:02:03,netdev=user.0,addr=03 \ -netdev user,id=user.0,hostfwd=tcp::34965-:22 \ -device vfio-pci,host=0000:04:00.0 -device vfio-pci,host=0000:04:00.1 \ -netdev type=tap,id=vhostnet0,script=no,vhost=on,queues=8 \ -device virtio-net-pci,netdev=vhostnet0,ioeventfd=on,mq=on,vectors=17 \ -hda "/path/to/ubuntu-14.04-template.qcow2" \ -snapshot -vga none -display none Move the tap interface in a netns, and configure it: ip netns add ns-tap ip netns exec ns-tap ip l set lo up ip link set tap0 netns ns-tap ip netns exec ns-tap ip l set tap0 down ip netns exec ns-tap ip l set addr 02:00:00:00:00:01 dev tap0 ip netns exec ns-tap ip l set tap0 up ip netns exec ns-tap ip a a 1.1.1.1/24 dev tap0 ip netns exec ns-tap arp -s 1.1.1.2 02:00:00:00:00:00 ip netns exec ns-tap ip a Move the ixgbe interface in a netns, and configure it: IXGBE=ixgbe2 ip netns add ns-ixgbe ip netns exec ns-ixgbe ip l set lo up ip link set ${IXGBE} netns ns-ixgbe ip netns exec ns-ixgbe ip l set ${IXGBE} down ip netns exec ns-ixgbe ip l set addr 02:00:00:00:00:00 dev ${IXGBE} ip netns exec ns-ixgbe ip l set ${IXGBE} up ip netns exec ns-ixgbe ip a a 1.1.1.2/24 dev ${IXGBE} ip netns exec ns-ixgbe arp -s 1.1.1.1 02:00:00:00:00:01 ip netns exec ns-ixgbe ip a Guest configuration ------------------- List of pci devices: 00:02.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 00:03.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL-8029(AS) [10ec:8029] 00:04.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 00:05.0 Ethernet controller [0200]: Red Hat, Inc Virtio network device [1af4:1000] Compile dpdk: cd dpdk.org make config T=x86_64-native-linuxapp-gcc make -j4 Prepare environment: mkdir -p /mnt/huge mount -t hugetlbfs nodev /mnt/huge echo 256 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages modprobe uio_pci_generic python tools/dpdk_nic_bind.py -b uio_pci_generic 0000:00:02.0 python tools/dpdk_nic_bind.py -b uio_pci_generic 0000:00:05.0 Run test ======== The test uses iperf to validate connectivity between the 2 netns of the host and trough the guest. Iperf is run with: # flow1: host -(ixgbe)-> guest -(virtio)-> host ip netns exec ns-tap iperf -s ip netns exec ns-ixgbe iperf -c 1.1.1.1 -t 10 # flow2: host -(virtio)-> guest -(ixgbe)-> host ip netns exec ns-ixgbe iperf -s ip netns exec ns-tap iperf -c 1.1.1.2 -t 10 The guest runs testpmd with csum forward engine, its configuration depends on the test case. test1: large packets (lro/tso) ------------------------------ Configuration of testpmd: ./build/app/testpmd -l 0,1 --log-level 8 -- --total-num-mbufs=16384 \ -i --port-topology=chained --disable-hw-vlan-filter \ --disable-hw-vlan-strip --enable-rx-cksum --enable-lro \ --crc-strip --txqflags=0 set fwd csum tso set 1440 0 csum set ip hw 0 csum set tcp hw 0 tso set 1440 1 #csum set ip hw 1 # not supported by virtio csum set tcp hw 1 start Iperf log: root at ubuntu1404:~# ip netns exec ns-ixgbe iperf -c 1.1.1.1 -t 10 ------------------------------------------------------------ Client connecting to 1.1.1.1, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 1.1.1.2 port 54460 connected with 1.1.1.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 6.14 GBytes 5.27 Gbits/sec root at ubuntu1404:~# ip netns exec ns-tap iperf -c 1.1.1.2 -t 10 ------------------------------------------------------------ Client connecting to 1.1.1.2, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 1.1.1.1 port 58312 connected with 1.1.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 6.70 GBytes 5.76 Gbits/sec Example of what we see with "set verbose 1" in testpmd: -- flow1: ixgbe2 -> port0 (ixgbe) -> testpmd -> port1 (virtio) <-> tap0 port=0, mbuf=0x7f968ad9fdc0, pkt_len=24682, nb_segs=13: rx: l2_len=14 ethertype=800 l3_len=20 l4_proto=6 l4_len=32 flags=PKT_RX_L4_CKSUM_UNKNOWN PKT_RX_IP_CKSUM_UNKNOWN tx: m->l2_len=14 m->l3_len=20 m->l4_len=32 tx: m->tso_segsz=1440 tx: flags=PKT_TX_IP_CKSUM PKT_TX_L4_NO_CKSUM PKT_TX_TCP_SEG PKT_TX_IPV4 -- flow2: tap0 -> port1 (virtio)-> testpmd -> port0 (ixgbe) -> ixgbe2 port=1, mbuf=0x7f968acc9f40, pkt_len=42058, nb_segs=21: rx: l2_len=14 ethertype=800 l3_len=20 l4_proto=6 l4_len=32 flags=PKT_RX_L4_CKSUM_NONE PKT_RX_IP_CKSUM_UNKNOWN PKT_RX_LRO rx: m->lro_segsz=1440 tx: m->l2_len=14 m->l3_len=20 m->l4_len=32 tx: m->tso_segsz=1440 tx: flags=PKT_TX_IP_CKSUM PKT_TX_L4_NO_CKSUM PKT_TX_TCP_SEG PKT_TX_IPV4 test2: hardware checksum only ----------------------------- Configuration of testpmd: ./build/app/testpmd -l 0,1 --log-level 8 -- --total-num-mbufs=16384 \ -i --port-topology=chained --disable-hw-vlan-filter \ --disable-hw-vlan-strip --enable-rx-cksum --crc-strip --txqflags=0 set fwd csum csum set ip hw 0 csum set tcp hw 0 csum set tcp hw 1 start Iperf log: root at ubuntu1404:~# ip netns exec ns-ixgbe iperf -c 1.1.1.1 -t 10 ------------------------------------------------------------ Client connecting to 1.1.1.1, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 1.1.1.2 port 54462 connected with 1.1.1.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 4.49 GBytes 3.86 Gbits/sec root at ubuntu1404:~# ip netns exec ns-tap iperf -c 1.1.1.2 -t 10 ------------------------------------------------------------ Client connecting to 1.1.1.2, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 1.1.1.1 port 58314 connected with 1.1.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 6.66 GBytes 5.72 Gbits/sec Example of what we see with "set verbose 1" in testpmd: -- flow1: ixgbe2 -> port0 (ixgbe) -> testpmd -> port1 (virtio) <-> tap0 port=0, mbuf=0x7f0adca89b40, pkt_len=1514, nb_segs=1: rx: l2_len=14 ethertype=800 l3_len=20 l4_proto=6 l4_len=32 flags=PKT_RX_L4_CKSUM_UNKNOWN PKT_RX_IP_CKSUM_UNKNOWN tx: m->l2_len=14 m->l3_len=20 m->l4_len=32 tx: flags=PKT_TX_TCP_CKSUM PKT_TX_IPV4 -- flow2: tap0 -> port1 (virtio)-> testpmd -> port0 (ixgbe) -> ixgbe2 port=1, mbuf=0x7f0adcb98d80, pkt_len=1514, nb_segs=1: rx: l2_len=14 ethertype=800 l3_len=20 l4_proto=6 l4_len=32 flags=PKT_RX_L4_CKSUM_NONE PKT_RX_IP_CKSUM_UNKNOWN tx: m->l2_len=14 m->l3_len=20 m->l4_len=32 tx: flags=PKT_TX_IP_CKSUM PKT_TX_TCP_CKSUM PKT_TX_IPV4 test3: no offload ----------------- Configuration of testpmd: ./build/app/testpmd -l 0,1 --log-level 8 -- --total-num-mbufs=16384 \ -i --port-topology=chained --disable-hw-vlan-filter --disable-hw-vlan-strip set fwd csum start Iperf log: root at ubuntu1404:~# ip netns exec ns-ixgbe iperf -c 1.1.1.1 -t 10 ------------------------------------------------------------ Client connecting to 1.1.1.1, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 1.1.1.2 port 54466 connected with 1.1.1.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 4.29 GBytes 3.68 Gbits/sec root at ubuntu1404:~# ip netns exec ns-tap iperf -c 1.1.1.2 -t 10 ------------------------------------------------------------ Client connecting to 1.1.1.2, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 1.1.1.1 port 58316 connected with 1.1.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 6.66 GBytes 5.72 Gbits/sec Example of what we see with "set verbose 1" in testpmd: -- flow1: ixgbe2 -> port0 (ixgbe) -> testpmd -> port1 (virtio) <-> tap0 port=0, mbuf=0x7faf38b3e700, pkt_len=1514, nb_segs=1: rx: l2_len=14 ethertype=800 l3_len=20 l4_proto=6 l4_len=32 flags=PKT_RX_L4_CKSUM_UNKNOWN PKT_RX_IP_CKSUM_UNKNOWN tx: flags=PKT_TX_L4_NO_CKSUM PKT_TX_IPV4 -- flow2: tap0 -> port1 (virtio)-> testpmd -> port0 (ixgbe) -> ixgbe2 port=1, mbuf=0x7faf38b71500, pkt_len=1514, nb_segs=1: rx: l2_len=14 ethertype=800 l3_len=20 l4_proto=6 l4_len=32 flags=PKT_RX_L4_CKSUM_UNKNOWN PKT_RX_IP_CKSUM_UNKNOWN tx: flags=PKT_TX_L4_NO_CKSUM PKT_TX_IPV4 -- 2.8.1
