Ok, I'll try with something closer to that configuration... But still x86 :)
> -----Original Message----- > From: Ilya Maximets [mailto:[email protected]] > Sent: Friday, October 13, 2017 1:06 PM > To: O Mahony, Billy <[email protected]>; [email protected] > Cc: Maxime Coquelin <[email protected]>; Heetae Ahn > <[email protected]> > Subject: Re: [ovs-dev] [PATCH] netdev-dpdk: Fix calling vhost API with > negative > vid. > > On 13.10.2017 14:38, O Mahony, Billy wrote: > > Hi Ilya, > > > > > >> Issue can be reproduced by stopping DPDK application (testpmd) inside > >> guest while heavy traffic flows to this VM. > >> > > > > I tried both quitting testpmd without stopping the forwarding task and> > > simply > killing testpmd without crashing vswitch in the host. > > > > What versions of dpdk are you using in the guest and host? > > Versions below, but I don't think that it's so important. > > Host: 17.05.2 > Guest: 16.07-rc1 > > > > > Are you using dpdkvhostuser or dpdkvhostuserclient type ports? > > dpdkvhostuserclient. > > The complete test scenario where I saw this behaviour was: > > 2 VMs with 4 queues per vhostuserclient port. > VM1 - OVS - VM2 > > VM1 runs testpmd with --rxq=4 --txq=4 --nb-cores=4 --eth-peer=0,MAC2 -- > forward-mode=mac > VM2 runs testpmd with --rxq=4 --txq=4 --nb-cores=4 --eth-peer=0,MAC1 -- > forward-mode=txonly > > OVS with 8 pmd threads (1 core per queue). > action=NORMAL > > Steps: > > 1. Starting testpmd in both VMs (non-interactive mode) > 2. Waiting a while > 3. Pushing <enter> in VM1 console. > --> OVS crashes while testpmd termination. > > The most important thing, I guess, is that I'm using ARMv8 machine for that. > It could be not so easy to reproduce on x86 system (I didn't try). > > Best regards, Ilya Maximets. > > > > > Thanks, > > Billy. > > > >> -----Original Message----- > >> From: [email protected] [mailto:ovs-dev- > >> [email protected]] On Behalf Of Ilya Maximets > >> Sent: Friday, October 6, 2017 11:50 AM > >> To: [email protected] > >> Cc: Ilya Maximets <[email protected]>; Maxime Coquelin > >> <[email protected]>; Heetae Ahn > <[email protected]> > >> Subject: [ovs-dev] [PATCH] netdev-dpdk: Fix calling vhost API with negative > vid. > >> > >> Currently, rx and tx functions for vhost interfaces always obtain > >> 'vid' twice. First time inside 'is_vhost_running' for checking the > >> value and the second time in enqueue/dequeue function calls to > >> send/receive packets. But second time we're not checking the returned > >> value. If vhost device will be destroyed between checking and > >> enqueue/dequeue, DPDK API will be called with '-1' instead of valid 'vid'. > DPDK API does not validate the 'vid'. > >> This leads to getting random memory value as a pointer to internal > >> device structure inside DPDK. Access by this pointer leads to > >> segmentation fault. For > >> example: > >> > >> |00503|dpdk|INFO|VHOST_CONFIG: read message > >> VHOST_USER_GET_VRING_BASE > >> [New Thread 0x7fb6754910 (LWP 21246)] > >> > >> Program received signal SIGSEGV, Segmentation fault. > >> rte_vhost_enqueue_burst at lib/librte_vhost/virtio_net.c:630 > >> 630 if (dev->features & (1 << VIRTIO_NET_F_MRG_RXBUF)) > >> (gdb) bt full > >> #0 rte_vhost_enqueue_burst at lib/librte_vhost/virtio_net.c:630 > >> dev = 0xffffffff > >> #1 __netdev_dpdk_vhost_send at lib/netdev-dpdk.c:1803 > >> tx_pkts = <optimized out> > >> cur_pkts = 0x7f340084f0 > >> total_pkts = 32 > >> dropped = 0 > >> i = <optimized out> > >> retries = 0 > >> ... > >> (gdb) p *((struct netdev_dpdk *) netdev) > >> $8 = { ... , > >> flags = (NETDEV_UP | NETDEV_PROMISC), ... , > >> vid = {v = -1}, > >> vhost_reconfigured = false, ... } > >> > >> Issue can be reproduced by stopping DPDK application (testpmd) inside > >> guest while heavy traffic flows to this VM. > >> > >> Fix that by obtaining and checking the 'vid' only once. > >> > >> CC: Ciara Loftus <[email protected]> > >> Fixes: 0a0f39df1d5a ("netdev-dpdk: Add support for DPDK 16.07") > >> Signed-off-by: Ilya Maximets <[email protected]> > >> --- > >> lib/netdev-dpdk.c | 14 +++++++------- > >> 1 file changed, 7 insertions(+), 7 deletions(-) > >> > >> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index > >> c60f46f..bf30bb0 100644 > >> --- a/lib/netdev-dpdk.c > >> +++ b/lib/netdev-dpdk.c > >> @@ -1637,18 +1637,18 @@ netdev_dpdk_vhost_rxq_recv(struct netdev_rxq > >> *rxq, > >> struct dp_packet_batch *batch) { > >> struct netdev_dpdk *dev = netdev_dpdk_cast(rxq->netdev); > >> - int qid = rxq->queue_id; > >> struct ingress_policer *policer = > >> netdev_dpdk_get_ingress_policer(dev); > >> uint16_t nb_rx = 0; > >> uint16_t dropped = 0; > >> + int qid = rxq->queue_id; > >> + int vid = netdev_dpdk_get_vid(dev); > >> > >> - if (OVS_UNLIKELY(!is_vhost_running(dev) > >> + if (OVS_UNLIKELY(vid < 0 || !dev->vhost_reconfigured > >> || !(dev->flags & NETDEV_UP))) { > >> return EAGAIN; > >> } > >> > >> - nb_rx = rte_vhost_dequeue_burst(netdev_dpdk_get_vid(dev), > >> - qid * VIRTIO_QNUM + VIRTIO_TXQ, > >> + nb_rx = rte_vhost_dequeue_burst(vid, qid * VIRTIO_QNUM + > >> + VIRTIO_TXQ, > >> dev->dpdk_mp->mp, > >> (struct rte_mbuf **) batch->packets, > >> NETDEV_MAX_BURST); @@ -1783,10 > >> +1783,11 @@ __netdev_dpdk_vhost_send(struct netdev *netdev, int qid, > >> unsigned int total_pkts = cnt; > >> unsigned int dropped = 0; > >> int i, retries = 0; > >> + int vid = netdev_dpdk_get_vid(dev); > >> > >> qid = dev->tx_q[qid % netdev->n_txq].map; > >> > >> - if (OVS_UNLIKELY(!is_vhost_running(dev) || qid < 0 > >> + if (OVS_UNLIKELY(vid < 0 || !dev->vhost_reconfigured || qid < 0 > >> || !(dev->flags & NETDEV_UP))) { > >> rte_spinlock_lock(&dev->stats_lock); > >> dev->stats.tx_dropped+= cnt; @@ -1805,8 +1806,7 @@ > >> __netdev_dpdk_vhost_send(struct netdev *netdev, int qid, > >> int vhost_qid = qid * VIRTIO_QNUM + VIRTIO_RXQ; > >> unsigned int tx_pkts; > >> > >> - tx_pkts = rte_vhost_enqueue_burst(netdev_dpdk_get_vid(dev), > >> - vhost_qid, cur_pkts, cnt); > >> + tx_pkts = rte_vhost_enqueue_burst(vid, vhost_qid, cur_pkts, > >> + cnt); > >> if (OVS_LIKELY(tx_pkts)) { > >> /* Packets have been sent.*/ > >> cnt -= tx_pkts; > >> -- > >> 2.7.4 > >> > >> _______________________________________________ > >> dev mailing list > >> [email protected] > >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
