On Fri, May 31, 2019 at 8:58 AM Ilya Maximets <i.maxim...@samsung.com> wrote:
>
> On 31.05.2019 1:08, William Tu wrote:
> > Hi Ilya,
> >
> > Thanks for the comments.
> >
> > On Thu, May 30, 2019 at 12:47 AM Ilya Maximets <i.maxim...@samsung.com>
> > wrote:
> >>
> >> On 29.05.2019 20:57, William Tu wrote:
> >>> On Wed, May 29, 2019 at 10:51 AM William Tu <u9012...@gmail.com> wrote:
> >>>>
> >>
> >> Thanks for the logs. It looks like a bug in DPDK.
> >> Does rte_eal_hotplug_remove() works without issues?
> >> Unfortunately, I have no setup to test myself.
> >
> > Right, no issue. reason below:
> >
> >>
> >> You may also use following patch to get more information about detaching
> >> failure:
> >> https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html
> >
> > Thanks, -1 is returned
> > 2019-05-30T19:41:26.467Z|00055|netdev_dpdk|ERR|Device
> > 'vdev:net_af_xdp0,iface=enp2s0,queue=0' can not be detached: Operation
> > not permitted.
> >
> > The root cause is at
> > rte_dev_remove
> > local_dev_remove
> > ret = dev->bus->unplug(dev);
> > rte_pmd_af_xdp_remove()
> >
> > At rte_pmd_af_xdp_remove()
> > /* find the ethdev entry */
> > eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
> > if (eth_dev == NULL)
> > return -1; // failed here
>
> Thanks for detailed investigation. So, it's just a DPDK bug.
> I think that we need to simply replace 'return -1' with 'return 0' here.
>
> The life-cycle of a DPDK port is following:
>
> rte_dev_probe() <-- attaches device to dpdk bus.
> rte_eth_dev_configure()
> rte_eth_dev_start()
> rte_eth_dev_stop()
> rte_eth_dev_close() <-- closes the eth device.
> rte_dev_remove() <-- detaches device from dpdk bus.
>
Thanks for the explanation. Much clear to me now.
> af_xdp pmd driver advertises RTE_ETH_DEV_CLOSE_REMOVE, so it frees all
> the resources on rte_eth_dev_close(). rte_dev_remove() tries to detach
> device from the vdev bus. It subsequently calls rte_pmd_af_xdp_remove()
> that tries to free already freed resources and fails.
> I think that rte_pmd_af_xdp_remove() should just ignore this case
> reporting successful removal, otherwise device will never be detached
> from vdev bus reporting failure of driver->remove() callback.
>
> Will you prepare the patch for this?
Yes, I will test it and send out patch to dpdk-dev.
Regards,
William
> If you wish, I could prepare it myself on Monday.
>
> Best regards, Ilya Maximets.
>
> >
> > And the reason rte_eth_dev_allocated find no device is that
> > at _rte_eth_dev_allocated
> > the rte_eth_devices[i].data is an empty string.
> >
> > And the reason is that at rte_eth_dev_close
> > it calls
> > if ((dev->data->dev_flags & RTE_ETH_DEV_CLOSE_REMOVE) != 0) {
> > /* new behaviour: send event + reset state + free all data */
> > rte_eth_dev_release_port(dev);
> >
> > the rte_eth_dev_release_port clear the dev->data->name
> > memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
> >
> >>
> >> Another possible issue is that you're using driver base name (net_af_xdp)
> >> as a name of DPDK device. Maybe this causes some issues.
> >> Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0".
> >
> > Yes, I use the above comment.
> >
> > Regards,
> > William
> >
> >>
> >> Best regards, Ilya Maximets.
> >>
> >>>>>
> >>>>> Another option you may try is:
> >>>>>
> >>>>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \
> >>>>> options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0"
> >>>>>
> >>>> It actually works! When adding the device:
> >>>> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device
> >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK
> >>>> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id:
> >>>> 0, core id: 11 created.
> >>>> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd
> >>>> threads on numa node 0
> >>>> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already
> >>>> stopped
> >>>> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is
> >>>> not supported on port 0
> >>>>
> >>>> However , when removing the device, some errors show up.
> >>>> ovs-vsctl del-port br0 afxdp-p0
> >>>> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted
> >>>> interface afxdp-p0 on port 1
> >>>> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id:
> >>>> 0, core id: 11 destroyed.
> >>>> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the
> >>>> device (net_af_xdp)
> >>>> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device
> >>>> on primary process
> >>>> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device
> >>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached
> >>>>
> >>>> I also measured the performance, it works fine and shows around 7Mpps.
> >>>>
> >>>>> so DPDK will not have to guess the bus type.
> >>>>>
> >>>>>>
> >>>>>> $ ovs-vsctl show
> >>>>>> Port "afxdp-p0"
> >>>>>> Interface "afxdp-p0"
> >>>>>> type: dpdk
> >>>>>> options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
> >>>>>>
> >>>>>> The performance of physical device loopback shows around 7Mpps with
> >>>>>> 64B pkt.
> >>>>>>
> >>>>>> $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
> >>>>>>
> >>>>>> actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"
> >>>>>>
> >>>>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.
> >>>>>
> >>>>> You may use 'dpdk-latest' OVS branch. The proper fix exists there.
> >>>>> In general, patches that intended to work only with latest DPDK that
> >>>>> not yet supported on master should be made on top of 'dpdk-latest'
> >>>>> branch and have [PATCH dpdk-latest] subject prefix.
> >>>>
> >>>> Got it, thank you.
> >>>> William
> >>>>
> >>>>>
> >>>>> Best regards, Ilya Maximets.
> >>>
> >>>
> >
> >
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
