On 31.05.2019 1:08, William Tu wrote:
> Hi Ilya,
>
> Thanks for the comments.
>
> On Thu, May 30, 2019 at 12:47 AM Ilya Maximets <[email protected]> wrote:
>>
>> On 29.05.2019 20:57, William Tu wrote:
>>> On Wed, May 29, 2019 at 10:51 AM William Tu <[email protected]> wrote:
>>>>
>>
>> Thanks for the logs. It looks like a bug in DPDK.
>> Does rte_eal_hotplug_remove() works without issues?
>> Unfortunately, I have no setup to test myself.
>
> Right, no issue. reason below:
>
>>
>> You may also use following patch to get more information about detaching
>> failure: https://mail.openvswitch.org/pipermail/ovs-dev/2019-May/359367.html
>
> Thanks, -1 is returned
> 2019-05-30T19:41:26.467Z|00055|netdev_dpdk|ERR|Device
> 'vdev:net_af_xdp0,iface=enp2s0,queue=0' can not be detached: Operation
> not permitted.
>
> The root cause is at
> rte_dev_remove
> local_dev_remove
> ret = dev->bus->unplug(dev);
> rte_pmd_af_xdp_remove()
>
> At rte_pmd_af_xdp_remove()
> /* find the ethdev entry */
> eth_dev = rte_eth_dev_allocated(rte_vdev_device_name(dev));
> if (eth_dev == NULL)
> return -1; // failed here
Thanks for detailed investigation. So, it's just a DPDK bug.
I think that we need to simply replace 'return -1' with 'return 0' here.
The life-cycle of a DPDK port is following:
rte_dev_probe() <-- attaches device to dpdk bus.
rte_eth_dev_configure()
rte_eth_dev_start()
rte_eth_dev_stop()
rte_eth_dev_close() <-- closes the eth device.
rte_dev_remove() <-- detaches device from dpdk bus.
af_xdp pmd driver advertises RTE_ETH_DEV_CLOSE_REMOVE, so it frees all
the resources on rte_eth_dev_close(). rte_dev_remove() tries to detach
device from the vdev bus. It subsequently calls rte_pmd_af_xdp_remove()
that tries to free already freed resources and fails.
I think that rte_pmd_af_xdp_remove() should just ignore this case
reporting successful removal, otherwise device will never be detached
from vdev bus reporting failure of driver->remove() callback.
Will you prepare the patch for this?
If you wish, I could prepare it myself on Monday.
Best regards, Ilya Maximets.
>
> And the reason rte_eth_dev_allocated find no device is that
> at _rte_eth_dev_allocated
> the rte_eth_devices[i].data is an empty string.
>
> And the reason is that at rte_eth_dev_close
> it calls
> if ((dev->data->dev_flags & RTE_ETH_DEV_CLOSE_REMOVE) != 0) {
> /* new behaviour: send event + reset state + free all data */
> rte_eth_dev_release_port(dev);
>
> the rte_eth_dev_release_port clear the dev->data->name
> memset(eth_dev->data, 0, sizeof(struct rte_eth_dev_data));
>
>>
>> Another possible issue is that you're using driver base name (net_af_xdp)
>> as a name of DPDK device. Maybe this causes some issues.
>> Try following devargs: "vdev:net_af_xdp0,iface=enp2s0,queue=0".
>
> Yes, I use the above comment.
>
> Regards,
> William
>
>>
>> Best regards, Ilya Maximets.
>>
>>>>>
>>>>> Another option you may try is:
>>>>>
>>>>> ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type=dpdk \
>>>>> options:dpdk-devargs="vdev:net_af_xdp,iface=enp2s0,queue=0"
>>>>>
>>>> It actually works! When adding the device:
>>>> 2019-05-29T17:45:47.128Z|00044|netdev_dpdk|INFO|Device
>>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' attached to DPDK
>>>> 2019-05-29T17:45:47.128Z|00045|dpif_netdev|INFO|PMD thread on numa_id:
>>>> 0, core id: 11 created.
>>>> 2019-05-29T17:45:47.128Z|00046|dpif_netdev|INFO|There are 1 pmd
>>>> threads on numa node 0
>>>> 2019-05-29T17:45:47.128Z|00047|dpdk|INFO|Device with port_id=0 already
>>>> stopped
>>>> 2019-05-29T17:45:47.208Z|00048|netdev_dpdk|WARN|Rx checksum offload is
>>>> not supported on port 0
>>>>
>>>> However , when removing the device, some errors show up.
>>>> ovs-vsctl del-port br0 afxdp-p0
>>>> 2019-05-29T17:47:00.318Z|00053|bridge|INFO|bridge br0: deleted
>>>> interface afxdp-p0 on port 1
>>>> 2019-05-29T17:47:00.318Z|00054|dpif_netdev|INFO|PMD thread on numa_id:
>>>> 0, core id: 11 destroyed.
>>>> 2019-05-29T17:47:00.651Z|00055|dpdk|ERR|EAL: Driver cannot detach the
>>>> device (net_af_xdp)
>>>> 2019-05-29T17:47:00.651Z|00056|dpdk|ERR|EAL: Failed to detach device
>>>> on primary process
>>>> 2019-05-29T17:47:00.651Z|00057|netdev_dpdk|ERR|Device
>>>> 'vdev:net_af_xdp,iface=enp2s0,queue=0' can not be detached
>>>>
>>>> I also measured the performance, it works fine and shows around 7Mpps.
>>>>
>>>>> so DPDK will not have to guess the bus type.
>>>>>
>>>>>>
>>>>>> $ ovs-vsctl show
>>>>>> Port "afxdp-p0"
>>>>>> Interface "afxdp-p0"
>>>>>> type: dpdk
>>>>>> options: {dpdk-devargs="net_af_xdp,iface=enp2s0,queue=0"}
>>>>>>
>>>>>> The performance of physical device loopback shows around 7Mpps with 64B
>>>>>> pkt.
>>>>>>
>>>>>> $ ovs-ofctl add-flow br0 "in_port=afxdp-p0, \
>>>>>>
>>>>>> actions=set_field:14->in_port,set_field:a0:36:9f:33:b1:40->dl_src,afxdp-p0"
>>>>>>
>>>>>> Note: I have to set e_RTE_METER_GREEN to 0 to pass compile.
>>>>>
>>>>> You may use 'dpdk-latest' OVS branch. The proper fix exists there.
>>>>> In general, patches that intended to work only with latest DPDK that
>>>>> not yet supported on master should be made on top of 'dpdk-latest'
>>>>> branch and have [PATCH dpdk-latest] subject prefix.
>>>>
>>>> Got it, thank you.
>>>> William
>>>>
>>>>>
>>>>> Best regards, Ilya Maximets.
>>>
>>>
>
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
