On 07.03.2019 13:29, ppnaik wrote:
> Hi Ilya,
>
> Thanks for your suggestion.
> We tried pinning the PMD threads to core. But still the issue exists that
> only single queue is working instead of multiqueue.
> We tried using 'dpdkvhostuserclient' instead of 'dpdkvhostuser' but we are
> getting the following error and the VM is not able to find the interface.
Hi.
Thanks for logs. See comments inline.
>
> ovs-vswitchd log
>
> 2019-03-07T07:42:40.517Z|00212|dpdk|INFO|VHOST_CONFIG: vhost-user client:
> socket created, fd: 1091
> 2019-03-07T07:42:40.517Z|00213|netdev_dpdk|INFO|vHost User device
> 'dpdkvhostuser0' created in 'client' mode, using client socket
> '/tmp/dpdkvhostclient0'
> 2019-03-07T07:42:40.517Z|00214|dpdk|WARN|VHOST_CONFIG: failed to connect to
> /tmp/dpdkvhostclient0: No such file or directory
> 2019-03-07T07:42:40.517Z|00215|dpdk|INFO|VHOST_CONFIG: /tmp/dpdkvhostclient0:
> reconnecting...
> 2019-03-07T07:42:40.517Z|00216|dpif_netdev|WARN|There's no available
> (non-isolated) pmd thread on numa node 1. Queue 0 on port 'dpdk-p0' will be
> assigned to the pmd on core 1 (numa node 0). Expect reduced performance.
>
> The output of the following commands now is:
>
>
> ovs-appctl dpif-netdev/pmd-rxq-show
>
> pmd thread numa_id 0 core_id 1:
> isolated : false
> port: dpdk-p0 queue-id: 0 pmd usage: 0 %
> pmd thread numa_id 0 core_id 2:
> isolated : false
> port: dpdkvhostuser0 queue-id: 0 pmd usage: 0 %
>
>
> ovs-vsctl show:
>
> 356b0bfb-979c-408a-be1e-4567d52ef57f
> Bridge "br0"
> Port "dpdk-p0"
> Interface "dpdk-p0"
> type: dpdk
> options: {dpdk-devargs="0000:81:00.1,n_rxq=2"}
You configured 'n_rxq' as part of 'dpdk-devargs'. This should be separate
config to take effect:
ovs-vsctl set Interface dpdk-p0 type=dpdk options:n_rxq=2
options:dpdk-devargs="0000:81:00.1"
> Port "br0"
> Interface "br0"
> type: internal
> Port "dpdkvhostuser0"
> Interface "dpdkvhostuser0"
> type: dpdkvhostuserclient
> options: {vhost-server-path="/tmp/dpdkvhostuser0"}
>
> ovs-vswitchd --version
> ovs-vswitchd (Open vSwitch) 2.11.90
> DPDK 18.11.0
>
> qemu-system-x86_64 --version
> QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.34), Copyright
> (c) 2003-2008 Fabrice Bellard
Oh. Unfortunately vhost-user-client supported in qemu only since 2.7 version.
You need to switch back to usual vhost-user ports or update the qemu.
>
> Can you please help us with what we are missing?
>
> Thanks,
> Priyanka
>
>
> On 2019-03-06 18:13, Ilya Maximets wrote:
>> On 06.03.2019 15:18, ppnaik wrote:
>>> Thanks for the response Ilya.
>>> We could get this setup working now.
>>>
>>> However, we could not get it working when we want to give two queues to the
>>> VM interface.
>>>
>>> We added the queue option when creating the interface on OVS.
>>
>> There is no 'n_rxq'/'n_txq' options for vhost interfaces. Number of queues
>> obtained from the virtio device when QEMU connects. You could have this
>> in ovsdb, but it doesn't affect anything.
>> OTOH, physical ports has 'n_rxq' configurable.
>>
>>> We also enabled multiqueue in VM XML and gave the interface and set the
>>> vectors too.
>>>
>>> ethtool inside the VM shows:
>>>
>>> ethtool -l ens3
>>> Channel parameters for ens3:
>>> Pre-set maximums:
>>> RX: 0
>>> TX: 0
>>> Other: 0
>>> Combined: 2
>>> Current hardware settings:
>>> RX: 0
>>> TX: 0
>>> Other: 0
>>> Combined: 2
>>>
>>> However, a DPDK application inside the VM is not able to get packets from
>>> both queues. It still works with one queue.
>>
>> I assume that you have only one PMD thread that polls both physical
>> and virtual ports. Each PMD thread uses only one Tx queue per port.
>> So, if you want to utilize more queues in virtual interface, you need to
>> create more PMD threads using pmd-cpu-mask.
>> For example, with pmd-cpu-mask=0x30 there will be 2 PMD threads. One on
>> CPU #5 and another on CPU #6. Thread on core #5 will send packets to
>> let's say Tx queue #0, Thread on core #6 will send packets to Tx queue #1.
>> These packets will appear on Rx queue #0 and Rx queue #1 accordingly
>> inside VM.
>> But you need to be sure that both threads has packets to send, i.e. they
>> both polls some rx queues of other ports. For exmaple, you may configure
>> 2 queues on physical port and assign them to different PMD threads (this
>> should be done automatically). You may check the distribution of rx queues
>> between threads by 'ovs-appctl dpif-netdev/pmd-rxq-show'.
>> In this case, packets that appears on different rx queues of hw port will
>> go to different queues of vhost-user port.
>>
>>>
>>> Please help us resolve this issue.
>>>
>>> Thanks,
>>> Priyanka
>>>
>>>
>>> On 2019-03-06 16:48, Ilya Maximets wrote:
>>>> Hi.
>>>>
>>>> At first you need to look at ovs-vswitchd.log and the log of qemu.
>>>> There might be some errors.
>>>>
>>>> Some thoughts inline.
>>>>
>>>> Best regards, Ilya Maximets.
>>>>
>>>>> Hi All,
>>>>>
>>>>> Our setup is as follows:
>>>>>
>>>>> We have two servers which are connected peer to peer over 40G
>>>>> interfaces.
>>>>>
>>>>> On one server we have setup OVS and added the physical 40G interface as
>>>>> a DPDK interface to the ovs bridge.
>>>>>
>>>>> We created another dpdkvhostuser interface for the VM. We added this
>>>>> interface to the VM (by editing the XML). We are able to see this
>>>>> interface inside the VM and have configure IP to the interface.
>>>>>
>>>>> We want to communicate between the other server and VM inside this
>>>>> server through the OVS interface created for the VM.
>>>>>
>>>>> The steps we followed (on the server with OVS) are:
>>>>>
>>>>> modprobe uio
>>>>
>>>> IMHO, it's better to use vfio-pci. But it's up to you.
>>>>
>>>>>
>>>>> cd /usr/src/dpdk-18.11/x86_64-native-linuxapp-gcc/kmod/
>>>>>
>>>>> insmod igb_uio.ko
>>>>>
>>>>> cd /usr/src/dpdk-18.11/usertools/
>>>>> ./dpdk-devbind.py --bind=igb_uio 0000:81:00.1
>>>>>
>>>>> export PATH=$PATH:/usr/local/share/openvswitch/scripts
>>>>> export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
>>>>>
>>>>> ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock
>>>>> --remote=db:Open_vSwitch,Open_vSwitch,manager_options
>>>>> --private-key=db:Open_vSwitch,SSL,private_key
>>>>> --certificate=db:Open_vSwitch,SSL,certificate
>>>>> --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach
>>>>> --log-file
>>>>>
>>>>> ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
>>>>> ovs-ctl --no-ovsdb-server --db-sock="$DB_SOCK" start
>>>>>
>>>>> ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
>>>>> ovs-vsctl add-port br0 dpdk-p0 -- set Interface dpdk-p0 type=dpdk
>>>>> options:dpdk-devargs=0000:81:00.1
>>>>> ovs-vsctl add-port br0 dpdkvhostuser0 -- set Interface
>>>>> dpdkvhostuser0 type=dpdkvhostuser ofport_request=3
>>>>
>>>> Consider using 'dpdkvhostuserclient' a.k.a. 'vhost-user-client' ports
>>>> instead
>>>> because server mode 'dpdkvhostuser' ports are deprecated in OVS.
>>>>
>>>>>
>>>>> ovs-ofctl add-flow br0 in_port=1,action=output:3
>>>>> ovs-ofctl add-flow br0 in_port=3,action=output:1
>>>>>
>>>>> echo 'vm.nr_hugepages=2048' > /etc/sysctl.d/hugepages.conf
>>>>
>>>> Here you're allocating 2048 pages of 2MB. This is not enough
>>>> for your setup. You're trying to allocate 4096 MB for qemu memory
>>>> backing + OVS will need some hugepage memory for the mempools and stuff.
>>>> It'll be at total:
>>>> 4096 + 1024 (default for OVS if you have only 1 NUMA node) MB,
>>>> i.e. you need at least 512 more pages.
>>>>
>>>> Do you need to reload sysctl for changes to be applied?
>>>>
>>>>> grep HugePages_ /proc/meminfo
>>>>
>>>> So, where is the output? If the output is empty, you have no pages
>>>> allocated.
>>>>
>>>>>
>>>>> edit VM XML to add this interface:
>>>>
>>>> If you're starting new VM with updated XML than I'll suggest using
>>>> better libvirt
>>>> syntax. i.e. it's better to use sections like "memoryBacking", "interface"
>>>> instead of manual attaching of cmdline arguments.
>>>> See
>>>> http://docs.openvswitch.org/en/latest/topics/dpdk/vhost-user/#sample-xml
>>>>
>>>>>
>>>>> first line:
>>>>> <domain type='kvm'
>>>>> xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
>>>>>
>>>>> add before </domain> tag:
>>>>>
>>>>> <qemu:commandline>
>>>>> <qemu:arg value='-chardev'/>
>>>>> <qemu:arg
>>>>>
>>>>>
>>>>> value='socket,id=char1,path=/usr/local/var/run/openvswitch/dpdkvhostuser0'/>
>>>>> <qemu:arg value='-netdev'/>
>>>>> <qemu:arg
>>>>>
>>>>> value='vhost-user,id=mynet1,chardev=char1,vhostforce=on,queues=1'/>
>>>>> <qemu:arg value='-device'/>
>>>>> <qemu:arg
>>>>>
>>>>>
>>>>> value='virtio-net-pci,mac=00:00:00:00:00:02,netdev=mynet1,mq=on,vectors=4'/>
>>>>> <qemu:arg value='-m'/>
>>>>> <qemu:arg value='4096'/>
>>>>> <qemu:arg value='-object'/>
>>>>> <qemu:arg
>>>>>
>>>>>
>>>>> value='memory-backend-file,id=mem1,size=4096M,mem-path=/dev/hugepages,share=on'/>
>>>>> <qemu:arg value='-mem-prealloc'/>
>>>>> <qemu:arg value='-numa'/>
>>>>> <qemu:arg value='node,memdev=mem1'/>
>>>>> </qemu:commandline>
>>>>>
>>>>> Please help us resolve this issue. I assumed ping would work between
>>>>> the other server and the VM. But it is not working in our case. Also,
>>>>> let us know if we are missing some setup step or if there is some
>>>>> misconfiguration. If ping would not work can you let us know a way to
>>>>> verify the connectivity?
>>>>>
>>>>> Thanks,
>>>>> Priyanka
>>>
>>>
>
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
