Thank you so much Sean, It helped a lot and that I have VMs active running :) That's so kind of you to spend your time. It would be nice if this apparmor thing is mentioned somewhere in the documentation, will help newbies like me.
@Hui - Thank you for the info and the config file :) Thanks, Prathyusha On Wed, Nov 25, 2015 at 7:28 AM, Hui Xiang <hui.xi...@canonical.com> wrote: > FYI, bug 1513367 is opened for this apparmor problem to track when > booting vms failed with ovs-dpdk enabled. > > https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1513367 > > On Tue, Nov 24, 2015 at 9:49 PM, Mooney, Sean K <sean.k.moo...@intel.com> > wrote: > >> Out of interest >> >> Have you removed apparmor or placed all Libvirt apparmor profies into >> complain mode? >> >> >> >> If not you will get permission denied errors. >> >> >> >> You can confirm by checking dmesg to see if you have any permission >> denied messages from apparmor >> >> Or run aa-status and see if the the Libvirt profie is in enforce/complain >> mode. >> >> >> >> The /tmp/qemu.orig file is just a file we write the original qemu >> command to for debugging. It is not needed >> >> But all uses should be able to read/write to /tmp. >> >> >> >> We wrap the qemu/kvm binary with a script that on Ubuntu can be found >> here /usr/bin/kvm >> >> >> >> If you comment out echo "qemu ${args[@]}" > /tmp/qemu.orig in this script >> it will silence that warning. >> >> >> >> >> https://github.com/openstack/networking-ovs-dpdk/blob/master/devstack/libs/ovs-dpdk#L104 >> >> >> >> I may remove this from our wrapper script as we most never use it for >> debugging anymore however in the past it was >> >> Useful to compare the original qemu command line and the update qemu >> command line. >> >> >> >> I don’t know if I have mentioned this before but we also have a Ubuntu >> version of our getting start guide that should merge shortly >> >> >> >> https://review.openstack.org/#/c/243190/6/doc/source/getstarted/ubuntu.rst >> >> >> >> Regards >> >> Sean. >> >> >> >> *From:* Prathyusha Guduri [mailto:prathyushaconne...@gmail.com] >> *Sent:* Tuesday, November 24, 2015 12:42 PM >> *To:* Mooney, Sean K >> *Cc:* OpenStack Development Mailing List (not for usage questions) >> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk] >> >> >> >> Hi All, >> >> I also found another error while launching an instance. >> >> libvirtError: internal error: process exited while connecting to monitor: >> /usr/bin/kvm-spice: line 42: /tmp/qemu.orig: Permission denied >> >> I dont want to change any permissions manually and again face the >> dependency issues. So kindly help >> >> Thanks, >> >> Prathyusha >> >> >> >> >> >> >> >> On Tue, Nov 24, 2015 at 4:02 PM, Prathyusha Guduri < >> prathyushaconne...@gmail.com> wrote: >> >> Hi Sean, >> >> Thanks for you kind help. >> >> I did the following. >> >> # apt-get install ubuntu-cloud-keyring >> # echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu >> <http://www.google.com/url?q=http%3A%2F%2Fubuntu-cloud.archive.canonical.com%2Fubuntu&sa=D&sntz=1&usg=AFQjCNGlCfoplf1zSbILNxCSVK23zRxa2A>" >> \ >> "trusty-updates/kilo main" > >> /etc/apt/sources.list.d/cloudarchive-kilo.list >> # apt-get update && apt-get dist-upgrade >> >> and then uninstalled the libvirt and qemu that were installed manually >> and then ran stack.sh after cleaning and unstacking. >> >> Now fortunately libvirt and qemu satisfy minimum requirements. >> >> $ virsh --version >> 1.2.12 >> >> $ kvm --version >> /usr/bin/kvm: line 42: /tmp/qemu.orig: Permission denied >> QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.3~cloud0), >> Copyright (c) 2003-2008 Fabrice Bellard >> >> Am using an ubuntu 14.04 system >> $ uname -a >> Linux ubuntu-Precision-Tower-5810 3.13.0-24-generic #46-Ubuntu SMP Thu >> Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux >> >> After stack.sh which was successful, tried creating a new instance - >> which gave an ERROR again. >> >> $ nova list >> >> +--------------------------------------+----------------+--------+------------+-------------+-------------------------------------------------------+ >> | ID | Name | Status | Task >> State | Power State | Networks >> | >> >> +--------------------------------------+----------------+--------+------------+-------------+-------------------------------------------------------+ >> | 31a7e160-d04c-4216-91cf-30ce86c2b1fa | demo-instance1 | ERROR | >> - | NOSTATE | private=10.0.0.3, >> fd34:f4c5:412:0:f816:3eff:fea4:b9fe | >> >> $ sudo service ovs-dpdk status >> sourcing config >> ovs alive >> VHOST_CONFIG: bind to /var/run/openvswitch/vhufb8052e5-d3 >> 2015-11-24T10:23:25Z|00126|dpdk|INFO|Socket >> /var/run/openvswitch/vhufb8052e5-d3 created for vhost-user port >> vhufb8052e5-d3 >> 2015-11-24T10:23:25Z|00004|dpif_netdev(pmd18)|INFO|Core 2 processing port >> 'vhufb8052e5-d3' >> 2015-11-24T10:23:25Z|00002|dpif_netdev(pmd19)|INFO|Core 8 processing port >> 'dpdk0' >> 2015-11-24T10:23:25Z|00127|bridge|INFO|bridge br-int: added interface >> vhufb8052e5-d3 on port 6 >> 2015-11-24T10:23:25Z|00005|dpif_netdev(pmd18)|INFO|Core 2 processing port >> 'dpdk0' >> 2015-11-24T10:23:26Z|00128|connmgr|INFO|br-int<->unix: 1 flow_mods in the >> last 0 s (1 deletes) >> 2015-11-24T10:23:26Z|00129|ofp_util|INFO|normalization changed ofp_match, >> details: >> 2015-11-24T10:23:26Z|00130|ofp_util|INFO| pre: >> in_port=5,nw_proto=58,tp_src=136 >> 2015-11-24T10:23:26Z|00131|ofp_util|INFO|post: in_port=5 >> 2015-11-24T10:23:26Z|00132|connmgr|INFO|br-int<->unix: 1 flow_mods in the >> last 0 s (1 deletes) >> 2015-11-24T10:23:26Z|00133|connmgr|INFO|br-int<->unix: 1 flow_mods in the >> last 0 s (1 deletes) >> 2015-11-24T10:23:29Z|00134|bridge|WARN|could not open network device >> vhufb8052e5-d3 (No such device) >> VHOST_CONFIG: socket created, fd:52 >> VHOST_CONFIG: bind to /var/run/openvswitch/vhufb8052e5-d3 >> 2015-11-24T10:23:29Z|00135|dpdk|INFO|Socket >> /var/run/openvswitch/vhufb8052e5-d3 created for vhost-user port >> vhufb8052e5-d3 >> 2015-11-24T10:23:29Z|00006|dpif_netdev(pmd18)|INFO|Core 2 processing port >> 'vhufb8052e5-d3' >> 2015-11-24T10:23:29Z|00003|dpif_netdev(pmd19)|INFO|Core 8 processing port >> 'dpdk0' >> 2015-11-24T10:23:29Z|00136|bridge|INFO|bridge br-int: added interface >> vhufb8052e5-d3 on port 7 >> 2015-11-24T10:23:30Z|00007|dpif_netdev(pmd18)|INFO|Core 2 processing port >> 'dpdk0' >> 0 >> >> I understand that ovs-dpdk is running. The error log of n-cpu.log is >> >> 2015-11-24 15:47:27.957 ^[[01;31mERROR nova.compute.manager >> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone >> None^[[01;31m] ^[[01;35m^[[01;31mNo compute node record for host >> ubuntu-Precision-Tower-5810^[[00m >> 2015-11-24 15:47:27.964 ^[[01;33mWARNING nova.compute.monitors >> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone >> None^[[01;33m] ^[[01;35m^[[01;33mExcluding nova.compute.monitors.cpu >> monitor virt_driver. Not in the list of enabled monitors >> (CONF.compute_monitors).^[[00m >> 2015-11-24 15:47:27.964 ^[[00;36mINFO nova.compute.resource_tracker >> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone >> None^[[00;36m] ^[[01;35m^[[00;36mAuditing locally available compute >> resources for node ubuntu-Precision-Tower-5810^[[00m >> 2015-11-24 15:47:28.087 ^[[00;32mDEBUG nova.compute.resource_tracker >> [^[[01;36mreq-24fc3f16-ccd5-4e2d-b583-60ade23bc1ed ^[[00;36mNone >> None^[[00;32m] ^[[01;35m^[[00;32mHypervisor: free VCPUs: 9^[[00m >> ^[[00;33mfrom (pid=7880) _report_hypervisor_resource_view >> /opt/stack/nova/nova/compute/resource_tracker.py:588^[[00m >> >> I dont understand where am I missing out now. >> >> >> >> >> >> >> >> On Mon, Nov 23, 2015 at 7:50 PM, Mooney, Sean K <sean.k.moo...@intel.com> >> wrote: >> >> Hi >> >> qemu version was 2.0.0 does not support mapping hugepage as shared. >> as a result the dpdk implementation of vhost-user cannot function with >> this version. >> Similarly libvirt 1.2.2 has no knowledge of vhost-user. >> >> If you are on fedora 21 then the virt-preview repo packages the required >> Libvirt and qemu. >> >> On Ubuntu the kilo cloud archive also packages version that meet the >> minimum versions. >> >> Looking at the log I would agree that this error is probably related to >> manually installing Libvirt and >> not adding the appropriate libvirtd configuration options. >> >> Devstack does not have native support for installing a vhost-user >> compatible Libvirt or qemu. >> when using our networking-ovs-dpdk plugin we ask you to enable the >> virt-prevew or cloud archive >> Before stacking so that when devestack install Libvirt/qemu it get >> compatible versions. >> for centos as we have been unable to find an equivalent we explicitly >> install Libvirt/qemu. >> >> In general we don’t provide support for installing the required version >> Libvirt/qemu in >> the networking-ovs-dpdk devstack plugin. >> >> What os are you currently using? ubuntu? >> in your case enabling the kilo cloud archive, uinstalling Libvirt/qemu >> and then restacking >> should provide the appropriate packages. >> >> In liberty we use that standard neutron openvswich agent binary and set >> the agent type to DPDK OVS Agent to enable >> the ovsdpdk ml2 driver to manage the node. In mitaka the standard ovs >> neutron agent and standard openvswitch ml2 drivers will be used instead. >> >> >> >> To confirm if ovs-dpdk is running you can use the following command >> >> sudo service ovs-dpdk status >> >> this will check if ovs-dpdk is running by reading the pid files and >> checking if the ovsdb and ovs-vswitchd >> process are running and tail the last 20ish lines of the vswitchd log >> file. >> >> note this is not a systemd service so the systemd equivalent command will >> not work. >> >> >> >> Alternative you can user ps and grep >> >> ps aux | grep ovs >> >> >> >> regards >> >> sean. >> >> >> >> >> >> >> >> *From:* Prathyusha Guduri [mailto:prathyushaconne...@gmail.com] >> *Sent:* Monday, November 23, 2015 9:16 AM >> *To:* OpenStack Development Mailing List (not for usage questions) >> *Subject:* Re: [openstack-dev] [networking-ovs-dpdk] >> >> >> >> Hi Sean, >> >> Thanks for your help before. It would be great if you look into another >> issue too. >> >> Am able to run stack.sh successfully and all services are up. But, >> >> libvirt version was 1.2.2 and qemu version was 2.0.0 >> >> To satisfy the minimum requirement of qemu- version >=2.1 and >> libvirt-version >= 1.2.10 >> >> I manually installed qemu and libvirt from respective sources. >> >> Now >> $ kvm --version >> /usr/bin/kvm: line 42: /tmp/qemu.orig: Permission denied >> QEMU emulator version 2.1.3, Copyright (c) 2003-2008 Fabrice Bellard >> >> $ virsh --version >> 1.2.10 >> >> So basic requirement is satisfied. >> >> Before creating an instance ran the below command, >> $ nova flavor-key m1.tiny set "hw:mem_page_size=large" >> >> Now created an instance >> $ nova boot --flavor m1.tiny --image cirros-0.3.4-x86_64-uec --nic >> net-id=445e2dc5-221b-48ea-aea4-d04dee12fc7f --security-group default >> demo-instance1 >> >> It gives the ERROR : >> >> 2015-11-23 13:19:59.654 ERROR nova.virt.libvirt.host >> [req-2d9d060d-1934-4e9e-af1c-010e177bea11 None None] Connection to >> libvirt failed: error from service: CheckAuthorization: Did not receive >> a reply. Possible causes include: the remote application did not send a >> reply, the message bus security policy blocked the reply, the reply timeout >> expired, or the network connection was broken. >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host Traceback (most >> recent call last): >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File >> "/opt/stack/nova/nova/virt/libvirt/host.py", line 527, in get_connection >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host conn = >> self._get_connection() >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File >> "/opt/stack/nova/nova/virt/libvirt/host.py", line 514, in _get_connection >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host wrapped_conn = >> self._get_new_connection() >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File >> "/opt/stack/nova/nova/virt/libvirt/host.py", line 466, in >> _get_new_connection >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host wrapped_conn = >> self._connect(self._uri, self._read_only) >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File >> "/opt/stack/nova/nova/virt/libvirt/host.py", line 320, in _connect >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host >> libvirt.openAuth, uri, auth, flags) >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File >> "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in >> proxy_call >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host rv = execute(f, >> *args, **kwargs) >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File >> "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in >> execute >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host six.reraise(c, >> e, tb) >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File >> "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in >> tworker >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host rv = meth(*args, >> **kwargs) >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host File >> "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 105, in openAuth >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host if ret is >> None:raise libvirtError('virConnectOpenAuth() failed') >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host libvirtError: error >> from service: CheckAuthorization: Did not receive a reply. Possible causes >> include: the remote application did not send a reply, the message bus >> security policy blocked the reply, the reply timeout expired, or the >> network connection was broken. >> 2015-11-23 13:19:59.654 TRACE nova.virt.libvirt.host >> Traceback (most recent call last): >> >> I suspect this is because, I manually installed libvirt and qemu. My >> doubt is why devstack is not installing a correct version when it is >> supposed to. why a version less than min requirement is being installed??? >> >> Now that because am installing manually, there might be a problem with >> groups - devstack creates some group and installs but manual installation >> doesn't bother about that groups. >> Can you please suggest a way on how do avoid that??? >> >> >> >> Also, I just want to make sure that the agent running is >> neutron-openvswitch only. No ovsdpdk agent running. >> >> $ ps -Al | grep neutron >> 0 S 1000 8882 8859 3 80 0 - 49946 ep_pol pts/34 00:02:24 >> neutron-openvsw >> >> But >> >> $ neutron agent-list >> 3385a430-5738-43cb-b853-059add5ab602 | DPDK OVS Agent | >> ubuntu-Precision-Tower-5810 | :-) | True | >> neutron-openvswitch-agent >> >> So this implies that dpdk agent is running right??? I remember reading in >> launchpad bugs that ovsdpdk agent is removed and that now openvswitch takes >> care of everything. Just wanted to confirm that my setup has ovs-dpdk >> running. >> >> Regards, >> >> Prathyusha >> >> >> >> >> >> On Wed, Nov 18, 2015 at 7:23 PM, James Page <james.p...@ubuntu.com> >> wrote: >> >> Hi Sean >> >> >> >> On Wed, Nov 18, 2015 at 12:30 PM, Mooney, Sean K <sean.k.moo...@intel.com> >> wrote: >> >> Hi james >> >> Yes we are planning on testing the packaged release to see if it is >> compatible with our ml2 driver and the >> >> Changes we are submitting upstream. If it is we will add a use binary >> flag to our devstack plugin to skip the >> >> Compilation step and use that instead on 15.10 or 14.04 >> cloud-archive:liberty >> >> >> >> Excellent. >> >> >> >> As part of your packaging did ye fix pciutils to correctly report the >> unused drivers when an interface is bound >> >> The dpdk driver? Also does it support both igb_uio and/or vfio-pci >> drivers for dpdk interface? >> >> >> >> Re pcituils, we've not done any work in that area - can you give an >> example of what you would expect? >> >> >> >> The dpdk package supports both driver types in /etc/dpdk/interfaces - >> when you declare an adapter for use, you get to specify the module you want >> to use as well; we're relying the in-tree kernel drivers (uio-pci-generic >> and vfio-pci) right now. >> >> >> >> >> >> Anyway yes I hope to check it out and seeing what ye have done. When >> ovs-dpdk starts getting packaged in more operating systems >> >> We will probably swap our default to the binary install though we will >> keep the source install option as it allows us to work on new features >> >> Before they are packaged and to have better performance. >> >> >> >> That sounds sensible; re 'better performance' - yeah we do have to >> baseline the optimizations at compile time right now (ssse3 only right now) >> , but I really hope that does change so that we can move to a runtime CPU >> feature detection model, allowing the best possible performance through the >> packages we have in Ubuntu (or any other distribution for that matter). >> >> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> >> >> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > -- > Best Regards. > Hui. > > OpenStack Engineer > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev