Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-25 Thread Jeffrey Zhang 
​another issue we(kolla gate) meet is libvirtd got segment fault.
Here is the log from host messages file. But i do not get other issue in
nova.conf expect can not connect libvirt.

Dec 25 13:40:54 localhost kernel: libvirtd[5720]: segfault at 8 ip
7fa70379b823 sp 7fa6faa061c0 error 4 in libvirt.so.0.2000.0[
7fa703648000+353000]​

On Fri, Dec 23, 2016 at 10:00 PM, Neil Jerram  wrote:

> On Fri, Dec 23, 2016 at 1:37 PM Steve Gordon  wrote:
>
>> - Original Message -
>> > From: "Neil Jerram" 
>> > To: "OpenStack Development Mailing List (not for usage questions)" <
>> openstack-dev@lists.openstack.org>
>> >
>> > I appreciate that even libvirt 2.0.0 will be ancient history by now, to
>> its
>> > developers, but I am seeing further issues that look associated with the
>> > recent CentOS 7 transition from libvirt 1.2.7 to libvirt 2.0.0, and
>> would
>> > appreciate any comments on them that people may have.  I believe these
>> > issues are independent of those that have already been discussed on
>> other
>> > threads.
>> >
>> > First, this traceback in nova-compute.log:
>> >
>> > Traceback (most recent call last):
>> >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
>> > 2156, in _build_resources
>> > yield resources
>> >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
>> > 2009, in _build_and_run_instance
>> > block_device_info=block_device_info)
>> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
>> line
>> > 2534, in spawn
>> > block_device_info=block_device_info)
>> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
>> line
>> > 4620, in _create_domain_and_network
>> > xml, pause=pause, power_on=power_on)
>> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
>> line
>> > 4550, in _create_domain
>> > guest.launch(pause=pause)
>> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
>> line
>> > 142, in launch
>> > self._encoded_xml, errors='ignore')
>> >   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
>> 195,
>> > in __exit__
>> > six.reraise(self.type_, self.value, self.tb)
>> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
>> line
>> > 137, in launch
>> > return self._domain.createWithFlags(flags)
>> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186,
>> in
>> > doit
>> > result = proxy_call(self._autowrap, f, *args, **kwargs)
>> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144,
>> in
>> > proxy_call
>> > rv = execute(f, *args, **kwargs)
>> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125,
>> in
>> > execute
>> > six.reraise(c, e, tb)
>> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83,
>> in
>> > tworker
>> > rv = meth(*args, **kwargs)
>> >   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in
>> > createWithFlags
>> > if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
>> failed',
>> > dom=self)
>> > libvirtError: Cannot find '' in path: No such file or directory
>> >
>> > which I believe is caused by the empty path attribute in this part of
>> the
>> > XML:
>> >
>> > 
>> >   
>> >   
>> >   
>> >   
>> >   
>> >   > > function='0x0'/>
>> > 
>> >
>> > which is in turn caused, I think, by
>> > https://github.com/openstack/nova/blob/master/nova/virt/
>> libvirt/designer.py#L61
>> >
>> > Is it plausible that libvirt 1.2.7 would have avoided trying to invoke a
>> > script with an empty path, whereas libvirt 2.0.0 does not?
>>
>> It seems someone filed this as https://bugs.launchpad.net/
>> nova/+bug/1649527 from a Nova POV. The Libvirt folks helped me narrow
>> this down to this commit being the first one that would have exhibited this
>> behavior:
>>
>> https://libvirt.org/git/?p=libvirt.git;a=commit;h=
>> 9c17d665fdc5f0ab74500a14c30627014c11b2c0
>>
>> Michal Privoznik provided some additional context:
>>
>> "Previously, libvirt merely just appended 'script=' onto qemu cmd line
>> according to what  contained letting qemu execute the
>> script. That was flawed from security POV (we don't want qemu to be
>> allowed to execute anything) thus libvirt is executing the script now.
>> However, we obviously forgot about this corner case."
>>
>> This actually apparently first appeared in v1.3.3-rc1~71, so it's not new
>> in Libvirt 2.0.0 *but*:
>>
>> * The Ubuntu-based gate is apparently running v1.3.1 at the moment.
>> * The RHEL 7.2 and aligned CentOS included v1.2.17.
>>
>> This is at least partially why this was not seen until recently,
>
>
> Thanks for pinning all of that down!
>
>
>> but it also seems like it might be specific to certain guest networking
>> setup.
>
>
> Yes, there are only a few networking backends that use  type=ethernet>, I believe; mine is

Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-23 Thread Neil Jerram 
On Fri, Dec 23, 2016 at 1:37 PM Steve Gordon  wrote:

> - Original Message -
> > From: "Neil Jerram" 
> > To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> >
> > I appreciate that even libvirt 2.0.0 will be ancient history by now, to
> its
> > developers, but I am seeing further issues that look associated with the
> > recent CentOS 7 transition from libvirt 1.2.7 to libvirt 2.0.0, and would
> > appreciate any comments on them that people may have.  I believe these
> > issues are independent of those that have already been discussed on other
> > threads.
> >
> > First, this traceback in nova-compute.log:
> >
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> > 2156, in _build_resources
> > yield resources
> >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> > 2009, in _build_and_run_instance
> > block_device_info=block_device_info)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> line
> > 2534, in spawn
> > block_device_info=block_device_info)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> line
> > 4620, in _create_domain_and_network
> > xml, pause=pause, power_on=power_on)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> line
> > 4550, in _create_domain
> > guest.launch(pause=pause)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
> line
> > 142, in launch
> > self._encoded_xml, errors='ignore')
> >   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 195,
> > in __exit__
> > six.reraise(self.type_, self.value, self.tb)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
> line
> > 137, in launch
> > return self._domain.createWithFlags(flags)
> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in
> > doit
> > result = proxy_call(self._autowrap, f, *args, **kwargs)
> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in
> > proxy_call
> > rv = execute(f, *args, **kwargs)
> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in
> > execute
> > six.reraise(c, e, tb)
> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in
> > tworker
> > rv = meth(*args, **kwargs)
> >   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in
> > createWithFlags
> > if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
> failed',
> > dom=self)
> > libvirtError: Cannot find '' in path: No such file or directory
> >
> > which I believe is caused by the empty path attribute in this part of the
> > XML:
> >
> > 
> >   
> >   
> >   
> >   
> >   
> >> function='0x0'/>
> > 
> >
> > which is in turn caused, I think, by
> >
> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61
> >
> > Is it plausible that libvirt 1.2.7 would have avoided trying to invoke a
> > script with an empty path, whereas libvirt 2.0.0 does not?
>
> It seems someone filed this as
> https://bugs.launchpad.net/nova/+bug/1649527 from a Nova POV. The Libvirt
> folks helped me narrow this down to this commit being the first one that
> would have exhibited this behavior:
>
>
> https://libvirt.org/git/?p=libvirt.git;a=commit;h=9c17d665fdc5f0ab74500a14c30627014c11b2c0
>
> Michal Privoznik provided some additional context:
>
> "Previously, libvirt merely just appended 'script=' onto qemu cmd line
> according to what  contained letting qemu execute the
> script. That was flawed from security POV (we don't want qemu to be
> allowed to execute anything) thus libvirt is executing the script now.
> However, we obviously forgot about this corner case."
>
> This actually apparently first appeared in v1.3.3-rc1~71, so it's not new
> in Libvirt 2.0.0 *but*:
>
> * The Ubuntu-based gate is apparently running v1.3.1 at the moment.
> * The RHEL 7.2 and aligned CentOS included v1.2.17.
>
> This is at least partially why this was not seen until recently,


Thanks for pinning all of that down!


> but it also seems like it might be specific to certain guest networking
> setup.


Yes, there are only a few networking backends that use , I believe; mine is networking-calico, which uses
VIF_TYPE_TAP; I think (from code reading) that the others are 'ivs',
'iovisor', 'midonet' and 'vrouter'.  (Although another thing to keep in
mind here is that os-vif should soon be taking over this area of code - so
any fixes in nova/virt/libvirt may also be needed in os-vif.)


You mentioned you have multiple interfaces attached to the guest, which VIF
> types and associated Neutron plugins are you using?
>

Note that the multiple interfaces issue is an additional one.  The empty
path issue causes all VIF_TYPE_TAP tests to fail, whether with just

Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-23 Thread Steve Gordon 
- Original Message -
> From: "Neil Jerram" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> 
> I appreciate that even libvirt 2.0.0 will be ancient history by now, to its
> developers, but I am seeing further issues that look associated with the
> recent CentOS 7 transition from libvirt 1.2.7 to libvirt 2.0.0, and would
> appreciate any comments on them that people may have.  I believe these
> issues are independent of those that have already been discussed on other
> threads.
> 
> First, this traceback in nova-compute.log:
> 
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 2156, in _build_resources
> yield resources
>   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 2009, in _build_and_run_instance
> block_device_info=block_device_info)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
> 2534, in spawn
> block_device_info=block_device_info)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
> 4620, in _create_domain_and_network
> xml, pause=pause, power_on=power_on)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
> 4550, in _create_domain
> guest.launch(pause=pause)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line
> 142, in launch
> self._encoded_xml, errors='ignore')
>   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195,
> in __exit__
> six.reraise(self.type_, self.value, self.tb)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line
> 137, in launch
> return self._domain.createWithFlags(flags)
>   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in
> doit
> result = proxy_call(self._autowrap, f, *args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in
> proxy_call
> rv = execute(f, *args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in
> execute
> six.reraise(c, e, tb)
>   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in
> tworker
> rv = meth(*args, **kwargs)
>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in
> createWithFlags
> if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',
> dom=self)
> libvirtError: Cannot find '' in path: No such file or directory
> 
> which I believe is caused by the empty path attribute in this part of the
> XML:
> 
> 
>   
>   
>   
>   
>   
>function='0x0'/>
> 
> 
> which is in turn caused, I think, by
> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61
> 
> Is it plausible that libvirt 1.2.7 would have avoided trying to invoke a
> script with an empty path, whereas libvirt 2.0.0 does not?

It seems someone filed this as https://bugs.launchpad.net/nova/+bug/1649527 
from a Nova POV. The Libvirt folks helped me narrow this down to this commit 
being the first one that would have exhibited this behavior:


https://libvirt.org/git/?p=libvirt.git;a=commit;h=9c17d665fdc5f0ab74500a14c30627014c11b2c0

Michal Privoznik provided some additional context:

"Previously, libvirt merely just appended 'script=' onto qemu cmd line
according to what  contained letting qemu execute the
script. That was flawed from security POV (we don't want qemu to be
allowed to execute anything) thus libvirt is executing the script now.
However, we obviously forgot about this corner case."

This actually apparently first appeared in v1.3.3-rc1~71, so it's not new in 
Libvirt 2.0.0 *but*:

* The Ubuntu-based gate is apparently running v1.3.1 at the moment.
* The RHEL 7.2 and aligned CentOS included v1.2.17.

This is at least partially why this was not seen until recently, but it also 
seems like it might be specific to certain guest networking setup. You 
mentioned you have multiple interfaces attached to the guest, which VIF types 
and associated Neutron plugins are you using?

Thanks,

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-21 Thread Steve Gordon 


- Original Message -
> From: "Neil Jerram" <n...@tigera.io>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Cc: "Steve Gordon" <sgor...@redhat.com>
> Sent: Monday, December 19, 2016 5:53:02 AM
> Subject: Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 
> (CentOS 7.3)
> 
> On Fri, Dec 16, 2016 at 4:16 PM Neil Jerram <n...@tigera.io> wrote:
> 
> > On Fri, Dec 16, 2016 at 4:08 PM Steve Gordon <sgor...@redhat.com> wrote:
> >
> > > If you haven't already I would suggest grabbing qemu-kvm-ev from the
> > CentOS
> > > > Virt SIG repos:
> > > >
> > > > http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/
> > > >
> > > > You can enable the repository using this release RPM:
> > > >
> > > >
> > > >
> > http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/centos-release-qemu-ev-1.0-1.el7.noarch.rpm
> > > >
> > > >
> > > Would you expect that to help with virt_type = qemu (as well as with
> > > virt_type = kvm, which I assume is the more common setting)?  If so I'll
> > be
> > > very excited to try this!
> >
> > For this particular flag issue I am not 100% sure yet as I'm still
> > checking with some of the qemu folks, but I think it would still be worth a
> > try.
> >
> >
> > Well I have at least one booting instance now, and there is no mention of
> > 'tsc_adjust not found' in the instance's log.
> >
> > So looking more promising - thanks!
> >
> >
> 
> Most of my testing on CentOS 7.3 is now working again, but I am reliably
> seeing issues in 3 cases connected with
> - multiple interfaces into a VM
> - a VM being rebooted.
> 
> Should I report those in more detail here, or is there some better place
> such as a CentOS- or libvirt-focussed list?

Hi Neil,

Sorry for the delay, the places I would suggest to get in touch with Libvirt 
folks are:

* Mailing list: https://www.redhat.com/mailman/listinfo/libvirt-users
* IRC: #virt on irc.oftc.net (often quite slow but stick around and someone 
will try help, lots of lurkers)
* Bugzilla: 
https://bugzilla.redhat.com/enter_bug.cgi?product=Virtualization%20Tools

Hope that helps,

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-19 Thread Neil Jerram 
On Fri, Dec 16, 2016 at 4:16 PM Neil Jerram  wrote:

> On Fri, Dec 16, 2016 at 4:08 PM Steve Gordon  wrote:
>
> > If you haven't already I would suggest grabbing qemu-kvm-ev from the
> CentOS
> > > Virt SIG repos:
> > >
> > > http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/
> > >
> > > You can enable the repository using this release RPM:
> > >
> > >
> > >
> http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/centos-release-qemu-ev-1.0-1.el7.noarch.rpm
> > >
> > >
> > Would you expect that to help with virt_type = qemu (as well as with
> > virt_type = kvm, which I assume is the more common setting)?  If so I'll
> be
> > very excited to try this!
>
> For this particular flag issue I am not 100% sure yet as I'm still
> checking with some of the qemu folks, but I think it would still be worth a
> try.
>
>
> Well I have at least one booting instance now, and there is no mention of
> 'tsc_adjust not found' in the instance's log.
>
> So looking more promising - thanks!
>
>

Most of my testing on CentOS 7.3 is now working again, but I am reliably
seeing issues in 3 cases connected with
- multiple interfaces into a VM
- a VM being rebooted.

Should I report those in more detail here, or is there some better place
such as a CentOS- or libvirt-focussed list?

Thanks,
 Neil
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-16 Thread Neil Jerram 
On Fri, Dec 16, 2016 at 4:08 PM Steve Gordon <sgor...@redhat.com> wrote:

> - Original Message -
> > From: "Neil Jerram" <n...@tigera.io>
> > To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> > Sent: Friday, December 16, 2016 10:53:02 AM
> > Subject: Re: [openstack-dev] Issues with libvirt transition 1.2.7 ->
> 2.0.0 (CentOS 7.3)
> >
> > On Fri, Dec 16, 2016 at 3:35 PM Steve Gordon <sgor...@redhat.com> wrote:
> >
> > >
> > >
> > > - Original Message -
> > > > From: "Neil Jerram" <n...@tigera.io>
> > > > To: "OpenStack Development Mailing List (not for usage questions)" <
> > > openstack-dev@lists.openstack.org>
> > > > Sent: Friday, December 16, 2016 6:40:57 AM
> > > > Subject: [openstack-dev] Issues with libvirt transition 1.2.7 ->
> 2.0.0
> > >   (CentOS 7.3)
> > > >
> > > > I appreciate that even libvirt 2.0.0 will be ancient history by now,
> to
> > > its
> > > > developers, but I am seeing further issues that look associated with
> the
> > > > recent CentOS 7 transition from libvirt 1.2.7 to libvirt 2.0.0, and
> would
> > > > appreciate any comments on them that people may have.  I believe
> these
> > > > issues are independent of those that have already been discussed on
> other
> > > > threads.
> > > >
> > > > First, this traceback in nova-compute.log:
> > > >
> > > > Traceback (most recent call last):
> > > >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py",
> line
> > > > 2156, in _build_resources
> > > > yield resources
> > > >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py",
> line
> > > > 2009, in _build_and_run_instance
> > > > block_device_info=block_device_info)
> > > >   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> > > line
> > > > 2534, in spawn
> > > > block_device_info=block_device_info)
> > > >   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> > > line
> > > > 4620, in _create_domain_and_network
> > > > xml, pause=pause, power_on=power_on)
> > > >   File
> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> > > line
> > > > 4550, in _create_domain
> > > > guest.launch(pause=pause)
> > > >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
> > > line
> > > > 142, in launch
> > > > self._encoded_xml, errors='ignore')
> > > >   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py",
> line
> > > 195,
> > > > in __exit__
> > > > six.reraise(self.type_, self.value, self.tb)
> > > >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
> > > line
> > > > 137, in launch
> > > > return self._domain.createWithFlags(flags)
> > > >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line
> 186, in
> > > > doit
> > > > result = proxy_call(self._autowrap, f, *args, **kwargs)
> > > >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line
> 144, in
> > > > proxy_call
> > > > rv = execute(f, *args, **kwargs)
> > > >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line
> 125, in
> > > > execute
> > > > six.reraise(c, e, tb)
> > > >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line
> 83, in
> > > > tworker
> > > > rv = meth(*args, **kwargs)
> > > >   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in
> > > > createWithFlags
> > > > if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
> > > failed',
> > > > dom=self)
> > > > libvirtError: Cannot find '' in path: No such file or directory
> > > >
> > > > which I believe is caused by the empty path attribute in this part
> of the
> > > > XML:
> > > >
> > > > 
> > > >   
> > > >   
> > > >   
> > > >   
&g

Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-16 Thread Steve Gordon 
- Original Message -
> From: "Neil Jerram" <n...@tigera.io>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> <openstack-dev@lists.openstack.org>
> Sent: Friday, December 16, 2016 10:53:02 AM
> Subject: Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 
> (CentOS 7.3)
> 
> On Fri, Dec 16, 2016 at 3:35 PM Steve Gordon <sgor...@redhat.com> wrote:
> 
> >
> >
> > - Original Message -
> > > From: "Neil Jerram" <n...@tigera.io>
> > > To: "OpenStack Development Mailing List (not for usage questions)" <
> > openstack-dev@lists.openstack.org>
> > > Sent: Friday, December 16, 2016 6:40:57 AM
> > > Subject: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0
> >   (CentOS 7.3)
> > >
> > > I appreciate that even libvirt 2.0.0 will be ancient history by now, to
> > its
> > > developers, but I am seeing further issues that look associated with the
> > > recent CentOS 7 transition from libvirt 1.2.7 to libvirt 2.0.0, and would
> > > appreciate any comments on them that people may have.  I believe these
> > > issues are independent of those that have already been discussed on other
> > > threads.
> > >
> > > First, this traceback in nova-compute.log:
> > >
> > > Traceback (most recent call last):
> > >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> > > 2156, in _build_resources
> > > yield resources
> > >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> > > 2009, in _build_and_run_instance
> > > block_device_info=block_device_info)
> > >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> > line
> > > 2534, in spawn
> > > block_device_info=block_device_info)
> > >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> > line
> > > 4620, in _create_domain_and_network
> > > xml, pause=pause, power_on=power_on)
> > >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> > line
> > > 4550, in _create_domain
> > > guest.launch(pause=pause)
> > >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
> > line
> > > 142, in launch
> > > self._encoded_xml, errors='ignore')
> > >   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> > 195,
> > > in __exit__
> > > six.reraise(self.type_, self.value, self.tb)
> > >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
> > line
> > > 137, in launch
> > > return self._domain.createWithFlags(flags)
> > >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in
> > > doit
> > > result = proxy_call(self._autowrap, f, *args, **kwargs)
> > >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in
> > > proxy_call
> > > rv = execute(f, *args, **kwargs)
> > >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in
> > > execute
> > > six.reraise(c, e, tb)
> > >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in
> > > tworker
> > > rv = meth(*args, **kwargs)
> > >   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in
> > > createWithFlags
> > > if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
> > failed',
> > > dom=self)
> > > libvirtError: Cannot find '' in path: No such file or directory
> > >
> > > which I believe is caused by the empty path attribute in this part of the
> > > XML:
> > >
> > > 
> > >   
> > >   
> > >   
> > >   
> > >   
> > >> > function='0x0'/>
> > > 
> > >
> > > which is in turn caused, I think, by
> > >
> > https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61
> > >
> > > Is it plausible that libvirt 1.2.7 would have avoided trying to invoke a
> > > script with an empty path, whereas libvirt 2.0.0 does not?
> > >
> > > Secondly - if I move past the problem above by changing
> > >
> > https://github.com/openstack/nova/blob/m

Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-16 Thread Neil Jerram 
On Fri, Dec 16, 2016 at 3:35 PM Steve Gordon  wrote:

>
>
> - Original Message -
> > From: "Neil Jerram" 
> > To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> > Sent: Friday, December 16, 2016 6:40:57 AM
> > Subject: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0
>   (CentOS 7.3)
> >
> > I appreciate that even libvirt 2.0.0 will be ancient history by now, to
> its
> > developers, but I am seeing further issues that look associated with the
> > recent CentOS 7 transition from libvirt 1.2.7 to libvirt 2.0.0, and would
> > appreciate any comments on them that people may have.  I believe these
> > issues are independent of those that have already been discussed on other
> > threads.
> >
> > First, this traceback in nova-compute.log:
> >
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> > 2156, in _build_resources
> > yield resources
> >   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> > 2009, in _build_and_run_instance
> > block_device_info=block_device_info)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> line
> > 2534, in spawn
> > block_device_info=block_device_info)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> line
> > 4620, in _create_domain_and_network
> > xml, pause=pause, power_on=power_on)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
> line
> > 4550, in _create_domain
> > guest.launch(pause=pause)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
> line
> > 142, in launch
> > self._encoded_xml, errors='ignore')
> >   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 195,
> > in __exit__
> > six.reraise(self.type_, self.value, self.tb)
> >   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py",
> line
> > 137, in launch
> > return self._domain.createWithFlags(flags)
> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in
> > doit
> > result = proxy_call(self._autowrap, f, *args, **kwargs)
> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in
> > proxy_call
> > rv = execute(f, *args, **kwargs)
> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in
> > execute
> > six.reraise(c, e, tb)
> >   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in
> > tworker
> > rv = meth(*args, **kwargs)
> >   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in
> > createWithFlags
> > if ret == -1: raise libvirtError ('virDomainCreateWithFlags()
> failed',
> > dom=self)
> > libvirtError: Cannot find '' in path: No such file or directory
> >
> > which I believe is caused by the empty path attribute in this part of the
> > XML:
> >
> > 
> >   
> >   
> >   
> >   
> >   
> >> function='0x0'/>
> > 
> >
> > which is in turn caused, I think, by
> >
> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61
> >
> > Is it plausible that libvirt 1.2.7 would have avoided trying to invoke a
> > script with an empty path, whereas libvirt 2.0.0 does not?
> >
> > Secondly - if I move past the problem above by changing
> >
> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61
> > to say 'conf.script = None' - I then find:
> > - no apparent error in nova-compute.log
> > - but my instances don't boot
> > - the following messages in the libvirt log:
> >
> > Domain id=4 is tainted: high-privileges
> > char device redirected to /dev/pts/1 (label charserial1)
> > CPU feature tsc_adjust not found
> >
> > I guess it's the last message that is the critical one here - can anyone
> > tell me more about it?
>
> Hi Neil,
>
> The second seems likely to be related to the addition of support for VMX
> TSC scaling. What version of qemu are you running?


Thanks for your reply, Steve.  I have these qemu packages installed:

[root@neil-fv-0-centos-liberty-compute-node01 qemu]# yum list installed |
grep qemu
ipxe-roms-qemu.noarch20160127-5.git6366fa7a.el7
libvirt-daemon-driver-qemu.x86_642.0.0-10.el7_3.2
@updates
qemu-img.x86_64  10:1.5.3-126.el7
@base
qemu-kvm.x86_64  10:1.5.3-126.el7
@base
qemu-kvm-common.x86_64   10:1.5.3-126.el7
@base

I should probably also mention that I'm using virt_type = qemu, because my
compute nodes are GCE instances.

[root@neil-fv-0-centos-liberty-compute-node01 qemu]# cat
/etc/nova/nova-compute.conf
[libvirt]
virt_type = qemu

If you haven't already I would suggest grabbing qemu-kvm-ev from the CentOS
> Virt SIG repos:
>
> http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/
>
> You can enable the repository using this release RPM:
>
>
>

Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-16 Thread Steve Gordon 


- Original Message -
> From: "Neil Jerram" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Friday, December 16, 2016 6:40:57 AM
> Subject: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0
> (CentOS 7.3)
> 
> I appreciate that even libvirt 2.0.0 will be ancient history by now, to its
> developers, but I am seeing further issues that look associated with the
> recent CentOS 7 transition from libvirt 1.2.7 to libvirt 2.0.0, and would
> appreciate any comments on them that people may have.  I believe these
> issues are independent of those that have already been discussed on other
> threads.
> 
> First, this traceback in nova-compute.log:
> 
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 2156, in _build_resources
> yield resources
>   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 2009, in _build_and_run_instance
> block_device_info=block_device_info)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
> 2534, in spawn
> block_device_info=block_device_info)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
> 4620, in _create_domain_and_network
> xml, pause=pause, power_on=power_on)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
> 4550, in _create_domain
> guest.launch(pause=pause)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line
> 142, in launch
> self._encoded_xml, errors='ignore')
>   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195,
> in __exit__
> six.reraise(self.type_, self.value, self.tb)
>   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line
> 137, in launch
> return self._domain.createWithFlags(flags)
>   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in
> doit
> result = proxy_call(self._autowrap, f, *args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in
> proxy_call
> rv = execute(f, *args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in
> execute
> six.reraise(c, e, tb)
>   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in
> tworker
> rv = meth(*args, **kwargs)
>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in
> createWithFlags
> if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',
> dom=self)
> libvirtError: Cannot find '' in path: No such file or directory
> 
> which I believe is caused by the empty path attribute in this part of the
> XML:
> 
> 
>   
>   
>   
>   
>   
>function='0x0'/>
> 
> 
> which is in turn caused, I think, by
> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61
> 
> Is it plausible that libvirt 1.2.7 would have avoided trying to invoke a
> script with an empty path, whereas libvirt 2.0.0 does not?
> 
> Secondly - if I move past the problem above by changing
> https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61
> to say 'conf.script = None' - I then find:
> - no apparent error in nova-compute.log
> - but my instances don't boot
> - the following messages in the libvirt log:
> 
> Domain id=4 is tainted: high-privileges
> char device redirected to /dev/pts/1 (label charserial1)
> CPU feature tsc_adjust not found
> 
> I guess it's the last message that is the critical one here - can anyone
> tell me more about it?

Hi Neil,

The second seems likely to be related to the addition of support for VMX TSC 
scaling. What version of qemu are you running? If you haven't already I would 
suggest grabbing qemu-kvm-ev from the CentOS Virt SIG repos:

http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/

You can enable the repository using this release RPM:


http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/centos-release-qemu-ev-1.0-1.el7.noarch.rpm

Thanks,

Steve

> Thanks,
>   Neil
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

-- 
Steve Gordon,
Principal Product Manager,
Red Hat OpenStack Platform

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Issues with libvirt transition 1.2.7 -> 2.0.0 (CentOS 7.3)

2016-12-16 Thread Neil Jerram 
Another thought/query about this: is the libvirt transition from 1.2.7 to
2.0.0 less than usually conservative for an RHEL/CentOS series, and if so,
is there a wider reason or move there that it would help to be aware of?
Also is there a way of continuing to use CentOS 7 as a platform but still
using libvirt 1.2.7?

Thanks - Neil


On Fri, Dec 16, 2016 at 11:40 AM Neil Jerram  wrote:

I appreciate that even libvirt 2.0.0 will be ancient history by now, to its
developers, but I am seeing further issues that look associated with the
recent CentOS 7 transition from libvirt 1.2.7 to libvirt 2.0.0, and would
appreciate any comments on them that people may have.  I believe these
issues are independent of those that have already been discussed on other
threads.

First, this traceback in nova-compute.log:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
2156, in _build_resources
yield resources
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
2009, in _build_and_run_instance
block_device_info=block_device_info)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
2534, in spawn
block_device_info=block_device_info)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
4620, in _create_domain_and_network
xml, pause=pause, power_on=power_on)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line
4550, in _create_domain
guest.launch(pause=pause)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line
142, in launch
self._encoded_xml, errors='ignore')
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195,
in __exit__
six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line
137, in launch
return self._domain.createWithFlags(flags)
  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in
doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in
proxy_call
rv = execute(f, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in
execute
six.reraise(c, e, tb)
  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in
tworker
rv = meth(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in
createWithFlags
if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',
dom=self)
libvirtError: Cannot find '' in path: No such file or directory

which I believe is caused by the empty path attribute in this part of the
XML:


  
  
  
  
  
  


which is in turn caused, I think, by
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61

Is it plausible that libvirt 1.2.7 would have avoided trying to invoke a
script with an empty path, whereas libvirt 2.0.0 does not?

Secondly - if I move past the problem above by changing
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/designer.py#L61
to say 'conf.script = None' - I then find:
- no apparent error in nova-compute.log
- but my instances don't boot
- the following messages in the libvirt log:

Domain id=4 is tainted: high-privileges
char device redirected to /dev/pts/1 (label charserial1)
CPU feature tsc_adjust not found

I guess it's the last message that is the critical one here - can anyone
tell me more about it?

Thanks,
  Neil
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev