Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Simon Leinen]

Ihar Hrachyshka writes:
> I just saw the plan merged in master:
> https://review.openstack.org/#/c/451492/ That's cool! Can we also
> backport the change to Ocata and maybe Newton so that we don't hit the
> same bug there? The backport is already up:
> https://review.openstack.org/#/c/456506/

For Ocata this makes sense, but note that the Newton UCA doesn't include
the libvirt-bin package.
-- 
Simon.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Ihar Hrachyshka]

I just saw the plan merged in master:
https://review.openstack.org/#/c/451492/ That's cool! Can we also
backport the change to Ocata and maybe Newton so that we don't hit the
same bug there? The backport is already up:
https://review.openstack.org/#/c/456506/

Thanks,
Ihar

On Mon, Apr 3, 2017 at 4:06 PM, Clark Boylan  wrote:
> Hello,
>
> One of the major sets of issues currently affecting gate testing is
> Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us
> and they happen frequently [0][1][2]. These issues appear to only affect
> Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in
> #openstack-nova it is clear that Libvirt isn't interested in debugging
> such an old version of Libvirt (1.3.1). And while it isn't entirely
> clear to me which exact version would be acceptable to them the Ubuntu
> Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0).
>
> I have pushed a change to devstack [3] to enable using UCA which pulls
> in new Libvirt and mostly seems to work. I think we should consider
> switching to UCA as this may fix our Libvirt problems and if it doesn't,
> we will be closer to a version of Libvirt that upstream should be
> willing to fix.
>
> This isn't the most straightfoward switch as UCA has a different repo
> for each OpenStack release. libvirt-python is sensitve to the underlying
> library changing; it is backward compatible but libvirt-python built
> against older libvirt won't work against new libvirt. The result is a
> libvirt-python wheel built on our wheel mirror does not work with UCA.
> On the positive side both the OpenStack puppet modules and OpenStack
> Ansible are using UCA with their deployment tooling so this should get
> us closer to what people are using in the wild.
>
> After some thought I think my preferred method of rolling this out would
> be to blacklist libvirt-python from our wheel mirror building entirely
> and force installs to happen from source so that we are base Xenial and
> $UCA_version independent (local testing of these builds show it only
> takes a few seconds). Then have specific jobs (like devstack) explicitly
> opt into the UCA repo appropriate for them (if any). This last bit is
> from feedback from OpenStack Ansible that having the base images be
> fairly clean is desirable, but it would also be hard to know which
> version of UCA is appropriate for our Xenial images (this likely differs
> based on the job).
>
> Now its entirely possible that newer Libvirt will be worse than current
> (old) Libvirt; however, being closer to upstream should make getting
> fixes easier. Would be great if those with a better understanding of
> Libvirt could chime in on this if I am completely wrong here.
>
> Finally it is worth noting that we will get newer packages of other
> software as well, most notably openvswitch will be version 2.6.1 instead
> of 2.5.0.
>
> Any other thoughts or ideas?
>
> [0] http://status.openstack.org/elastic-recheck/#1646779
> [1] http://status.openstack.org/elastic-recheck/#1643911
> [2] http://status.openstack.org/elastic-recheck/#1638982
> [3] https://review.openstack.org/451492
>
> Thank you,
> Clark
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[James Page]

Hi Clark

On Tue, 4 Apr 2017 at 00:08 Clark Boylan  wrote:

> One of the major sets of issues currently affecting gate testing is
> Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us
> and they happen frequently [0][1][2]. These issues appear to only affect
> Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in
> #openstack-nova it is clear that Libvirt isn't interested in debugging
> such an old version of Libvirt (1.3.1). And while it isn't entirely
> clear to me which exact version would be acceptable to them the Ubuntu
> Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0).
>

tl;dr - Christian (cpaelzer) is working towards resolution of the libvirt
1.3.1 issues in Xenial, but right now we're stuck on reproduction of the
issues outside of the gate.


> I have pushed a change to devstack [3] to enable using UCA which pulls
> in new Libvirt and mostly seems to work. I think we should consider
> switching to UCA as this may fix our Libvirt problems and if it doesn't,
> we will be closer to a version of Libvirt that upstream should be
> willing to fix.
>
> This isn't the most straightfoward switch as UCA has a different repo
> for each OpenStack release. libvirt-python is sensitve to the underlying
> library changing; it is backward compatible but libvirt-python built
> against older libvirt won't work against new libvirt. The result is a
> libvirt-python wheel built on our wheel mirror does not work with UCA.
> On the positive side both the OpenStack puppet modules and OpenStack
> Ansible are using UCA with their deployment tooling so this should get
> us closer to what people are using in the wild.
>

+1 on taking this approach - this also aligned with what we deploy and test
with for the OpenStack Charms.

Please stick to using the updates pockets from the Ubuntu Cloud Archive
(which I can see in the review referenced that you are doing); all UCA
testing of packaging and version updates is done in the proposed pockets so
this should ensure a better level of stability for the OpenStack gate.

[...]

> Finally it is worth noting that we will get newer packages of other
> software as well, most notably openvswitch will be version 2.6.1 instead
> of 2.5.0.


Worth noting that in the same way that we update OpenStack packages in the
UCA for new minor and patch releases, we also do the same for Open vSwitch
patch releases - so the Ocata UCA will get released Open vSwitch versions
on the 2.6.x series.

The Pike UCA will (probably) get a newer version of Ceph which might be of
interest for gate testing as well.

Cheers

James
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[James Page]

On Tue, 4 Apr 2017 at 14:38 Daniel P. Berrange  wrote:

> On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote:
> > Hello,
> >
> > One of the major sets of issues currently affecting gate testing is
> > Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us
> > and they happen frequently [0][1][2]. These issues appear to only affect
> > Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in
> > #openstack-nova it is clear that Libvirt isn't interested in debugging
> > such an old version of Libvirt (1.3.1). And while it isn't entirely
> > clear to me which exact version would be acceptable to them the Ubuntu
> > Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0).
>
> If going to the libvirt upstream community for help, we'd generally want
> to see the latest upstream release being used. Ideally along with
> willingness
> to test git master if investigating a troublesome issue, but we understand
> using git master is not practical for many people.
>
> If using an old version provided by an OS distro, then we would generally
> expect the OS distro maintainers to lead the investigation, and take the
> responsibility for reproducing on latest upstream. Upstream libvirt simply
> doesn't have bandwidth to do the OS distro maintainers job for them when
> using old distro versions.
>

Agree on the responsibility for maintenance of libvirt in Ubuntu. Christian
Ehrhardt (cpaelzer) has been working to help resolve the libvirt 1.3.1 bugs
currently impacting the gate and does have updated packages available for
testing, but right now we're not able to reproduce the bugs outside of the
gate environment so verifying that these actually resolve the underlying
issues is proving problematic.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Monty Taylor]

On 04/04/2017 12:14 PM, Daniel P. Berrange wrote:

On Tue, Apr 04, 2017 at 09:21:27AM -0700, Clark Boylan wrote:

On Tue, Apr 4, 2017, at 06:36 AM, Daniel P. Berrange wrote:

On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote:

I have pushed a change to devstack [3] to enable using UCA which pulls
in new Libvirt and mostly seems to work. I think we should consider
switching to UCA as this may fix our Libvirt problems and if it doesn't,
we will be closer to a version of Libvirt that upstream should be
willing to fix.

This isn't the most straightfoward switch as UCA has a different repo
for each OpenStack release. libvirt-python is sensitve to the underlying
library changing; it is backward compatible but libvirt-python built
against older libvirt won't work against new libvirt. The result is a
libvirt-python wheel built on our wheel mirror does not work with UCA.


I'm surprised about that - could you elaborate on what's broken for you.
The libvirt.so provides a stable public API, and the standalone python
binding only uses public APIs from libvirt.so.  IOW you should be able
to build libvirt-python against 1.3.0 and then use it against 2.5.0 with
no problems.

NB, *before* libvirt-python lived on pypi, it used some non-public
libvirt.so symbols, so was tied to the exact libvirt.so it was built
against. Assuming you're using the pypi version this should no longer
apply.


The specific issue was "AttributeError: 'module' object has no attribute
'VIR_MIGRATE_POSTCOPY'" where module here is libvirt (full log and
traceback at [0]). The libvirt-python module here was built against
Libvirt 1.3.1 turned into a wheel and copied into our wheel mirror. Then
when running against Libvirt 2.5.0 Nova seems to have detected that
newer features should be present that are not reflected in the compiled
libvirt-python resulting in the error. This crashed nova compute.

Problem was easily corrected by preventing devstack from using our wheel
mirror for libvirt-python which resulted in a new installation built
against Libvirt 2.5.0.

It seems like the API is stable enough for backward compatibility but
not forward compatibility. Its also possible that Nova is doing version
checking in a buggy way and should be checking what the libvirt-python
version is and what it supports?


Ok, so yeah your last sentance is the correct interpretation. You've built
libvirt-python against libvirt v1.3.1, so it only includes support for
constants & methods that exist in that version. The VIR_MIGRATE_POSTCOPY
constant was introduced in v1.3.3, so it will not be included in the
libvirt-python you built.

When checking features Nova calls a libvirt API that returns the version
of the libvirtd daemon, which is v2.5.0, and then just blindly assumes
libvirt-python has the same version.

Unfortunately there is no way for Nova to determine what libvirt version
the python binding was built against, so it can't improve its version
check in this respect. To deal with this, Nova would two options:

 - Provide a nova.conf parameter to force it to assume an older libvirt
   version, thus disabling the features regardless of what libvirtd
   supports
 - Make nova check for existance of the python constants / APIs it is
   trying to use, in addition to checking the libvirt version

The first option is pretty trivial to do if needed. The second option would
be the more correct approach, but a much bigger maint burden, so I'm not
convinced it is worth it.


I'm not convinced either are worth it - I think if we just stop building 
libvirt wheels in our wheel mirror it'll resolve itself. Most places 
that are actually running OpenStack for real aren't going to be building 
their install packages blind to what version of the OS they're going to 
install on - so it's really only a problem with one of our gate 
optimizations.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Monty Taylor]

On 04/04/2017 12:05 PM, Ihar Hrachyshka wrote:

I mentioned that in a meeting today but for greater visibility...

This change in how we gate against LTS raises a question whether what
we claim in: 
https://review.openstack.org/#/c/402940/4/reference/project-testing-interface.rst
is still valid. The wording of the document is such that suggests we
test against LTS, and with the UCA adoption in gate, that will no
longer be true. I am not against the change per se, I just want the
decision to be thought through and advertised to consumers beyond
tight circle of upstream developers.


Totally - and that's a good point. I think we may just need to update 
the wording some. The intent has always been that we develop targetting 
the software we think is appropriate for OpenStack- but the LTS line and 
testing are to ensure we don't introduce a dependency that's 
unreasonably hard to backport on top of an LTS release.


An example of that would be growing a dependency on a kernel feature 
that would be too much to backport - or if we had decided to move to 
Python3 back in the RHEL6 days (before there was a python3 available)


RDO and UCA are both ways that the distro vendors themselves are 
providing backport software on top of their LTS releases. So if it's in 
one of those the distro vendor themselves have de-facto communicated 
that it's not unreasonable to backport that piece of software to run on 
top of the LTS - since they have already done it.


All of that is not spelled out clearly though - so I agree, it would be 
potentially useful to describe that situation more clearly in the PTI.



Ihar

On Mon, Apr 3, 2017 at 4:39 PM, Clark Boylan  wrote:

On Mon, Apr 3, 2017, at 04:33 PM, Ihar Hrachyshka wrote:

Are we going to keep some job not using the archive, to make sure we
don't
break people using packages from LTS? Maybe just periodic/non-voting
would
be enough to keep it working on older packages.


Old stable branches will continue to run against the base LTS release.
But considering that this is how users of Ubuntu get newer packages, the
deployment projects are using it for newer branches, and we know that
the base LTS is broken, I'm not sure how much benefit there would be to
keep testing on the base LTS if we make the switch.

Its important to note that anyone using base LTS is broken and we know
this because of our testing.

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Daniel P. Berrange]

On Tue, Apr 04, 2017 at 09:21:27AM -0700, Clark Boylan wrote:
> On Tue, Apr 4, 2017, at 06:36 AM, Daniel P. Berrange wrote:
> > On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote:
> > > I have pushed a change to devstack [3] to enable using UCA which pulls
> > > in new Libvirt and mostly seems to work. I think we should consider
> > > switching to UCA as this may fix our Libvirt problems and if it doesn't,
> > > we will be closer to a version of Libvirt that upstream should be
> > > willing to fix.
> > > 
> > > This isn't the most straightfoward switch as UCA has a different repo
> > > for each OpenStack release. libvirt-python is sensitve to the underlying
> > > library changing; it is backward compatible but libvirt-python built
> > > against older libvirt won't work against new libvirt. The result is a
> > > libvirt-python wheel built on our wheel mirror does not work with UCA.
> > 
> > I'm surprised about that - could you elaborate on what's broken for you.
> > The libvirt.so provides a stable public API, and the standalone python
> > binding only uses public APIs from libvirt.so.  IOW you should be able
> > to build libvirt-python against 1.3.0 and then use it against 2.5.0 with
> > no problems.
> > 
> > NB, *before* libvirt-python lived on pypi, it used some non-public
> > libvirt.so symbols, so was tied to the exact libvirt.so it was built
> > against. Assuming you're using the pypi version this should no longer
> > apply.
> 
> The specific issue was "AttributeError: 'module' object has no attribute
> 'VIR_MIGRATE_POSTCOPY'" where module here is libvirt (full log and
> traceback at [0]). The libvirt-python module here was built against
> Libvirt 1.3.1 turned into a wheel and copied into our wheel mirror. Then
> when running against Libvirt 2.5.0 Nova seems to have detected that
> newer features should be present that are not reflected in the compiled
> libvirt-python resulting in the error. This crashed nova compute.
> 
> Problem was easily corrected by preventing devstack from using our wheel
> mirror for libvirt-python which resulted in a new installation built
> against Libvirt 2.5.0.
> 
> It seems like the API is stable enough for backward compatibility but
> not forward compatibility. Its also possible that Nova is doing version
> checking in a buggy way and should be checking what the libvirt-python
> version is and what it supports?

Ok, so yeah your last sentance is the correct interpretation. You've built
libvirt-python against libvirt v1.3.1, so it only includes support for
constants & methods that exist in that version. The VIR_MIGRATE_POSTCOPY
constant was introduced in v1.3.3, so it will not be included in the
libvirt-python you built.

When checking features Nova calls a libvirt API that returns the version
of the libvirtd daemon, which is v2.5.0, and then just blindly assumes
libvirt-python has the same version.

Unfortunately there is no way for Nova to determine what libvirt version
the python binding was built against, so it can't improve its version
check in this respect. To deal with this, Nova would two options:

 - Provide a nova.conf parameter to force it to assume an older libvirt
   version, thus disabling the features regardless of what libvirtd
   supports
 - Make nova check for existance of the python constants / APIs it is
   trying to use, in addition to checking the libvirt version

The first option is pretty trivial to do if needed. The second option would
be the more correct approach, but a much bigger maint burden, so I'm not
convinced it is worth it.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Ihar Hrachyshka]

I mentioned that in a meeting today but for greater visibility...

This change in how we gate against LTS raises a question whether what
we claim in: 
https://review.openstack.org/#/c/402940/4/reference/project-testing-interface.rst
is still valid. The wording of the document is such that suggests we
test against LTS, and with the UCA adoption in gate, that will no
longer be true. I am not against the change per se, I just want the
decision to be thought through and advertised to consumers beyond
tight circle of upstream developers.

Ihar

On Mon, Apr 3, 2017 at 4:39 PM, Clark Boylan  wrote:
> On Mon, Apr 3, 2017, at 04:33 PM, Ihar Hrachyshka wrote:
>> Are we going to keep some job not using the archive, to make sure we
>> don't
>> break people using packages from LTS? Maybe just periodic/non-voting
>> would
>> be enough to keep it working on older packages.
>
> Old stable branches will continue to run against the base LTS release.
> But considering that this is how users of Ubuntu get newer packages, the
> deployment projects are using it for newer branches, and we know that
> the base LTS is broken, I'm not sure how much benefit there would be to
> keep testing on the base LTS if we make the switch.
>
> Its important to note that anyone using base LTS is broken and we know
> this because of our testing.
>
> Clark
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Clark Boylan]

On Tue, Apr 4, 2017, at 06:36 AM, Daniel P. Berrange wrote:
> On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote:
> > I have pushed a change to devstack [3] to enable using UCA which pulls
> > in new Libvirt and mostly seems to work. I think we should consider
> > switching to UCA as this may fix our Libvirt problems and if it doesn't,
> > we will be closer to a version of Libvirt that upstream should be
> > willing to fix.
> > 
> > This isn't the most straightfoward switch as UCA has a different repo
> > for each OpenStack release. libvirt-python is sensitve to the underlying
> > library changing; it is backward compatible but libvirt-python built
> > against older libvirt won't work against new libvirt. The result is a
> > libvirt-python wheel built on our wheel mirror does not work with UCA.
> 
> I'm surprised about that - could you elaborate on what's broken for you.
> The libvirt.so provides a stable public API, and the standalone python
> binding only uses public APIs from libvirt.so.  IOW you should be able
> to build libvirt-python against 1.3.0 and then use it against 2.5.0 with
> no problems.
> 
> NB, *before* libvirt-python lived on pypi, it used some non-public
> libvirt.so symbols, so was tied to the exact libvirt.so it was built
> against. Assuming you're using the pypi version this should no longer
> apply.

The specific issue was "AttributeError: 'module' object has no attribute
'VIR_MIGRATE_POSTCOPY'" where module here is libvirt (full log and
traceback at [0]). The libvirt-python module here was built against
Libvirt 1.3.1 turned into a wheel and copied into our wheel mirror. Then
when running against Libvirt 2.5.0 Nova seems to have detected that
newer features should be present that are not reflected in the compiled
libvirt-python resulting in the error. This crashed nova compute.

Problem was easily corrected by preventing devstack from using our wheel
mirror for libvirt-python which resulted in a new installation built
against Libvirt 2.5.0.

It seems like the API is stable enough for backward compatibility but
not forward compatibility. Its also possible that Nova is doing version
checking in a buggy way and should be checking what the libvirt-python
version is and what it supports?

[0]
http://logs.openstack.org/92/451492/4/check/gate-devstack-dsvm-updown-ubuntu-xenial/19dca66/logs/screen-n-cpu.txt.gz?level=ERROR#_2017-03-29_22_55_19_038

Hope this helps,
Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Jesse Pretorius]

On 4/4/17, 12:06 AM, "Clark Boylan"  wrote:

> After some thought I think my preferred method of rolling this out would
> be to blacklist libvirt-python from our wheel mirror building entirely
> and force installs to happen from source so that we are base Xenial and
> $UCA_version independent (local testing of these builds show it only
> takes a few seconds). Then have specific jobs (like devstack) explicitly
> opt into the UCA repo appropriate for them (if any). This last bit is
> from feedback from OpenStack Ansible that having the base images be
> fairly clean is desirable, but it would also be hard to know which
> version of UCA is appropriate for our Xenial images (this likely differs
> based on the job).

An approach that could be taken here is to bake the repo config (and install 
packages from that repo) for the UCA version that’s the earliest version we 
support for that OS. For example – the earliest version that supports Xenial is 
Newton, so pre-configure and install packages into the image from UCA 
Xenial/Newton. Then in any jobs which care about which version of UCA is used 
the repo config can be changed and the packages updated. This potentially saves 
a little gate time.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




Rackspace Limited is a company registered in England & Wales (company 
registered number 03897010) whose registered office is at 5 Millington Road, 
Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be 
viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may 
contain confidential or privileged information intended for the recipient. Any 
dissemination, distribution or copying of the enclosed material is prohibited. 
If you receive this transmission in error, please notify us immediately by 
e-mail at ab...@rackspace.com and delete the original message. Your cooperation 
is appreciated.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Daniel P. Berrange]

On Tue, Apr 04, 2017 at 07:16:51AM -0400, Sean Dague wrote:
> This is definitely a trade off, I know the last time we tried UCA we had
> such a high failure rate we had to revert. But, that was a much younger
> libvirt that was only just starting to get heavy testing in OpenStack.
> So it feels like it's worth a shot. It will at least be interesting to
> see if it makes things better.
> 
> The libvirt bump will bring in libvirtd and live migration postcopy for
> testability on the Nova side, both of which would be good things.

NB, you'd need corresponding QEMU bump too for post-copy, but IIUC the
UCA contain that, so it'd be fine.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Daniel P. Berrange]

On Mon, Apr 03, 2017 at 04:06:53PM -0700, Clark Boylan wrote:
> Hello,
> 
> One of the major sets of issues currently affecting gate testing is
> Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us
> and they happen frequently [0][1][2]. These issues appear to only affect
> Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in
> #openstack-nova it is clear that Libvirt isn't interested in debugging
> such an old version of Libvirt (1.3.1). And while it isn't entirely
> clear to me which exact version would be acceptable to them the Ubuntu
> Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0).

If going to the libvirt upstream community for help, we'd generally want
to see the latest upstream release being used. Ideally along with willingness
to test git master if investigating a troublesome issue, but we understand
using git master is not practical for many people.

If using an old version provided by an OS distro, then we would generally
expect the OS distro maintainers to lead the investigation, and take the
responsibility for reproducing on latest upstream. Upstream libvirt simply
doesn't have bandwidth to do the OS distro maintainers job for them when
using old distro versions.

> I have pushed a change to devstack [3] to enable using UCA which pulls
> in new Libvirt and mostly seems to work. I think we should consider
> switching to UCA as this may fix our Libvirt problems and if it doesn't,
> we will be closer to a version of Libvirt that upstream should be
> willing to fix.
> 
> This isn't the most straightfoward switch as UCA has a different repo
> for each OpenStack release. libvirt-python is sensitve to the underlying
> library changing; it is backward compatible but libvirt-python built
> against older libvirt won't work against new libvirt. The result is a
> libvirt-python wheel built on our wheel mirror does not work with UCA.

I'm surprised about that - could you elaborate on what's broken for you.
The libvirt.so provides a stable public API, and the standalone python
binding only uses public APIs from libvirt.so.  IOW you should be able
to build libvirt-python against 1.3.0 and then use it against 2.5.0 with
no problems.

NB, *before* libvirt-python lived on pypi, it used some non-public
libvirt.so symbols, so was tied to the exact libvirt.so it was built
against. Assuming you're using the pypi version this should no longer
apply.

> Now its entirely possible that newer Libvirt will be worse than current
> (old) Libvirt; however, being closer to upstream should make getting
> fixes easier. Would be great if those with a better understanding of
> Libvirt could chime in on this if I am completely wrong here.

As a general rule your expectation is right - newer libvirt should
generally be better. There is always the chance of screwups, but we issue
maint releases where needed - the only question mark would be whether UCA
pulls in any maint releases. I would like to think that if such a problem
happened, openstack would be able to escalate it to a Canonical maintainer
to get a maint release / patch into UCA, since presumably any such bug
would be important to Canonical customers using OpenStack too.

> Finally it is worth noting that we will get newer packages of other
> software as well, most notably openvswitch will be version 2.6.1 instead
> of 2.5.0.

IIUC, you'd get newer QEMU/KVM too, which is arguably just as desirable
as getting newer libvirt.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://entangle-photo.org   -o-http://search.cpan.org/~danberr/ :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Monty Taylor]

On 04/03/2017 06:06 PM, Clark Boylan wrote:

Hello,

One of the major sets of issues currently affecting gate testing is
Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us
and they happen frequently [0][1][2]. These issues appear to only affect
Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in
#openstack-nova it is clear that Libvirt isn't interested in debugging
such an old version of Libvirt (1.3.1). And while it isn't entirely
clear to me which exact version would be acceptable to them the Ubuntu
Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0).

I have pushed a change to devstack [3] to enable using UCA which pulls
in new Libvirt and mostly seems to work. I think we should consider
switching to UCA as this may fix our Libvirt problems and if it doesn't,
we will be closer to a version of Libvirt that upstream should be
willing to fix.

This isn't the most straightfoward switch as UCA has a different repo
for each OpenStack release. libvirt-python is sensitve to the underlying
library changing; it is backward compatible but libvirt-python built
against older libvirt won't work against new libvirt. The result is a
libvirt-python wheel built on our wheel mirror does not work with UCA.
On the positive side both the OpenStack puppet modules and OpenStack
Ansible are using UCA with their deployment tooling so this should get
us closer to what people are using in the wild.

After some thought I think my preferred method of rolling this out would
be to blacklist libvirt-python from our wheel mirror building entirely
and force installs to happen from source so that we are base Xenial and
$UCA_version independent (local testing of these builds show it only
takes a few seconds). Then have specific jobs (like devstack) explicitly
opt into the UCA repo appropriate for them (if any). This last bit is
from feedback from OpenStack Ansible that having the base images be
fairly clean is desirable, but it would also be hard to know which
version of UCA is appropriate for our Xenial images (this likely differs
based on the job).

Now its entirely possible that newer Libvirt will be worse than current
(old) Libvirt; however, being closer to upstream should make getting
fixes easier. Would be great if those with a better understanding of
Libvirt could chime in on this if I am completely wrong here.

Finally it is worth noting that we will get newer packages of other
software as well, most notably openvswitch will be version 2.6.1 instead
of 2.5.0.

Any other thoughts or ideas?


I think it's a great idea. I don't think there is actual value in trying 
to support ludicrously old libvirt.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Sean Dague]

This is definitely a trade off, I know the last time we tried UCA we had
such a high failure rate we had to revert. But, that was a much younger
libvirt that was only just starting to get heavy testing in OpenStack.
So it feels like it's worth a shot. It will at least be interesting to
see if it makes things better.

The libvirt bump will bring in libvirtd and live migration postcopy for
testability on the Nova side, both of which would be good things.

-Sean

On 04/03/2017 07:06 PM, Clark Boylan wrote:
> Hello,
> 
> One of the major sets of issues currently affecting gate testing is
> Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us
> and they happen frequently [0][1][2]. These issues appear to only affect
> Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in
> #openstack-nova it is clear that Libvirt isn't interested in debugging
> such an old version of Libvirt (1.3.1). And while it isn't entirely
> clear to me which exact version would be acceptable to them the Ubuntu
> Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0).
> 
> I have pushed a change to devstack [3] to enable using UCA which pulls
> in new Libvirt and mostly seems to work. I think we should consider
> switching to UCA as this may fix our Libvirt problems and if it doesn't,
> we will be closer to a version of Libvirt that upstream should be
> willing to fix.
> 
> This isn't the most straightfoward switch as UCA has a different repo
> for each OpenStack release. libvirt-python is sensitve to the underlying
> library changing; it is backward compatible but libvirt-python built
> against older libvirt won't work against new libvirt. The result is a
> libvirt-python wheel built on our wheel mirror does not work with UCA.
> On the positive side both the OpenStack puppet modules and OpenStack
> Ansible are using UCA with their deployment tooling so this should get
> us closer to what people are using in the wild.
> 
> After some thought I think my preferred method of rolling this out would
> be to blacklist libvirt-python from our wheel mirror building entirely
> and force installs to happen from source so that we are base Xenial and
> $UCA_version independent (local testing of these builds show it only
> takes a few seconds). Then have specific jobs (like devstack) explicitly
> opt into the UCA repo appropriate for them (if any). This last bit is
> from feedback from OpenStack Ansible that having the base images be
> fairly clean is desirable, but it would also be hard to know which
> version of UCA is appropriate for our Xenial images (this likely differs
> based on the job).
> 
> Now its entirely possible that newer Libvirt will be worse than current
> (old) Libvirt; however, being closer to upstream should make getting
> fixes easier. Would be great if those with a better understanding of
> Libvirt could chime in on this if I am completely wrong here.
> 
> Finally it is worth noting that we will get newer packages of other
> software as well, most notably openvswitch will be version 2.6.1 instead
> of 2.5.0.
> 
> Any other thoughts or ideas?
> 
> [0] http://status.openstack.org/elastic-recheck/#1646779
> [1] http://status.openstack.org/elastic-recheck/#1643911
> [2] http://status.openstack.org/elastic-recheck/#1638982
> [3] https://review.openstack.org/451492
> 
> Thank you,
> Clark
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Ian Wienand]

On 04/04/2017 09:06 AM, Clark Boylan wrote:
> I have pushed a change to devstack [3] to enable using UCA which pulls
> in new Libvirt and mostly seems to work. I think we should consider
> switching to UCA as this may fix our Libvirt problems and if it doesn't,
> we will be closer to a version of Libvirt that upstream should be
> willing to fix.

I'm not 100% sure where this leaves the
devstack-plugin-additional-pkg-repos work [1]; although to be honest
I've always thought that was a little unspecific.  We also have
"devstack-plugin-tar-installer" [2] which was working on installing
libvirt from upstream tarballs IIRC but looks dead?  There was quite
some discussion about this in [3] at the time.

> Finally it is worth noting that we will get newer packages of other
> software as well, most notably openvswitch will be version 2.6.1 instead
> of 2.5.0.

Just to write down the centos side of things -- we have the RDO repo
installed on all centos CI nodes.  That brings in the centos virt-sig
rebuild of qemu (qemu-kvm-ev) which is definitely required to run
trunk nova [4].  We are just using libvirt from standard updates
(2.0.0 based).  RDO provides openvswitch, which is 2.6.1 too.

> Then have specific jobs (like devstack) explicitly opt into the UCA
> repo appropriate for them (if any).

The idea being, presumably, that when devstack branches, it is
essentially pinned to whatever version of UCA it was using at the time
for its lifespan.

I do wonder if installing the latest version on the base images
simplifies things (so that by default, "apt-get install libvirt" in
any context gets the UCA version).  To handle the above, a devstack
branch would have to essentially do the opposite -- override the
default to whatever version it is pinned to.  More work at branch
point, however has the advantage that we don't have to constantly
communicate to people they need to "opt-in".

-i

[1] 
https://git.openstack.org/cgit/openstack/devstack-plugin-additional-pkg-repos
[2] https://git.openstack.org/cgit/openstack/devstack-plugin-tar-installer
[3] https://review.openstack.org/#/c/108714/
[4] http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Clark Boylan]

On Mon, Apr 3, 2017, at 04:33 PM, Ihar Hrachyshka wrote:
> Are we going to keep some job not using the archive, to make sure we
> don't
> break people using packages from LTS? Maybe just periodic/non-voting
> would
> be enough to keep it working on older packages.

Old stable branches will continue to run against the base LTS release.
But considering that this is how users of Ubuntu get newer packages, the
deployment projects are using it for newer branches, and we know that
the base LTS is broken, I'm not sure how much benefit there would be to
keep testing on the base LTS if we make the switch.

Its important to note that anyone using base LTS is broken and we know
this because of our testing.

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

[Ihar Hrachyshka]

Are we going to keep some job not using the archive, to make sure we don't
break people using packages from LTS? Maybe just periodic/non-voting would
be enough to keep it working on older packages.

On Mon, Apr 3, 2017 at 4:07 PM Clark Boylan  wrote:

> Hello,
>
> One of the major sets of issues currently affecting gate testing is
> Libvirt stability. Elastic-recheck is tracking Libvirt crashes for us
> and they happen frequently [0][1][2]. These issues appear to only affect
> Ubuntu Xenial (and not Trusty or CentOS or Fedora) and after talking in
> #openstack-nova it is clear that Libvirt isn't interested in debugging
> such an old version of Libvirt (1.3.1). And while it isn't entirely
> clear to me which exact version would be acceptable to them the Ubuntu
> Cloud Archive (UCA) does publish a much newer Libvirt (2.5.0).
>
> I have pushed a change to devstack [3] to enable using UCA which pulls
> in new Libvirt and mostly seems to work. I think we should consider
> switching to UCA as this may fix our Libvirt problems and if it doesn't,
> we will be closer to a version of Libvirt that upstream should be
> willing to fix.
>
> This isn't the most straightfoward switch as UCA has a different repo
> for each OpenStack release. libvirt-python is sensitve to the underlying
> library changing; it is backward compatible but libvirt-python built
> against older libvirt won't work against new libvirt. The result is a
> libvirt-python wheel built on our wheel mirror does not work with UCA.
> On the positive side both the OpenStack puppet modules and OpenStack
> Ansible are using UCA with their deployment tooling so this should get
> us closer to what people are using in the wild.
>
> After some thought I think my preferred method of rolling this out would
> be to blacklist libvirt-python from our wheel mirror building entirely
> and force installs to happen from source so that we are base Xenial and
> $UCA_version independent (local testing of these builds show it only
> takes a few seconds). Then have specific jobs (like devstack) explicitly
> opt into the UCA repo appropriate for them (if any). This last bit is
> from feedback from OpenStack Ansible that having the base images be
> fairly clean is desirable, but it would also be hard to know which
> version of UCA is appropriate for our Xenial images (this likely differs
> based on the job).
>
> Now its entirely possible that newer Libvirt will be worse than current
> (old) Libvirt; however, being closer to upstream should make getting
> fixes easier. Would be great if those with a better understanding of
> Libvirt could chime in on this if I am completely wrong here.
>
> Finally it is worth noting that we will get newer packages of other
> software as well, most notably openvswitch will be version 2.6.1 instead
> of 2.5.0.
>
> Any other thoughts or ideas?
>
> [0] http://status.openstack.org/elastic-recheck/#1646779
> [1] http://status.openstack.org/elastic-recheck/#1643911
> [2] http://status.openstack.org/elastic-recheck/#1638982
> [3] https://review.openstack.org/451492
>
> Thank you,
> Clark
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev