Re: [openstack-dev] supported dependency versioning and testing

[Jeremy Stanley]

On 2014-02-21 14:40:07 + (+), Daniel P. Berrange wrote:
[...]
> If I was to prioritize things given limited resources, I'd suggest that
> we should be validating that OpenStack has not broken its own code as the
> top priority. So testing lowest version would rank above testing highest
> version.
[...]

While I might agree with you on principle, in practice there's a
subtle assumption lurking in that statement: that testing the lowest
versions of dependencies is as easy as testing the most current
versions.

The monkey wrench in the works here is that testing the lowest
versions effectively requires one of two things. Either...

1) work with the Python community to improve pip by endowing it with
a proper dependency solver, or...

2) find some other means of getting arbitrary dependencies installed
at appropriate versions for testing (including newly proposed
dependencies which aren't yet packaged in mainstream Linux
distributions).

On the other hand, testing the latest of everything is what we do
now, mostly because that is pip's default (and at the end of the
day, its only reasonably reliable) behavior.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Joe Gordon]

On Fri, Feb 21, 2014 at 10:17 AM, Sean Dague  wrote:
> On 02/21/2014 11:02 AM, Daniel P. Berrange wrote:
>> On Fri, Feb 21, 2014 at 10:46:22AM -0500, Sean Dague wrote:
>>> On 02/21/2014 09:45 AM, Daniel P. Berrange wrote:
 On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote:
>
> So I'm one of the first people to utter "if it isn't tested, it's
> probably broken", however I also think we need to be realistic about the
> fact that if you did out the permutations of dependencies and config
> options, we'd have as many test matrix scenarios as grains of sand on
> the planet.
>
> I do think in some ways this is unique to OpenStack, in that our
> automated testing is head and shoulders above any other Open Source
> project out there, and most proprietary software systems I've seen.
>
> So this is about being pragmatic. In our dependency testing we are
> actually testing with most recent versions of everything. So I would
> think that even with libvirt, we should err in that direction.

 I'm very much against that, because IME, time & time again across
 all open source projects I've worked on, people silently introduce
 use of features/apis that only exist in newer versions without anyone
 ever noticing until it is too late.

> That being said, we also need to be a little bit careful about taking
> such a hard line about "supported vs. not" based on only what's in the
> gate. Because if we did the following things would be listed as
> unsupported (in increasing level of ridiculousness):
>
>  * Live migration
>  * Using qpid or zmq
>  * Running on anything other than Ubuntu 12.04
>  * Running on multiple nodes
>
> Supported to me means we think it should work, and if it doesn't, it's a
> high priority bug that will get fixed quickly. Testing is our sanity
> check. But it can't be considered that it will catch everything, at
> least not before the heat death of the universe.

 I agree we should be pragmatic here to some extent. We shouldn't aim to
 test every single intermediate version, or every possible permutation of
 versions - just a representative sample. Testing both lowest and highest
 versions is a reasonable sample set IMHO.
>>>
>>> Testing lower bounds is interesting, because of the way pip works. That
>>> being said, if someone wants to take ownership of building that job to
>>> start as a periodic job, I'm happy to point in the right direction. Just
>>> right now, it's a lower priority item than things like Tempest self
>>> testing, Heat actually gating, Neutron running in parallel, Nova API
>>> coverage.
>>
>> If it would be hard work to do it for python modules, we can at least
>> not remove the existing testing of an old libvirt version - simply add
>> an additional test with newer libvirt.
>
> Simply adding a test with newer libvirt isn't all that simple at the end
> of the day, as it requires building a new nodepool image. Because
> getting new libvirt in the existing test environment means cloud
> archive, and cloud archive means a ton of other new code as well. Plus
> in Juno we're presumably going to jump to 14.04 as our test base, which
> is going to be it's own big transition.
>
> So, I'm not opposed, but I also think bifurcating libvirt testing is a
> big enough change in the pipeline that it needs some pretty dedicated
> folks looking at it, and the implications there in. This isn't just a
> yaml change, set and forget it. And given where we are in the
> development cycle, I'm not sure trying to keep the gate stable with a
> new libvirt which we've known to be problematic, is the right time to do
> this.
>
> But, if someone is stepping up to work through it, can definitely mentor
> them on the right places to be poking.



So it sounds like the consensus here is:

* We should have a uniform policy (unless we take the platform vs app
distinction)
* Long term we want to have a lower bound gate job as well, but that
no one has stepped up to work on it yet
* Setting up libvirt min and libvirt max tests is non-trivial and
needs someone to work on it

So in the short term we shouldn't be forced to to hold libvirt back to
the minimal supported version in
gate(https://blueprints.launchpad.net/nova/+spec/support-libvirt-1x),
hopefully while someone steps up to get a minimal libvirt (and python
deps?) job in the gate.

>
> -Sean
>
> --
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Dean Troyer]

On Fri, Feb 21, 2014 at 1:42 PM, Mike Spreitzer  wrote:

> Ah, so the only distro regularly tested is Ubuntu 12.04?


Within the OpenStack Infrastructure Team -managed environment, generally
yes (with the addition of CentOS 6 for Py26 as noted in your quote below).
 However, that is not the only platform that testing is performed on and
certainly not the only platform that DevStack is supported on.


>  That's consistent with the other clues I am getting.  But it is
> inconsistent with the following remark found in
> http://devstack.org/overview.html :
>
> *The OpenStack Technical Committee (TC) has defined the current CI
> strategy to include the latest Ubuntu release and the latest RHEL release
> (for Python 2.6 testing).*
>

The bullet list immediately following gets very specific regarding our
distribution support policy.

devstack.org documents (sometimes confusingly as you've noted) only
DevStack itself.  CI uses DevStack as the primary install/configuration to
prepare for the Tempest tests, but is only one piece of that picture.  When
CI says they only test on Ubuntu LTS (I missed the letters LTS in that
quote too) that doesn't mean that DevStack is not tested only on LTS.

There are a number of vendors performing their own CI testing for
configurations that are not covered by OpenStack CI tests, including other
distributions.

It may not be obvious to core developers, but for us newbies there is a lot
> of inconsistent and incomplete documentation of what to expect and how to
> do things --- and it is scattered in a variety of places, easy to miss
> some; it really is a drag on a beginner's time.  I am trying to point out
> and fix doc problems as I discover them, but am not myself so sure what all
> the right answers are.
>

Thaks for helping.  OpenStack is huge; I've got the benefit of having
learned it as it grew.  We need this kind of input from fresh eyes to help
flush out the inconsistencies that I know I am blind to.

dt

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Mike Spreitzer]

Sean Dague  wrote on 02/20/2014 02:45:03 PM:
> ...
> That being said, we also need to be a little bit careful about taking
> such a hard line about "supported vs. not" based on only what's in the
> gate. Because if we did the following things would be listed as
> unsupported (in increasing level of ridiculousness):
> 
>  * Live migration
>  * Using qpid or zmq
>  * Running on anything other than Ubuntu 12.04
>  * Running on multiple nodes

Ah, so the only distro regularly tested is Ubuntu 12.04?  That's 
consistent with the other clues I am getting.  But it is inconsistent with 
the following remark found in http://devstack.org/overview.html :

The OpenStack Technical Committee (TC) has defined the current CI strategy 
to include the latest Ubuntu release and the latest RHEL release (for 
Python 2.6 testing).

It may not be obvious to core developers, but for us newbies there is a 
lot of inconsistent and incomplete documentation of what to expect and how 
to do things --- and it is scattered in a variety of places, easy to miss 
some; it really is a drag on a beginner's time.  I am trying to point out 
and fix doc problems as I discover them, but am not myself so sure what 
all the right answers are.

Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Sean Dague]

On 02/21/2014 11:02 AM, Daniel P. Berrange wrote:
> On Fri, Feb 21, 2014 at 10:46:22AM -0500, Sean Dague wrote:
>> On 02/21/2014 09:45 AM, Daniel P. Berrange wrote:
>>> On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote:

 So I'm one of the first people to utter "if it isn't tested, it's
 probably broken", however I also think we need to be realistic about the
 fact that if you did out the permutations of dependencies and config
 options, we'd have as many test matrix scenarios as grains of sand on
 the planet.

 I do think in some ways this is unique to OpenStack, in that our
 automated testing is head and shoulders above any other Open Source
 project out there, and most proprietary software systems I've seen.

 So this is about being pragmatic. In our dependency testing we are
 actually testing with most recent versions of everything. So I would
 think that even with libvirt, we should err in that direction.
>>>
>>> I'm very much against that, because IME, time & time again across
>>> all open source projects I've worked on, people silently introduce
>>> use of features/apis that only exist in newer versions without anyone
>>> ever noticing until it is too late.
>>>
 That being said, we also need to be a little bit careful about taking
 such a hard line about "supported vs. not" based on only what's in the
 gate. Because if we did the following things would be listed as
 unsupported (in increasing level of ridiculousness):

  * Live migration
  * Using qpid or zmq
  * Running on anything other than Ubuntu 12.04
  * Running on multiple nodes

 Supported to me means we think it should work, and if it doesn't, it's a
 high priority bug that will get fixed quickly. Testing is our sanity
 check. But it can't be considered that it will catch everything, at
 least not before the heat death of the universe.
>>>
>>> I agree we should be pragmatic here to some extent. We shouldn't aim to
>>> test every single intermediate version, or every possible permutation of
>>> versions - just a representative sample. Testing both lowest and highest
>>> versions is a reasonable sample set IMHO.
>>
>> Testing lower bounds is interesting, because of the way pip works. That
>> being said, if someone wants to take ownership of building that job to
>> start as a periodic job, I'm happy to point in the right direction. Just
>> right now, it's a lower priority item than things like Tempest self
>> testing, Heat actually gating, Neutron running in parallel, Nova API
>> coverage.
> 
> If it would be hard work to do it for python modules, we can at least
> not remove the existing testing of an old libvirt version - simply add
> an additional test with newer libvirt.

Simply adding a test with newer libvirt isn't all that simple at the end
of the day, as it requires building a new nodepool image. Because
getting new libvirt in the existing test environment means cloud
archive, and cloud archive means a ton of other new code as well. Plus
in Juno we're presumably going to jump to 14.04 as our test base, which
is going to be it's own big transition.

So, I'm not opposed, but I also think bifurcating libvirt testing is a
big enough change in the pipeline that it needs some pretty dedicated
folks looking at it, and the implications there in. This isn't just a
yaml change, set and forget it. And given where we are in the
development cycle, I'm not sure trying to keep the gate stable with a
new libvirt which we've known to be problematic, is the right time to do
this.

But, if someone is stepping up to work through it, can definitely mentor
them on the right places to be poking.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Kashyap Chamarthy]

On Thu, Feb 20, 2014 at 10:31:06AM -0800, Joe Gordon wrote:
> Hi All,
> 
> I discussion recently came up inside of nova about what it means
> supported version for a dependency means.  in libvirt we gate on the
> minimal version that we support but for all python dependencies we
> gate on the highest version that passes our requirements. While we all
> agree that having two different ways of choosing which version to test
> (min and max) is bad, there are good arguments for doing both.
> 
> testing most recent version:
> * We want to make sure we support the latest and greatest
> * Bug fixes
> * Quickly discover backwards incompatible changes so we can deal
> with them as they arise instead of in batch
> 
> Testing lowest version supported:
> * Make sure we don't land any code that breaks compatibility with
> the lowest version we say we support
> 
> 
> A few questions and ideas on how to move forward.
> * How do other projects deal with this? This problem isn't unique
> in OpenStack.
> * What are the issues with making one gate job use the latest
> versions and one use the lowest supported versions?
> * Given our finite resources what gets us the furthest?


tl;dr -- I've read the further replies in the thread. FWIW, the
suggestion of testing with lowest and higest versions sounds reasonable
to me.

I think I remember the bug you're alluding to here[1] -- I tried to
reproduce it a couple of times in a Fedora 20 environment, but later
moved on (noting relevant details in the bug) to other issues as I
realized after initial investigation that the fix exists in a _newer_
version of Libvirt (which the Gate machine needs to be updated to)[2].
Later, Sean Dague pointed on IRC there was another dependent bug[3]
which is preventing to bump up the Libvirt version on Gate.

Putting my Fedora distro user hat on: I try (as humanly as possible) to
keep on top of OpenStack bits with whatever is newest availalbe on
Fedora Rawhide (mostly - RPMs built from upstream git). And often with
its underlying Virtualization components - Libvirt/QEMU RPMs built from
git as well and ensure to test Minimal OpenStack (components I care
about) works without exploding. I'll do whatever I can to be helpful
here to continue to test the higher versions.


  [1] https://bugs.launchpad.net/nova/+bug/1254872 --libvirtError: Timed
  out during operation: cannot acquire state change lock 
  [2] https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix
  [3] https://bugs.launchpad.net/nova/+bug/1228977



-- 
/kashyap

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Daniel P. Berrange]

On Fri, Feb 21, 2014 at 10:46:22AM -0500, Sean Dague wrote:
> On 02/21/2014 09:45 AM, Daniel P. Berrange wrote:
> > On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote:
> >>
> >> So I'm one of the first people to utter "if it isn't tested, it's
> >> probably broken", however I also think we need to be realistic about the
> >> fact that if you did out the permutations of dependencies and config
> >> options, we'd have as many test matrix scenarios as grains of sand on
> >> the planet.
> >>
> >> I do think in some ways this is unique to OpenStack, in that our
> >> automated testing is head and shoulders above any other Open Source
> >> project out there, and most proprietary software systems I've seen.
> >>
> >> So this is about being pragmatic. In our dependency testing we are
> >> actually testing with most recent versions of everything. So I would
> >> think that even with libvirt, we should err in that direction.
> > 
> > I'm very much against that, because IME, time & time again across
> > all open source projects I've worked on, people silently introduce
> > use of features/apis that only exist in newer versions without anyone
> > ever noticing until it is too late.
> > 
> >> That being said, we also need to be a little bit careful about taking
> >> such a hard line about "supported vs. not" based on only what's in the
> >> gate. Because if we did the following things would be listed as
> >> unsupported (in increasing level of ridiculousness):
> >>
> >>  * Live migration
> >>  * Using qpid or zmq
> >>  * Running on anything other than Ubuntu 12.04
> >>  * Running on multiple nodes
> >>
> >> Supported to me means we think it should work, and if it doesn't, it's a
> >> high priority bug that will get fixed quickly. Testing is our sanity
> >> check. But it can't be considered that it will catch everything, at
> >> least not before the heat death of the universe.
> > 
> > I agree we should be pragmatic here to some extent. We shouldn't aim to
> > test every single intermediate version, or every possible permutation of
> > versions - just a representative sample. Testing both lowest and highest
> > versions is a reasonable sample set IMHO.
> 
> Testing lower bounds is interesting, because of the way pip works. That
> being said, if someone wants to take ownership of building that job to
> start as a periodic job, I'm happy to point in the right direction. Just
> right now, it's a lower priority item than things like Tempest self
> testing, Heat actually gating, Neutron running in parallel, Nova API
> coverage.

If it would be hard work to do it for python modules, we can at least
not remove the existing testing of an old libvirt version - simply add
an additional test with newer libvirt.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Sean Dague]

On 02/21/2014 09:45 AM, Daniel P. Berrange wrote:
> On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote:
>>
>> So I'm one of the first people to utter "if it isn't tested, it's
>> probably broken", however I also think we need to be realistic about the
>> fact that if you did out the permutations of dependencies and config
>> options, we'd have as many test matrix scenarios as grains of sand on
>> the planet.
>>
>> I do think in some ways this is unique to OpenStack, in that our
>> automated testing is head and shoulders above any other Open Source
>> project out there, and most proprietary software systems I've seen.
>>
>> So this is about being pragmatic. In our dependency testing we are
>> actually testing with most recent versions of everything. So I would
>> think that even with libvirt, we should err in that direction.
> 
> I'm very much against that, because IME, time & time again across
> all open source projects I've worked on, people silently introduce
> use of features/apis that only exist in newer versions without anyone
> ever noticing until it is too late.
> 
>> That being said, we also need to be a little bit careful about taking
>> such a hard line about "supported vs. not" based on only what's in the
>> gate. Because if we did the following things would be listed as
>> unsupported (in increasing level of ridiculousness):
>>
>>  * Live migration
>>  * Using qpid or zmq
>>  * Running on anything other than Ubuntu 12.04
>>  * Running on multiple nodes
>>
>> Supported to me means we think it should work, and if it doesn't, it's a
>> high priority bug that will get fixed quickly. Testing is our sanity
>> check. But it can't be considered that it will catch everything, at
>> least not before the heat death of the universe.
> 
> I agree we should be pragmatic here to some extent. We shouldn't aim to
> test every single intermediate version, or every possible permutation of
> versions - just a representative sample. Testing both lowest and highest
> versions is a reasonable sample set IMHO.

Testing lower bounds is interesting, because of the way pip works. That
being said, if someone wants to take ownership of building that job to
start as a periodic job, I'm happy to point in the right direction. Just
right now, it's a lower priority item than things like Tempest self
testing, Heat actually gating, Neutron running in parallel, Nova API
coverage.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Daniel P. Berrange]

On Thu, Feb 20, 2014 at 10:31:06AM -0800, Joe Gordon wrote:
> Hi All,
> 
> I discussion recently came up inside of nova about what it means
> supported version for a dependency means.  in libvirt we gate on the
> minimal version that we support but for all python dependencies we
> gate on the highest version that passes our requirements. While we all
> agree that having two different ways of choosing which version to test
> (min and max) is bad, there are good arguments for doing both.
> 
> testing most recent version:
> * We want to make sure we support the latest and greatest
> * Bug fixes
> * Quickly discover backwards incompatible changes so we can deal
> with them as they arise instead of in batch
> 
> Testing lowest version supported:
> * Make sure we don't land any code that breaks compatibility with
> the lowest version we say we support

I'm pretty strongly of the opinion that unless you test the minimum
declared version, you shouldn't claim to support it. Experience across
many open source projects is that far too often people silently introduce
features that require new versions of external deps and only get found
by the poor downstream user who tries to actually use the min required
version.

> A few questions and ideas on how to move forward.
> * How do other projects deal with this? This problem isn't unique
> in OpenStack.

The level of testing is fairly unique to openstack amongst
open source projects.

> * What are the issues with making one gate job use the latest
> versions and one use the lowest supported versions?

Double the resources is the obvious one I guess :-)

> * Only test some things on every commit or every day (periodic
> jobs)? But no one ever fixes those things when they break? who wants
> to own them? distros? deployers?

I think for testing done by openstack it should always be gating,
otherwise there's not much incentive to deal with the fallout.
If distros want to periodically test their own stack in a non-gating
manner let them make that choice themselves.

> * Other solutions?
> * Does it make sense to gate on the lowest version of libvirt but
> the highest version of python libs?

I think we should be consistent and at very least add testing of the
lowest python lib versions, so we can be confident that our declared
min versions are actually capable of working.

> * Given our finite resources what gets us the furthest?

As you say above, testing the lowest vs highest is targetting two different
use cases. 

  - Testing the lowest version demonstrates that OpenStack has not
broken its own code by introducing use of a new feature.

  - Testing the highest version demonstrates that OpenStack has not
been broken by 3rd party code introducing a regression.

If I was to prioritize things given limited resources, I'd suggest that
we should be validating that OpenStack has not broken its own code as the
top priority. So testing lowest version would rank above testing highest
version. I do think that both a very important to openstack though.

So if we have the resources to cope, I do think it would be very valuable
if we were able to have 2 sets of jobs, one focused on the highest version
and one focused on the lowest version both gating.

Of course, there are also intermediate versions to worry about. Given
that our test suite is fully opensource, I think it is pretty reasonable
to say that distro vendors / other downstream consumers should take the
responsibility to testing any intermediate versions that are in their
own distro.

One might argue that distros should also have responsibility for testing
the highest version, but I think there is value in having openstack
keep that responsibility to avoid too much duplication of effort, and to
ensure that openstack releases keep a good reputation for quality operation
wrt latest versions of external deps.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Daniel P. Berrange]

On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote:
> 
> So I'm one of the first people to utter "if it isn't tested, it's
> probably broken", however I also think we need to be realistic about the
> fact that if you did out the permutations of dependencies and config
> options, we'd have as many test matrix scenarios as grains of sand on
> the planet.
> 
> I do think in some ways this is unique to OpenStack, in that our
> automated testing is head and shoulders above any other Open Source
> project out there, and most proprietary software systems I've seen.
> 
> So this is about being pragmatic. In our dependency testing we are
> actually testing with most recent versions of everything. So I would
> think that even with libvirt, we should err in that direction.

I'm very much against that, because IME, time & time again across
all open source projects I've worked on, people silently introduce
use of features/apis that only exist in newer versions without anyone
ever noticing until it is too late.

> That being said, we also need to be a little bit careful about taking
> such a hard line about "supported vs. not" based on only what's in the
> gate. Because if we did the following things would be listed as
> unsupported (in increasing level of ridiculousness):
> 
>  * Live migration
>  * Using qpid or zmq
>  * Running on anything other than Ubuntu 12.04
>  * Running on multiple nodes
> 
> Supported to me means we think it should work, and if it doesn't, it's a
> high priority bug that will get fixed quickly. Testing is our sanity
> check. But it can't be considered that it will catch everything, at
> least not before the heat death of the universe.

I agree we should be pragmatic here to some extent. We shouldn't aim to
test every single intermediate version, or every possible permutation of
versions - just a representative sample. Testing both lowest and highest
versions is a reasonable sample set IMHO.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Mark McLoughlin]

On Thu, 2014-02-20 at 10:31 -0800, Joe Gordon wrote:
> Hi All,
> 
> I discussion recently came up inside of nova about what it means
> supported version for a dependency means.  in libvirt we gate on the
> minimal version that we support but for all python dependencies we
> gate on the highest version that passes our requirements. While we all
> agree that having two different ways of choosing which version to test
> (min and max) is bad, there are good arguments for doing both.
> 
> testing most recent version:
> * We want to make sure we support the latest and greatest
> * Bug fixes
> * Quickly discover backwards incompatible changes so we can deal
> with them as they arise instead of in batch
> 
> Testing lowest version supported:
> * Make sure we don't land any code that breaks compatibility with
> the lowest version we say we support

Good summary.

> A few questions and ideas on how to move forward.
> * How do other projects deal with this? This problem isn't unique
> in OpenStack.
> * What are the issues with making one gate job use the latest
> versions and one use the lowest supported versions?

I think this would be very useful.

Obviously it would take some effort on someone's part to set this up
initially, but I don't immediately see it being much of an ongoing
burden on developers.

> * Only test some things on every commit or every day (periodic
> jobs)? But no one ever fixes those things when they break? who wants
> to own them? distros? deployers?

The gate job above would most likely lead to us trying hard to maintain
support for older.

A periodic job would either go stale or we'd keep it going simply by
dropping support for older libraries. (Maybe that's ok)

> * Other solutions?
> * Does it make sense to gate on the lowest version of libvirt but
> the highest version of python libs?

We might be unconsciously drawing a platform vs app line here - that
libvirt is part of the platform and the python library stack is part of
our app - while still giving a nod to supporting the notion that the
python library stack is part of the platform.

Put it another way - we'd be wary of dropping support for an older
libvirt (because it rules out support for a platform) but not so much
with dropping support for an older python library (because meh, it's not
*really* part of the platform).

Or another way, we give explicit guidance about what exact versions of
libvirt we support (i.e. test with specific distros and whatever
versions of libvirt they have) and leave those versions newer than the
oldest version we explicitly support as a grey area. Similarly, we give
explicit guidance about the exact python library stack we support (i.e.
what we test now in the gate) and leave the older versions as a grey
area.

Perhaps rather than focusing on making this absolutely black and white,
we should focus on better communicating what we actually focus our
testing on? (i.e. rather than making the grey areas black, improve the
white areas)

Concretely, for every commit merged, we could publish:

  - the set of commits tested
  - details of the jobs passed:
  - the distro
  - installed packages and versions
  - output of pip freeze
  - configuration used
  - tests passed

Meh, feeling like I'm going off-topic a bit.

Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Sean Dague]

On 02/20/2014 06:30 PM, Sabari Murugesan wrote:
> But I do think running a job with lowest version may still help a
> developer realize that a feature in the latest library is not available
> in an older supported version. The person can then bump up the library's
> min version in the requirements. Today, it;s not possible to find this
> out until someone else runs into an issue with the older lib.
> 
> Ref:- I have noticed some bugs in the past and I did post
> 
>  about
> this to openstack-infra. From Jeremy Stanley's comment on that thread, I
> understand that it may be tough to setup it up this way due to
> transitive dependencies. 
> 
> -Sabari

So we've actually talked about doing this by using the
global-requirements lever. Basically modify the list to set everything
there to minimums, and see what happens. If you or someone else is
interested, I'd be happy to help walk you through it, but it personally
ended up low priority to get working.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Sabari Murugesan]

But I do think running a job with lowest version may still help a developer 
realize that a feature in the latest library is not available in an older 
supported version. The person can then bump up the library's min version in the 
requirements. Today, it;s not possible to find this out until someone else runs 
into an issue with the older lib.

Ref:- I have noticed some bugs in the past and I did post about this to 
openstack-infra. From Jeremy Stanley's comment on that thread, I understand 
that it may be tough to setup it up this way due to transitive dependencies. 

-Sabari

On Feb 20, 2014, at 3:06 PM, Sean Dague wrote:

> On 02/20/2014 05:50 PM, Christopher Yeoh wrote:
>> On Thu, 20 Feb 2014 14:45:03 -0500
>> Sean Dague  wrote:
>>> 
>>> So I'm one of the first people to utter "if it isn't tested, it's
>>> probably broken", however I also think we need to be realistic about
>>> the fact that if you did out the permutations of dependencies and
>>> config options, we'd have as many test matrix scenarios as grains of
>>> sand on the planet.
>> 
>> I think it makes sense to at test with all the most recent versions.
>> Perhaps all the lowest as well, though I'm not sure how common that
>> configuration would really be. And we can't do all of the various
>> combinations. 
>> 
>> But how about testing on a periodic job the version
>> configurations used by some of the major distros? So we know on which
>> distros we're going to be broken on by default (if we don't get around
>> to fixing them straight away) which will really help inform those
>> trying out the latest from git. Or are the distros doing this anyway
>> (either way having the info public and feeding back to our bug tracker
>> would be handy).
> 
> Honestly, I think our experience in even doing a homogenous gate means I
> think we should let the distros come in with 3rd party testing on those
> results. Because we don't really have the bw, in either people or
> machines, to explode that matrix much in infra.
> 
>   -Sean
> 
> -- 
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> https://urldefense.proofpoint.com/v1/url?u=http://dague.net/&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=t%2BccgWNa3RGhEJzfoLbZKcoWP6fDp0ZLoIyTd0CiIdk%3D%0A&m=kRy3u472erydn9aD1jOg%2FCQqzv1p0pQeNd%2BI8WXeZLk%3D%0A&s=493bac7a717a985bdb4d64eca320969ecb2a6df00d7ee8f5d8bcaf895dd24fe8
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=t%2BccgWNa3RGhEJzfoLbZKcoWP6fDp0ZLoIyTd0CiIdk%3D%0A&m=kRy3u472erydn9aD1jOg%2FCQqzv1p0pQeNd%2BI8WXeZLk%3D%0A&s=d484eedf637b01155f20eeda79378a9d506d6a39060517e70e868dcab571f3c6

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Christopher Yeoh]

On Thu, 20 Feb 2014 18:06:25 -0500
Sean Dague  wrote:
> 
> Honestly, I think our experience in even doing a homogenous gate
> means I think we should let the distros come in with 3rd party
> testing on those results. Because we don't really have the bw, in
> either people or machines, to explode that matrix much in infra.

Yes I think its even better if the distros would step up and
offer to provide the resources to do that testing and make the results
public.

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Sean Dague]

On 02/20/2014 05:50 PM, Christopher Yeoh wrote:
> On Thu, 20 Feb 2014 14:45:03 -0500
> Sean Dague  wrote:
>>
>> So I'm one of the first people to utter "if it isn't tested, it's
>> probably broken", however I also think we need to be realistic about
>> the fact that if you did out the permutations of dependencies and
>> config options, we'd have as many test matrix scenarios as grains of
>> sand on the planet.
> 
> I think it makes sense to at test with all the most recent versions.
> Perhaps all the lowest as well, though I'm not sure how common that
> configuration would really be. And we can't do all of the various
> combinations. 
> 
> But how about testing on a periodic job the version
> configurations used by some of the major distros? So we know on which
> distros we're going to be broken on by default (if we don't get around
> to fixing them straight away) which will really help inform those
> trying out the latest from git. Or are the distros doing this anyway
> (either way having the info public and feeding back to our bug tracker
> would be handy).

Honestly, I think our experience in even doing a homogenous gate means I
think we should let the distros come in with 3rd party testing on those
results. Because we don't really have the bw, in either people or
machines, to explode that matrix much in infra.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Christopher Yeoh]

On Thu, 20 Feb 2014 14:45:03 -0500
Sean Dague  wrote:
> 
> So I'm one of the first people to utter "if it isn't tested, it's
> probably broken", however I also think we need to be realistic about
> the fact that if you did out the permutations of dependencies and
> config options, we'd have as many test matrix scenarios as grains of
> sand on the planet.

I think it makes sense to at test with all the most recent versions.
Perhaps all the lowest as well, though I'm not sure how common that
configuration would really be. And we can't do all of the various
combinations. 

But how about testing on a periodic job the version
configurations used by some of the major distros? So we know on which
distros we're going to be broken on by default (if we don't get around
to fixing them straight away) which will really help inform those
trying out the latest from git. Or are the distros doing this anyway
(either way having the info public and feeding back to our bug tracker
would be handy).

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] supported dependency versioning and testing

[Sean Dague]

On 02/20/2014 01:31 PM, Joe Gordon wrote:
> Hi All,
> 
> I discussion recently came up inside of nova about what it means
> supported version for a dependency means.  in libvirt we gate on the
> minimal version that we support but for all python dependencies we
> gate on the highest version that passes our requirements. While we all
> agree that having two different ways of choosing which version to test
> (min and max) is bad, there are good arguments for doing both.
> 
> testing most recent version:
> * We want to make sure we support the latest and greatest
> * Bug fixes
> * Quickly discover backwards incompatible changes so we can deal
> with them as they arise instead of in batch
> 
> Testing lowest version supported:
> * Make sure we don't land any code that breaks compatibility with
> the lowest version we say we support
> 
> 
> A few questions and ideas on how to move forward.
> * How do other projects deal with this? This problem isn't unique
> in OpenStack.
> * What are the issues with making one gate job use the latest
> versions and one use the lowest supported versions?
> * Only test some things on every commit or every day (periodic
> jobs)? But no one ever fixes those things when they break? who wants
> to own them? distros? deployers?
> * Other solutions?
> * Does it make sense to gate on the lowest version of libvirt but
> the highest version of python libs?
> * Given our finite resources what gets us the furthest?

So I'm one of the first people to utter "if it isn't tested, it's
probably broken", however I also think we need to be realistic about the
fact that if you did out the permutations of dependencies and config
options, we'd have as many test matrix scenarios as grains of sand on
the planet.

I do think in some ways this is unique to OpenStack, in that our
automated testing is head and shoulders above any other Open Source
project out there, and most proprietary software systems I've seen.

So this is about being pragmatic. In our dependency testing we are
actually testing with most recent versions of everything. So I would
think that even with libvirt, we should err in that direction.

That being said, we also need to be a little bit careful about taking
such a hard line about "supported vs. not" based on only what's in the
gate. Because if we did the following things would be listed as
unsupported (in increasing level of ridiculousness):

 * Live migration
 * Using qpid or zmq
 * Running on anything other than Ubuntu 12.04
 * Running on multiple nodes

Supported to me means we think it should work, and if it doesn't, it's a
high priority bug that will get fixed quickly. Testing is our sanity
check. But it can't be considered that it will catch everything, at
least not before the heat death of the universe.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev