On Fri, Feb 21, 2014 at 10:46:22AM -0500, Sean Dague wrote: > On 02/21/2014 09:45 AM, Daniel P. Berrange wrote: > > On Thu, Feb 20, 2014 at 02:45:03PM -0500, Sean Dague wrote: > >> > >> So I'm one of the first people to utter "if it isn't tested, it's > >> probably broken", however I also think we need to be realistic about the > >> fact that if you did out the permutations of dependencies and config > >> options, we'd have as many test matrix scenarios as grains of sand on > >> the planet. > >> > >> I do think in some ways this is unique to OpenStack, in that our > >> automated testing is head and shoulders above any other Open Source > >> project out there, and most proprietary software systems I've seen. > >> > >> So this is about being pragmatic. In our dependency testing we are > >> actually testing with most recent versions of everything. So I would > >> think that even with libvirt, we should err in that direction. > > > > I'm very much against that, because IME, time & time again across > > all open source projects I've worked on, people silently introduce > > use of features/apis that only exist in newer versions without anyone > > ever noticing until it is too late. > > > >> That being said, we also need to be a little bit careful about taking > >> such a hard line about "supported vs. not" based on only what's in the > >> gate. Because if we did the following things would be listed as > >> unsupported (in increasing level of ridiculousness): > >> > >> * Live migration > >> * Using qpid or zmq > >> * Running on anything other than Ubuntu 12.04 > >> * Running on multiple nodes > >> > >> Supported to me means we think it should work, and if it doesn't, it's a > >> high priority bug that will get fixed quickly. Testing is our sanity > >> check. But it can't be considered that it will catch everything, at > >> least not before the heat death of the universe. > > > > I agree we should be pragmatic here to some extent. We shouldn't aim to > > test every single intermediate version, or every possible permutation of > > versions - just a representative sample. Testing both lowest and highest > > versions is a reasonable sample set IMHO. > > Testing lower bounds is interesting, because of the way pip works. That > being said, if someone wants to take ownership of building that job to > start as a periodic job, I'm happy to point in the right direction. Just > right now, it's a lower priority item than things like Tempest self > testing, Heat actually gating, Neutron running in parallel, Nova API > coverage.
If it would be hard work to do it for python modules, we can at least not remove the existing testing of an old libvirt version - simply add an additional test with newer libvirt. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
