On Mon, Apr 21, 2014 at 10:52:39PM +0000, Daryl Walleck wrote: > I nearly opened a spec for this, but I'd really like to get some > feedback first. One of the challenges I've seen lately for Nova > teams not using KVM or Xen (Ironic and LXC are just a few) is > how to properly run the subset of Compute tests that will run for > their hypervisor or driver. Regexes are what Ironic went with, > but I'm not sure how well that will work long term since it's > very much dependent on naming conventions. The good thing is > that the capabilities for each hypervisor/driver are well defined > (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so > it's just a matter of how to convey that information. I see a > few ways forward from here: > > 1. Expand the compute_features_group config section to include > all Compute actions and make sure all tests that require specific > capabilities have skipIfs or raise a skipException. This options seems > it would require the least work within Tempest, but the size of the > config will continue to grow as more Nova actions are added. > > 2. Create a new decorator class like was done with service tags > that defines what drivers the test does or does not work for, and have > the definitions of the different driver capabilities be referenced by > the decorator. This is nice because it gets rid of the config creep, > but it's also yet another decorator, which may not be desirable. > > I'm going to continue working through both of these possibilities, > but any feedback either solution would be appreciated.
It strikes me that if the test suites have a problem determining support status of APIs with the currently active driver, then the applications using OpenStack will likely suffer the same problem. Given that it'd be desirable to ensure we can solve it in a general way, rather than only consider the test suite needs. On the Nova side of things, I think it would be important to ensure that there is a single "OperationNotSupported" exception that is always raised when the API tries to exercise a feature that is not available with a specific hypervisor driver. If a test case in the test suite ever receives "OperationNotSupported" it could then just mark that test case as "skipped" rather than having exception propagate to result in a "fail". To me the nice thing about such an approach is that you do not need to ever maintain a "matrix" of supported features, as the test suite would just do the right thing whenever the driver is updated. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
