On Tue, Apr 22, 2014 at 07:23:20AM -0400, Sean Dague wrote: > On 04/22/2014 06:55 AM, Daniel P. Berrange wrote: > > On Mon, Apr 21, 2014 at 10:52:39PM +0000, Daryl Walleck wrote: > >> I nearly opened a spec for this, but I'd really like to get some > >> feedback first. One of the challenges I've seen lately for Nova > >> teams not using KVM or Xen (Ironic and LXC are just a few) is > >> how to properly run the subset of Compute tests that will run for > >> their hypervisor or driver. Regexes are what Ironic went with, > >> but I'm not sure how well that will work long term since it's > >> very much dependent on naming conventions. The good thing is > >> that the capabilities for each hypervisor/driver are well defined > >> (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so > >> it's just a matter of how to convey that information. I see a > >> few ways forward from here: > >> > >> 1. Expand the compute_features_group config section to include > >> all Compute actions and make sure all tests that require specific > >> capabilities have skipIfs or raise a skipException. This options seems > >> it would require the least work within Tempest, but the size of the > >> config will continue to grow as more Nova actions are added. > >> > >> 2. Create a new decorator class like was done with service tags > >> that defines what drivers the test does or does not work for, and have > >> the definitions of the different driver capabilities be referenced by > >> the decorator. This is nice because it gets rid of the config creep, > >> but it's also yet another decorator, which may not be desirable. > >> > >> I'm going to continue working through both of these possibilities, > >> but any feedback either solution would be appreciated. > > > > It strikes me that if the test suites have a problem determining > > support status of APIs with the currently active driver, then the > > applications using OpenStack will likely suffer the same problem. > > Given that it'd be desirable to ensure we can solve it in a general > > way, rather than only consider the test suite needs. > > > > On the Nova side of things, I think it would be important to ensure > > that there is a single "OperationNotSupported" exception that is > > always raised when the API tries to exercise a feature that is not > > available with a specific hypervisor driver. If a test case in the > > test suite ever receives "OperationNotSupported" it could then just > > mark that test case as "skipped" rather than having exception propagate > > to result in a "fail". To me the nice thing about such an approach is > > that you do not need to ever maintain a "matrix" of supported features, > > as the test suite would just do the right thing whenever the driver is > > updated. > > Agreed. Though I think we probably want the Nova API to be explicit > about what parts of the API it's ok to throw a Not Supported. Because I > don't think it's a blanket ok. On API endpoints where this is ok, we can > convert not supported to a skip.
Yep, this ties into a discussion I recall elsewhere about specifying exactly which parts of the Nova API are considered mandatory features for drivers to implement. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
