On 04/22/2014 06:55 AM, Daniel P. Berrange wrote: > On Mon, Apr 21, 2014 at 10:52:39PM +0000, Daryl Walleck wrote: >> I nearly opened a spec for this, but I'd really like to get some >> feedback first. One of the challenges I've seen lately for Nova >> teams not using KVM or Xen (Ironic and LXC are just a few) is >> how to properly run the subset of Compute tests that will run for >> their hypervisor or driver. Regexes are what Ironic went with, >> but I'm not sure how well that will work long term since it's >> very much dependent on naming conventions. The good thing is >> that the capabilities for each hypervisor/driver are well defined >> (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so >> it's just a matter of how to convey that information. I see a >> few ways forward from here: >> >> 1. Expand the compute_features_group config section to include >> all Compute actions and make sure all tests that require specific >> capabilities have skipIfs or raise a skipException. This options seems >> it would require the least work within Tempest, but the size of the >> config will continue to grow as more Nova actions are added. >> >> 2. Create a new decorator class like was done with service tags >> that defines what drivers the test does or does not work for, and have >> the definitions of the different driver capabilities be referenced by >> the decorator. This is nice because it gets rid of the config creep, >> but it's also yet another decorator, which may not be desirable. >> >> I'm going to continue working through both of these possibilities, >> but any feedback either solution would be appreciated. > > It strikes me that if the test suites have a problem determining > support status of APIs with the currently active driver, then the > applications using OpenStack will likely suffer the same problem. > Given that it'd be desirable to ensure we can solve it in a general > way, rather than only consider the test suite needs. > > On the Nova side of things, I think it would be important to ensure > that there is a single "OperationNotSupported" exception that is > always raised when the API tries to exercise a feature that is not > available with a specific hypervisor driver. If a test case in the > test suite ever receives "OperationNotSupported" it could then just > mark that test case as "skipped" rather than having exception propagate > to result in a "fail". To me the nice thing about such an approach is > that you do not need to ever maintain a "matrix" of supported features, > as the test suite would just do the right thing whenever the driver is > updated.
Agreed. Though I think we probably want the Nova API to be explicit about what parts of the API it's ok to throw a Not Supported. Because I don't think it's a blanket ok. On API endpoints where this is ok, we can convert not supported to a skip. -Sean -- Sean Dague http://dague.net
signature.asc
Description: OpenPGP digital signature
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev