Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On Tue, 22 Apr 2014 19:46:06 -0400 Sean Dague s...@dague.net wrote: I think the policy about what's allowed to be not implemented or not shouldn't be changing so quickly that it needs to be left to the projects to decide after the fact. +1. I think it is extremely important that we gate on this so we don't accidentally start not supporting a feature and not realise for a long time. If the current situation is very complicated then perhaps a tool could be used to produce information about how things currently look based on where we currently get not implemented errors. But that still needs to be reviewed manually to ensure its reasonable and Tempest should not be trying to adapt dynamically. to feature suport disappearing. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On 04/21/2014 06:52 PM, Daryl Walleck wrote: I nearly opened a spec for this, but I’d really like to get some feedback first. One of the challenges I’ve seen lately for Nova teams not using KVM or Xen (Ironic and LXC are just a few) is how to properly run the subset of Compute tests that will run for their hypervisor or driver. Regexes are what Ironic went with, but I’m not sure how well that will work long term since it’s very much dependent on naming conventions. The good thing is that the capabilities for each hypervisor/driver are well defined (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so it’s just a matter of how to convey that information. I see a few ways forward from here: 1. Expand the compute_features_group config section to include all Compute actions and make sure all tests that require specific capabilities have skipIfs or raise a skipException. This options seems it would require the least work within Tempest, but the size of the config will continue to grow as more Nova actions are added. 2. Create a new decorator class like was done with service tags that defines what drivers the test does or does not work for, and have the definitions of the different driver capabilities be referenced by the decorator. This is nice because it gets rid of the config creep, but it’s also yet another decorator, which may not be desirable. I’m going to continue working through both of these possibilities, but any feedback either solution would be appreciated. Ironic mostly went with regexes for expediency to get something gating before their driver actually implements the requirements for the compute API. Nova API is Nova API, the compute driver should be irrelevant. The part that is optional is specified by extensions (at the granularity level of an extension enable/disable). Creating all the knobs that are optional for extensions is good, and we're definitely not there yet. However if an API behaves differently based on compute driver, that's a problem with that compute driver. I realize today that we're not there yet, but we have to be headed in that direction. The diagnostics API was an instance where this was pretty bad, and meant it was in no way an API, because the client had no idea what data payload it was getting back. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On Mon, Apr 21, 2014 at 10:52:39PM +, Daryl Walleck wrote: I nearly opened a spec for this, but I'd really like to get some feedback first. One of the challenges I've seen lately for Nova teams not using KVM or Xen (Ironic and LXC are just a few) is how to properly run the subset of Compute tests that will run for their hypervisor or driver. Regexes are what Ironic went with, but I'm not sure how well that will work long term since it's very much dependent on naming conventions. The good thing is that the capabilities for each hypervisor/driver are well defined (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so it's just a matter of how to convey that information. I see a few ways forward from here: 1. Expand the compute_features_group config section to include all Compute actions and make sure all tests that require specific capabilities have skipIfs or raise a skipException. This options seems it would require the least work within Tempest, but the size of the config will continue to grow as more Nova actions are added. 2. Create a new decorator class like was done with service tags that defines what drivers the test does or does not work for, and have the definitions of the different driver capabilities be referenced by the decorator. This is nice because it gets rid of the config creep, but it's also yet another decorator, which may not be desirable. I'm going to continue working through both of these possibilities, but any feedback either solution would be appreciated. It strikes me that if the test suites have a problem determining support status of APIs with the currently active driver, then the applications using OpenStack will likely suffer the same problem. Given that it'd be desirable to ensure we can solve it in a general way, rather than only consider the test suite needs. On the Nova side of things, I think it would be important to ensure that there is a single OperationNotSupported exception that is always raised when the API tries to exercise a feature that is not available with a specific hypervisor driver. If a test case in the test suite ever receives OperationNotSupported it could then just mark that test case as skipped rather than having exception propagate to result in a fail. To me the nice thing about such an approach is that you do not need to ever maintain a matrix of supported features, as the test suite would just do the right thing whenever the driver is updated. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On 04/22/2014 06:55 AM, Daniel P. Berrange wrote: On Mon, Apr 21, 2014 at 10:52:39PM +, Daryl Walleck wrote: I nearly opened a spec for this, but I'd really like to get some feedback first. One of the challenges I've seen lately for Nova teams not using KVM or Xen (Ironic and LXC are just a few) is how to properly run the subset of Compute tests that will run for their hypervisor or driver. Regexes are what Ironic went with, but I'm not sure how well that will work long term since it's very much dependent on naming conventions. The good thing is that the capabilities for each hypervisor/driver are well defined (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so it's just a matter of how to convey that information. I see a few ways forward from here: 1. Expand the compute_features_group config section to include all Compute actions and make sure all tests that require specific capabilities have skipIfs or raise a skipException. This options seems it would require the least work within Tempest, but the size of the config will continue to grow as more Nova actions are added. 2. Create a new decorator class like was done with service tags that defines what drivers the test does or does not work for, and have the definitions of the different driver capabilities be referenced by the decorator. This is nice because it gets rid of the config creep, but it's also yet another decorator, which may not be desirable. I'm going to continue working through both of these possibilities, but any feedback either solution would be appreciated. It strikes me that if the test suites have a problem determining support status of APIs with the currently active driver, then the applications using OpenStack will likely suffer the same problem. Given that it'd be desirable to ensure we can solve it in a general way, rather than only consider the test suite needs. On the Nova side of things, I think it would be important to ensure that there is a single OperationNotSupported exception that is always raised when the API tries to exercise a feature that is not available with a specific hypervisor driver. If a test case in the test suite ever receives OperationNotSupported it could then just mark that test case as skipped rather than having exception propagate to result in a fail. To me the nice thing about such an approach is that you do not need to ever maintain a matrix of supported features, as the test suite would just do the right thing whenever the driver is updated. Agreed. Though I think we probably want the Nova API to be explicit about what parts of the API it's ok to throw a Not Supported. Because I don't think it's a blanket ok. On API endpoints where this is ok, we can convert not supported to a skip. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On 04/22/2014 07:23 AM, Sean Dague wrote: On 04/22/2014 06:55 AM, Daniel P. Berrange wrote: On Mon, Apr 21, 2014 at 10:52:39PM +, Daryl Walleck wrote: I nearly opened a spec for this, but I'd really like to get some feedback first. One of the challenges I've seen lately for Nova teams not using KVM or Xen (Ironic and LXC are just a few) is how to properly run the subset of Compute tests that will run for their hypervisor or driver. Regexes are what Ironic went with, but I'm not sure how well that will work long term since it's very much dependent on naming conventions. The good thing is that the capabilities for each hypervisor/driver are well defined (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so it's just a matter of how to convey that information. I see a few ways forward from here: 1. Expand the compute_features_group config section to include all Compute actions and make sure all tests that require specific capabilities have skipIfs or raise a skipException. This options seems it would require the least work within Tempest, but the size of the config will continue to grow as more Nova actions are added. 2. Create a new decorator class like was done with service tags that defines what drivers the test does or does not work for, and have the definitions of the different driver capabilities be referenced by the decorator. This is nice because it gets rid of the config creep, but it's also yet another decorator, which may not be desirable. I'm going to continue working through both of these possibilities, but any feedback either solution would be appreciated. It strikes me that if the test suites have a problem determining support status of APIs with the currently active driver, then the applications using OpenStack will likely suffer the same problem. Given that it'd be desirable to ensure we can solve it in a general way, rather than only consider the test suite needs. On the Nova side of things, I think it would be important to ensure that there is a single OperationNotSupported exception that is always raised when the API tries to exercise a feature that is not available with a specific hypervisor driver. If a test case in the test suite ever receives OperationNotSupported it could then just mark that test case as skipped rather than having exception propagate to result in a fail. To me the nice thing about such an approach is that you do not need to ever maintain a matrix of supported features, as the test suite would just do the right thing whenever the driver is updated. Agreed. Though I think we probably want the Nova API to be explicit about what parts of the API it's ok to throw a Not Supported. Because I don't think it's a blanket ok. On API endpoints where this is ok, we can convert not supported to a skip. Definitely agreed with Dan's points here. We already raise NotImplementedError for this in the code. Assuming it makes it all the way back up to the API, it should be converted to a 501 Not Implemented response. It doesn't look like this is handled in a general way in the API code. Each API extension that supports this is handling it manually. However, it is used a lot. A quick grep in nova/api shows 50 cases of raising webob.exc.HTTPNotImplemented. It would be nice to identify the specific cases where this isn't working as expected. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On Tue, Apr 22, 2014 at 07:23:20AM -0400, Sean Dague wrote: On 04/22/2014 06:55 AM, Daniel P. Berrange wrote: On Mon, Apr 21, 2014 at 10:52:39PM +, Daryl Walleck wrote: I nearly opened a spec for this, but I'd really like to get some feedback first. One of the challenges I've seen lately for Nova teams not using KVM or Xen (Ironic and LXC are just a few) is how to properly run the subset of Compute tests that will run for their hypervisor or driver. Regexes are what Ironic went with, but I'm not sure how well that will work long term since it's very much dependent on naming conventions. The good thing is that the capabilities for each hypervisor/driver are well defined (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so it's just a matter of how to convey that information. I see a few ways forward from here: 1. Expand the compute_features_group config section to include all Compute actions and make sure all tests that require specific capabilities have skipIfs or raise a skipException. This options seems it would require the least work within Tempest, but the size of the config will continue to grow as more Nova actions are added. 2. Create a new decorator class like was done with service tags that defines what drivers the test does or does not work for, and have the definitions of the different driver capabilities be referenced by the decorator. This is nice because it gets rid of the config creep, but it's also yet another decorator, which may not be desirable. I'm going to continue working through both of these possibilities, but any feedback either solution would be appreciated. It strikes me that if the test suites have a problem determining support status of APIs with the currently active driver, then the applications using OpenStack will likely suffer the same problem. Given that it'd be desirable to ensure we can solve it in a general way, rather than only consider the test suite needs. On the Nova side of things, I think it would be important to ensure that there is a single OperationNotSupported exception that is always raised when the API tries to exercise a feature that is not available with a specific hypervisor driver. If a test case in the test suite ever receives OperationNotSupported it could then just mark that test case as skipped rather than having exception propagate to result in a fail. To me the nice thing about such an approach is that you do not need to ever maintain a matrix of supported features, as the test suite would just do the right thing whenever the driver is updated. Agreed. Though I think we probably want the Nova API to be explicit about what parts of the API it's ok to throw a Not Supported. Because I don't think it's a blanket ok. On API endpoints where this is ok, we can convert not supported to a skip. Yep, this ties into a discussion I recall elsewhere about specifying exactly which parts of the Nova API are considered mandatory features for drivers to implement. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On 04/22/2014 08:35 AM, Daniel P. Berrange wrote: On Tue, Apr 22, 2014 at 07:23:20AM -0400, Sean Dague wrote: On 04/22/2014 06:55 AM, Daniel P. Berrange wrote: On Mon, Apr 21, 2014 at 10:52:39PM +, Daryl Walleck wrote: I nearly opened a spec for this, but I'd really like to get some feedback first. One of the challenges I've seen lately for Nova teams not using KVM or Xen (Ironic and LXC are just a few) is how to properly run the subset of Compute tests that will run for their hypervisor or driver. Regexes are what Ironic went with, but I'm not sure how well that will work long term since it's very much dependent on naming conventions. The good thing is that the capabilities for each hypervisor/driver are well defined (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so it's just a matter of how to convey that information. I see a few ways forward from here: 1. Expand the compute_features_group config section to include all Compute actions and make sure all tests that require specific capabilities have skipIfs or raise a skipException. This options seems it would require the least work within Tempest, but the size of the config will continue to grow as more Nova actions are added. 2. Create a new decorator class like was done with service tags that defines what drivers the test does or does not work for, and have the definitions of the different driver capabilities be referenced by the decorator. This is nice because it gets rid of the config creep, but it's also yet another decorator, which may not be desirable. I'm going to continue working through both of these possibilities, but any feedback either solution would be appreciated. It strikes me that if the test suites have a problem determining support status of APIs with the currently active driver, then the applications using OpenStack will likely suffer the same problem. Given that it'd be desirable to ensure we can solve it in a general way, rather than only consider the test suite needs. On the Nova side of things, I think it would be important to ensure that there is a single OperationNotSupported exception that is always raised when the API tries to exercise a feature that is not available with a specific hypervisor driver. If a test case in the test suite ever receives OperationNotSupported it could then just mark that test case as skipped rather than having exception propagate to result in a fail. To me the nice thing about such an approach is that you do not need to ever maintain a matrix of supported features, as the test suite would just do the right thing whenever the driver is updated. Agreed. Though I think we probably want the Nova API to be explicit about what parts of the API it's ok to throw a Not Supported. Because I don't think it's a blanket ok. On API endpoints where this is ok, we can convert not supported to a skip. Yep, this ties into a discussion I recall elsewhere about specifying exactly which parts of the Nova API are considered mandatory features for drivers to implement. I put that in as a proposal to discuss at the design summit. http://summit.openstack.org/cfp/details/55 -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
I see your point. I had assume that the Hypervisor support matrix was something that was blessed, but that's what I get for assuming. :-) Once there's a list of required operations, I think that would become more clear. That said, there are server actions right now that even KVM doesn't implement (Change Password), which was one of the reasons I first implemented feature flags in Tempest. There's also some actions that don't necessarily make sense for certain drivers (resizing a bare metal server). If we make no assumptions about the underlying driver, shouldn't these capability flags go away altogether, or stay for convenience purposes? Either way, I'll hold off on this idea until after the summit discussion. Thanks for the feedback! Daryl -Original Message- From: Sean Dague [mailto:s...@dague.net] Sent: Tuesday, April 22, 2014 5:40 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor On 04/21/2014 06:52 PM, Daryl Walleck wrote: I nearly opened a spec for this, but I’d really like to get some feedback first. One of the challenges I’ve seen lately for Nova teams not using KVM or Xen (Ironic and LXC are just a few) is how to properly run the subset of Compute tests that will run for their hypervisor or driver. Regexes are what Ironic went with, but I’m not sure how well that will work long term since it’s very much dependent on naming conventions. The good thing is that the capabilities for each hypervisor/driver are well defined (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so it’s just a matter of how to convey that information. I see a few ways forward from here: 1. Expand the compute_features_group config section to include all Compute actions and make sure all tests that require specific capabilities have skipIfs or raise a skipException. This options seems it would require the least work within Tempest, but the size of the config will continue to grow as more Nova actions are added. 2. Create a new decorator class like was done with service tags that defines what drivers the test does or does not work for, and have the definitions of the different driver capabilities be referenced by the decorator. This is nice because it gets rid of the config creep, but it’s also yet another decorator, which may not be desirable. I’m going to continue working through both of these possibilities, but any feedback either solution would be appreciated. Ironic mostly went with regexes for expediency to get something gating before their driver actually implements the requirements for the compute API. Nova API is Nova API, the compute driver should be irrelevant. The part that is optional is specified by extensions (at the granularity level of an extension enable/disable). Creating all the knobs that are optional for extensions is good, and we're definitely not there yet. However if an API behaves differently based on compute driver, that's a problem with that compute driver. I realize today that we're not there yet, but we have to be headed in that direction. The diagnostics API was an instance where this was pretty bad, and meant it was in no way an API, because the client had no idea what data payload it was getting back. -Sean -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On Tue, Apr 22, 2014 at 4:23 AM, Sean Dague s...@dague.net wrote: Agreed. Though I think we probably want the Nova API to be explicit about what parts of the API it's ok to throw a Not Supported. Because I don't think it's a blanket ok. On API endpoints where this is ok, we can convert not supported to a skip. -Sean I'd favor going even further and let any such exception convert to a skip, at least in the main test suite. Keep Tempest a point-and-shoot suite that can be pointed at any cloud and do the right thing. We can add another test or utility (perhaps in tools/?) to interpret results and attach meaning to them WRT skips/fails validated against individual projects' current notion of what is mandatory. A list of these tests must pass against any driver instead of a driver feature matrix. This would allow such policies to easily change over time outside of the actual test code. -Adam ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On 04/22/2014 07:33 PM, Adam Gandelman wrote: On Tue, Apr 22, 2014 at 4:23 AM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: Agreed. Though I think we probably want the Nova API to be explicit about what parts of the API it's ok to throw a Not Supported. Because I don't think it's a blanket ok. On API endpoints where this is ok, we can convert not supported to a skip. -Sean I'd favor going even further and let any such exception convert to a skip, at least in the main test suite. Keep Tempest a point-and-shoot suite that can be pointed at any cloud and do the right thing. We can add another test or utility (perhaps in tools/?) to interpret results and attach meaning to them WRT skips/fails validated against individual projects' current notion of what is mandatory. A list of these tests must pass against any driver instead of a driver feature matrix. This would allow such policies to easily change over time outside of the actual test code. I think the policy about what's allowed to be not implemented or not shouldn't be changing so quickly that it needs to be left to the projects to decide after the fact. It also really needs to be documented in our API. If a not implemented is allowable for a particular API call, the client needs to know that's valid response, in advance. There are a lot of challenges with interpreting things after the fact, especially if your path was cut short because things just started skipping. And past experiences with auto skipping mostly meant that project breaks got in (which is why we don't do that any more). -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
I nearly opened a spec for this, but I'd really like to get some feedback first. One of the challenges I've seen lately for Nova teams not using KVM or Xen (Ironic and LXC are just a few) is how to properly run the subset of Compute tests that will run for their hypervisor or driver. Regexes are what Ironic went with, but I'm not sure how well that will work long term since it's very much dependent on naming conventions. The good thing is that the capabilities for each hypervisor/driver are well defined (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so it's just a matter of how to convey that information. I see a few ways forward from here: 1. Expand the compute_features_group config section to include all Compute actions and make sure all tests that require specific capabilities have skipIfs or raise a skipException. This options seems it would require the least work within Tempest, but the size of the config will continue to grow as more Nova actions are added. 2. Create a new decorator class like was done with service tags that defines what drivers the test does or does not work for, and have the definitions of the different driver capabilities be referenced by the decorator. This is nice because it gets rid of the config creep, but it's also yet another decorator, which may not be desirable. I'm going to continue working through both of these possibilities, but any feedback either solution would be appreciated. Daryl ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Selecting Compute tests by driver/hypervisor
On Mon, Apr 21, 2014 at 10:52:39PM +, Daryl Walleck wrote: I nearly opened a spec for this, but I'd really like to get some feedback first. One of the challenges I've seen lately for Nova teams not using KVM or Xen (Ironic and LXC are just a few) is how to properly run the subset of Compute tests that will run for their hypervisor or driver. Regexes are what Ironic went with, but I'm not sure how well that will work long term since it's very much dependent on naming conventions. The good thing is that the capabilities for each hypervisor/driver are well defined (https://wiki.openstack.org/wiki/HypervisorSupportMatrix), so it's just a matter of how to convey that information. I see a few ways forward from here: If you're willing to drive this effort then please submit a spec review for it. These kind of discussions are perfect for doing in a spec review. 1. Expand the compute_features_group config section to include all Compute actions and make sure all tests that require specific capabilities have skipIfs or raise a skipException. This options seems it would require the least work within Tempest, but the size of the config will continue to grow as more Nova actions are added. This is the only path forward I can see for this issue. We're going to have to get better about consistently skipping based on config feature flags. This is also a big part of what is required moving forward with the branchless tempest work. [1] The config growth is really unavoidable considering the myriad of configuration possibilities and new features we get in OpenStack. We will just have to come up with some new ways and tooling to deal with the rapid growth in config file size. I honestly think the best approach for doing this is probably abstracting away individual conditional skip calls and make a unified feature skip decorator. Similar to what we already do for the requires_ext() decorator. That way we can have consistent logic and style around how to annotate what a test requires. 2. Create a new decorator class like was done with service tags that defines what drivers the test does or does not work for, and have the definitions of the different driver capabilities be referenced by the decorator. This is nice because it gets rid of the config creep, but it's also yet another decorator, which may not be desirable. The problem with this approach is that tempest isn't supposed to care about what is underneath the api layer. You should just tell it what the OpenStack deployment is capable of and let it go. If a driver or any other backend configuration doesn't support certain functionality then that should be an explicit knob in the config file. Also, another harm with this approach is that in effect we end up codifying in tempest, through decorators which seems really messy, the set of features that should work for a particular configuration/driver; which feels way outside the scope of tempest. I'm going to continue working through both of these possibilities, but any feedback either solution would be appreciated. Thanks, Matt Treinish [1] https://github.com/openstack/qa-specs/blob/master/specs/branchless-tempest.rst ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
