Re: [openstack-dev] [OpenStack][Cinder] Driver qualification
On 25/07/13 18:44 -0600, John Griffith wrote: Hey Everyone, Something I've been kicking around for quite a while now but never really been able to get around to is the idea of requiring that drivers in Cinder run a qualification test and submit results prior to introduction in to Cinder. FWIW, big +1. This is something we'll face in Marconi as well. We've been figuring out a set of drivers that should live in the code base and we'd like to keep all other drivers out of it. However, this means we need to have a way to: 1) Know what drivers exist out there - at least let implementers have them listed somewhere) 2) Make sure those drivers are compliant before being listed anywhere in the project. We have some base unit test that test all Marconi's resources lifecycle and we'll, most probably, do the same for functional tests. I'd love to see this happening and to help making it happen, feel free to ping me. Cheers, FF -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack][Cinder] Driver qualification
On 30 July 2013 21:36, Mark Washenberger mark.washenber...@markwash.net wrote: - I'd love to use this as a way to OpenStack-validate plugins that live outside of the project--and then kick most plugins out of the project to some other semi-canonical area. I don't think I'm alone within the cinder team in seeing significant benefit to keeping plugins within the cinder code base - they benefit not just from the eyes of reviewers, but also from the mindshare of developers. Since we know (or can go and look) at how various plugins are using (or abusing) the interfaces we provide, we can keep that in mind when making changes, and fix up obvious cases. Once things are in a separate repo, you get all sorts of painful version skew type problems, and everybody has less insight into how the code behaves as a whole. - I don't think the test infrastructure should depend wholly on devstack--while that makes a ton of sense for plugins that depend on other OpenStack services working, I think its pretty crufty that we have plugins that depend on other openstack services. I'd rather not to weigh the good down with the bad. The nature of some of the cinder drivers is that they require some other services - e.g. some drivers snapshot to glace, all drivers (should be able to) backup to glance, some drivers are tenant aware, etc. Devstack is as good a 'standard platform' as we've got at the moment... You could allow the validation suite to be run against any openstack install, but then automated collection of the cinder.conf and similar niceties becomes much harder. +1 for John's approach ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack][Cinder] Driver qualification
On Tue, Jul 30, 2013 at 11:47 AM, Walter A. Boring IV walter.bor...@hp.comwrote: So how frequently would this be required and when do the results need to be provided? To start I'd like to propose any new driver, and then each driver upon milestone release, or at the very least each cycle. I generally think this is a good idea, but as well all know that the project seems to be in flux at times around the milestones. Especially around G3 when we changed the way the CONF object was accessed, almost all drivers failed to work right after G3 when cinder.conf was configured with a multi-backend setup. It took a week or so after G3 before everyone IMO this is exactly why something like this would actually be more beneficial. was able to go through drivers and clean them up and get them to work in this scenario. I'm not trying to be a wet blanket on this idea, but I think we just have to be careful. I have no problem running my drivers through this and providing the results. What happens when something changes in cinder that causes drivers to fail and the test runs fail? How long We should be extra careful to keep this sort of thing from happening, but again without doing any sort of public testing/results here we run the risk of nobody knowing it's broken until a customer encounters it. I'm also not quite sure I see the issue here, I mean if something in the cinder base code breaks a driver it's still going to be broken, the only difference is that we'll actually know that it's broken. does a maintainer get to fix the issue before X happens as a result? Does this happen every milestone I-1, I-2, I-3? What happens if a maintainer can't do it for every milestone for whatever reason? I don't think we've arrived at a point where we say X happens yet. To start I'd view this as logging a bug if it doesn't pass. Ultimately it would probably be interesting to consider things like removing a driver but there would have to be some process setup regarding how we try and address the issue before doing something drastic. Just playing a bit of devil's advocate here. I do like the idea though, just depends on the rules setup and how it all applies when things don't go well for a particular driver. Sure, that's fine and I think you bring up a good point. The idea here is NOT to make things difficult or to try and keep drivers out etc, the idea is to release a better product. There are a number of folks that have drivers that are believed to work but since there's no formal testing/integration it's just an assumption in the community. This proposal would at least make it public information regarding whether a driver actually works or not. I mean really, I don't think this is asking too much considering we require any patch to the projects in OpenStack to run these tests against the LVM driver to make sure things work. This really isn't any different, except we don't *require* it for every check in. We just do checks to make sure drivers are actually doing what we expect and end-users aren't surprised to find their driver doesn't actually work. Cheers, Walt Hey Everyone, Something I've been kicking around for quite a while now but never really been able to get around to is the idea of requiring that drivers in Cinder run a qualification test and submit results prior to introduction in to Cinder. To elaborate a bit, the idea could start as something really simple like the following: 1. We'd add a functional_qual option/script to devstack 2. Driver maintainer runs this script to setup devstack and configure it to use their backend device on their own system. 3. Script does the usual devstack install/configure and runs the volume pieces of the Tempest gate tests. 4. Grabs output and checksums of the directories in the devstack and /opt/stack directories, bundles up the results for submission 5. Maintainer submits results So why would we do this you ask? Cinder is pretty heavy on the third party driver plugin model which is fantastic. On the other hand while there are a lot of folks who do great reviews that catch things like syntax or logic errors in the code, and unit tests do a reasonable job of exercising the code it's difficult for folks to truly verify these devices all work. I think it would be a very useful tool for initial introduction of a new driver and even perhaps some sort of check that's run and submitted again prior to milestone releases. This would also drive some more activity and contribution in to Tempest with respect to getting folks like myself motivated to contribute more tests (particularly in terms of new functionality) in to Tempest. I'd be interested to hear if folks have any interest or strong opinions on this (positive or otherwise). I know that some vendors like RedHat have this sort of thing in place for certifications, and to be honest that observation is something that caused me
Re: [openstack-dev] [OpenStack][Cinder] Driver qualification
John Griffith john.griff...@solidfire.com wrote on 07/26/2013 03:44:12 AM: snip I think it would be a very useful tool for initial introduction of a new driver and even perhaps some sort of check that's run and submitted again prior to milestone releases. snip +1. Do you see this happening for Havana? Or should this be a summit topic? Thanks, Avishay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [OpenStack][Cinder] Driver qualification
Hey Everyone, Something I've been kicking around for quite a while now but never really been able to get around to is the idea of requiring that drivers in Cinder run a qualification test and submit results prior to introduction in to Cinder. To elaborate a bit, the idea could start as something really simple like the following: 1. We'd add a functional_qual option/script to devstack 2. Driver maintainer runs this script to setup devstack and configure it to use their backend device on their own system. 3. Script does the usual devstack install/configure and runs the volume pieces of the Tempest gate tests. 4. Grabs output and checksums of the directories in the devstack and /opt/stack directories, bundles up the results for submission 5. Maintainer submits results So why would we do this you ask? Cinder is pretty heavy on the third party driver plugin model which is fantastic. On the other hand while there are a lot of folks who do great reviews that catch things like syntax or logic errors in the code, and unit tests do a reasonable job of exercising the code it's difficult for folks to truly verify these devices all work. I think it would be a very useful tool for initial introduction of a new driver and even perhaps some sort of check that's run and submitted again prior to milestone releases. This would also drive some more activity and contribution in to Tempest with respect to getting folks like myself motivated to contribute more tests (particularly in terms of new functionality) in to Tempest. I'd be interested to hear if folks have any interest or strong opinions on this (positive or otherwise). I know that some vendors like RedHat have this sort of thing in place for certifications, and to be honest that observation is something that caused me to start thinking about this again. There are a lot of gaps here regarding how the submission process would look, but we could start relatively simple and grow from there if it's valuable or just abandon the idea if it proves to be unpopular and a waste of time. Anyway, I'd love to get feed-back from folks and see what they think. Thanks, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack][Cinder] Driver qualification
On 07/25/2013 08:44 PM, John Griffith wrote: Hey Everyone, Something I've been kicking around for quite a while now but never really been able to get around to is the idea of requiring that drivers in Cinder run a qualification test and submit results prior to introduction in to Cinder. In general, a big +1 from me. I've been thinking about similar things for nova. I think it's fair to set some requirements that ensure a base level of functionality, quality, and a reasonably consistent user experience. I've been thinking along these lines for Nova, as well. I also want to require ongoing testing. We have a wide range of functional test coverage of the existing compute drivers [1]. I'd like to only accept drivers if there is some sort of CI running against it with results that are publicly available (Group A or B in [1]). I posted about this a bit before [2]. I really like the other thing Cinder has started doing, which is requiring a base feature set for all drivers [3]. I'd like to do the same for Nova, but haven't come up with a formalized proposal yet. [1] https://wiki.openstack.org/wiki/HypervisorSupportMatrix [2] http://lists.openstack.org/pipermail/openstack-dev/2013-July/011260.html [3] http://lists.openstack.org/pipermail/openstack-dev/2013-July/012119.html -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack][Cinder] Driver qualification
100% agree, its hard to handle these 3rd party type of drivers but I think we need to find out a way that will test it in a way that doesn't require having said 3rd party gear directly available. Could it be possible to have CI gating be blocked/tested by individual subfolders of cinder. For example when the solidfire driver is modified, this would cause a 'trigger' to say solidfire (via some API) that solidfire can respond with back saying said commit works. Not sure if that’s feasible, but it does seem to be a similar situation in nova, neturon, cinder as more and more 3rd party 'driver-like' code appears. From: John Griffith john.griff...@solidfire.commailto:john.griff...@solidfire.com Reply-To: OpenStack Development Mailing List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Thursday, July 25, 2013 5:44 PM To: OpenStack Development Mailing List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [OpenStack][Cinder] Driver qualification Hey Everyone, Something I've been kicking around for quite a while now but never really been able to get around to is the idea of requiring that drivers in Cinder run a qualification test and submit results prior to introduction in to Cinder. To elaborate a bit, the idea could start as something really simple like the following: 1. We'd add a functional_qual option/script to devstack 2. Driver maintainer runs this script to setup devstack and configure it to use their backend device on their own system. 3. Script does the usual devstack install/configure and runs the volume pieces of the Tempest gate tests. 4. Grabs output and checksums of the directories in the devstack and /opt/stack directories, bundles up the results for submission 5. Maintainer submits results So why would we do this you ask? Cinder is pretty heavy on the third party driver plugin model which is fantastic. On the other hand while there are a lot of folks who do great reviews that catch things like syntax or logic errors in the code, and unit tests do a reasonable job of exercising the code it's difficult for folks to truly verify these devices all work. I think it would be a very useful tool for initial introduction of a new driver and even perhaps some sort of check that's run and submitted again prior to milestone releases. This would also drive some more activity and contribution in to Tempest with respect to getting folks like myself motivated to contribute more tests (particularly in terms of new functionality) in to Tempest. I'd be interested to hear if folks have any interest or strong opinions on this (positive or otherwise). I know that some vendors like RedHat have this sort of thing in place for certifications, and to be honest that observation is something that caused me to start thinking about this again. There are a lot of gaps here regarding how the submission process would look, but we could start relatively simple and grow from there if it's valuable or just abandon the idea if it proves to be unpopular and a waste of time. Anyway, I'd love to get feed-back from folks and see what they think. Thanks, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack][Cinder] Driver qualification
Great idea and 100% agree. It'd be even better if maintainer can publish functional test results using their own back-ends on a regular basis (weekly/bi-weekly test report) to 'openstack-dev' mailing list. On Fri, Jul 26, 2013 at 9:05 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: 100% agree, its hard to handle these 3rd party type of drivers but I think we need to find out a way that will test it in a way that doesn't require having said 3rd party gear directly available. Could it be possible to have CI gating be blocked/tested by individual subfolders of cinder. For example when the solidfire driver is modified, this would cause a 'trigger' to say solidfire (via some API) that solidfire can respond with back saying said commit works. Not sure if that’s feasible, but it does seem to be a similar situation in nova, neturon, cinder as more and more 3rd party 'driver-like' code appears. From: John Griffith john.griff...@solidfire.com Reply-To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Date: Thursday, July 25, 2013 5:44 PM To: OpenStack Development Mailing List openstack-dev@lists.openstack.org Subject: [openstack-dev] [OpenStack][Cinder] Driver qualification Hey Everyone, Something I've been kicking around for quite a while now but never really been able to get around to is the idea of requiring that drivers in Cinder run a qualification test and submit results prior to introduction in to Cinder. To elaborate a bit, the idea could start as something really simple like the following: 1. We'd add a functional_qual option/script to devstack 2. Driver maintainer runs this script to setup devstack and configure it to use their backend device on their own system. 3. Script does the usual devstack install/configure and runs the volume pieces of the Tempest gate tests. 4. Grabs output and checksums of the directories in the devstack and /opt/stack directories, bundles up the results for submission 5. Maintainer submits results So why would we do this you ask? Cinder is pretty heavy on the third party driver plugin model which is fantastic. On the other hand while there are a lot of folks who do great reviews that catch things like syntax or logic errors in the code, and unit tests do a reasonable job of exercising the code it's difficult for folks to truly verify these devices all work. I think it would be a very useful tool for initial introduction of a new driver and even perhaps some sort of check that's run and submitted again prior to milestone releases. This would also drive some more activity and contribution in to Tempest with respect to getting folks like myself motivated to contribute more tests (particularly in terms of new functionality) in to Tempest. I'd be interested to hear if folks have any interest or strong opinions on this (positive or otherwise). I know that some vendors like RedHat have this sort of thing in place for certifications, and to be honest that observation is something that caused me to start thinking about this again. There are a lot of gaps here regarding how the submission process would look, but we could start relatively simple and grow from there if it's valuable or just abandon the idea if it proves to be unpopular and a waste of time. Anyway, I'd love to get feed-back from folks and see what they think. Thanks, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Regards Huang Zhiteng ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack][Cinder] Driver qualification
On Thu, Jul 25, 2013 at 7:37 PM, yang, xing xing.y...@emc.com wrote: +1. I like this idea. With this qualification test, is each driver still required to have its own unit test? Keep in mind this is just a proposal that I wanted to get feed-back on, I'll likely submit something more concrete shortly. That being said, no this would NOT eliminate or preclude any current process or requirements. The idea here is to augment the existing process and provide some method of visible functional testing compliance. Thanks, Xing On Jul 25, 2013, at 8:46 PM, John Griffith john.griff...@solidfire.com wrote: Hey Everyone, Something I've been kicking around for quite a while now but never really been able to get around to is the idea of requiring that drivers in Cinder run a qualification test and submit results prior to introduction in to Cinder. To elaborate a bit, the idea could start as something really simple like the following: 1. We'd add a functional_qual option/script to devstack 2. Driver maintainer runs this script to setup devstack and configure it to use their backend device on their own system. 3. Script does the usual devstack install/configure and runs the volume pieces of the Tempest gate tests. 4. Grabs output and checksums of the directories in the devstack and /opt/stack directories, bundles up the results for submission 5. Maintainer submits results So why would we do this you ask? Cinder is pretty heavy on the third party driver plugin model which is fantastic. On the other hand while there are a lot of folks who do great reviews that catch things like syntax or logic errors in the code, and unit tests do a reasonable job of exercising the code it's difficult for folks to truly verify these devices all work. I think it would be a very useful tool for initial introduction of a new driver and even perhaps some sort of check that's run and submitted again prior to milestone releases. This would also drive some more activity and contribution in to Tempest with respect to getting folks like myself motivated to contribute more tests (particularly in terms of new functionality) in to Tempest. I'd be interested to hear if folks have any interest or strong opinions on this (positive or otherwise). I know that some vendors like RedHat have this sort of thing in place for certifications, and to be honest that observation is something that caused me to start thinking about this again. There are a lot of gaps here regarding how the submission process would look, but we could start relatively simple and grow from there if it's valuable or just abandon the idea if it proves to be unpopular and a waste of time. Anyway, I'd love to get feed-back from folks and see what they think. Thanks, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
