Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
Boris Renski wrote: > There are a couple of additional things we are working on driver > verification front that, I believe, it is now time to socialize with the > dev community: > > 1. In addition to the scare tactic of deprecating the code for those > that don't test their drivers, we also want to implement a carrot-tactic > of granting a trademark privilege to those that do. Specifically, > storage vendors that have achieved Stage 4 or Thierry's ladder below, > will have the right granted by the openstack foundation to publically > endorse themselves as "OpenStack Verified Block Storage Vendors." I've > spoken to the vast majority of the foundation board members about > this as well as Mark and Jonathan, and, everybody appears to be onboard. > I plan to formally have a foundation board discussion on this topic > during the upcoming March 3rd board meeting and would like to gather the > feedback of the dev community on this. So please feedback away... The end result of the scare tactic will be that only tested drivers are kept *in* OpenStack. The others might be compatible with OpenStack, but they won't be shipped within the main code. So it's quite natural to me if the "Block Storage vendors", "Network plugin vendors" and "Compute hypervisor vendors" that fulfill 3rd-party testing requirements can call their drivers a part of "OpenStack", and have trademark usage rules they can use for public self-endorsement. > 2. As a stepping stone to #1, we are working on implementing an > interactive dashboard (not just for devs, but for OpenStack users as > well) that will display the results of driver tests against trunk > and stable dot releases pulled directly from CI even stream (mock > attached). The way we are thinking of implementing this right now is > next to each of the drivers in > https://wiki.openstack.org/wiki/CinderSupportMatrix, specify if the > compatibility is "self-reported" or "verified." Verified will be a link > to the dashboard for a particular driver kinda like in the mock. This sounds especially useful as we go through steps 1-2 of the ladder, and try to track the extent of our testing for current drivers. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
On 02/13/2014 09:51 AM, Avishay Traeger wrote: "Walter A. Boring IV" wrote on 02/13/2014 06:59:38 PM: What I would do different for the Icehouse release is this: If a driver doesn't pass the certification test by IceHouse RC1, then we have a bug filed against the driver. I would also put a warning message in the log for that driver that it doesn't pass the certification test. I would not remove it from the codebase. Also: if a driver hasn't even run the certification test by RC1, then we mark the driver as uncertified and deprecated in the code and throw an error at driver init time. We can have a option in cinder.conf that says ignore_uncertified_drivers=False. If an admin wants to ignore the error, they set the flag to True, and we let the driver init at next startup. The admin then takes full responsibility for running uncertified code. I think removing the drivers outright is premature for Icehouse, since the certification process is a new thing. For Juno, we remove any drivers that are still marked as uncertified and haven't run the tests. I think the purpose of the tests is to get vendors to actually run their code through tempest and prove to the community that they are willing to show that they are fixing their code. At the end of the day, it better serves the community and Cinder if we have many working drivers. My $0.02, Walt I like this. Make that $0.04 now :) I wrote a bit of code so we had something to discuss if anyone thinks it's a good enough compromise. https://review.openstack.org/#/c/73464/ Walt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
On Thu, Feb 13, 2014 at 10:30 AM, Dean Troyer wrote: > On Thu, Feb 13, 2014 at 4:51 AM, Thierry Carrez > wrote: > >> John Griffith wrote: >> > To add some controversy and keep the original intent of having only >> > known tested and working drivers in the Cinder release, I am going to >> > propose that any driver that has not submitted successful functional >> > testing by RC1 that that driver be removed. I'd at least like to see >> > driver maintainers try... if the test fails a test or two that's >> > something that can be discussed, but it seems that until now most >> > drivers just flat out are not even being tested. > > > +1 > >> >> I think there are multiple stages here. >> >> Stage 0: noone knows if drivers work >> Stage 1: we know the (potentially sad) state of the drivers that are in >> the release >> Stage 2: only drivers that pass tests are added, drivers that don't pass >> tests have a gap analysis and a plan to fix it >> Stage 3: drivers that fail tests are removed before release >> Stage 4: 3rd-party testing rigs must run tests on every change in order >> to stay in tree >> >> At the very minimum you should be at stage 1 for the Icehouse release, >> so I agree with your last paragraph. I'd recommend that you start the >> Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for >> the end of the Juno release. > > > Are any of these drivers new for Icehouse? I think adding broken drivers in > Icehouse is a mistake. The timing WRT Icehouse release schedule is > unfortunate but so is shipping immature drivers that have to be supported > and possibly deprecated. Should new drivers that are lacking have some > not-quite-supported status to allow them to be removed in Juno if not > brought up to par? Or moved into cinder/contrib? Yes, there are a boatload of new drivers being added. > > I don't mean to be picking on Cinder here, this seems to be recurring theme > in OpenStack. I think we benefit from strengthening the precedent that > makes it harder to get things in that are not ready even if the timing is > inconvenient. We're seeing this in project incubation and I think we all > benefit in the end. > > dt > > -- > > Dean Troyer > dtro...@gmail.com > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > I have another tact we can take on this in the interim. I like the contrib dir idea raised by Dean, a hybrid of that and the original proposal is we leave the certification optional, but we publish a certified driver list. We can also use the contrib idea with that as well, so the contrib dir would denote drivers that are not officially certified. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
"Walter A. Boring IV" wrote on 02/13/2014 06:59:38 PM: > What I would do different for the Icehouse release is this: > > If a driver doesn't pass the certification test by IceHouse RC1, then we > have a bug filed > against the driver. I would also put a warning message in the log for > that driver that it > doesn't pass the certification test. I would not remove it from the > codebase. > > Also: > if a driver hasn't even run the certification test by RC1, then we > mark the driver as > uncertified and deprecated in the code and throw an error at driver init > time. > We can have a option in cinder.conf that says > ignore_uncertified_drivers=False. > If an admin wants to ignore the error, they set the flag to True, and we > let the driver init at next startup. > The admin then takes full responsibility for running uncertified code. > >I think removing the drivers outright is premature for Icehouse, > since the certification process is a new thing. > For Juno, we remove any drivers that are still marked as uncertified and > haven't run the tests. > > I think the purpose of the tests is to get vendors to actually run their > code through tempest and > prove to the community that they are willing to show that they are > fixing their code. At the end of the day, > it better serves the community and Cinder if we have many working drivers. > > My $0.02, > Walt I like this. Make that $0.04 now :) Avishay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
On Thu, Feb 13, 2014 at 9:30 AM, Dean Troyer wrote: > Are any of these drivers new for Icehouse? I think adding broken drivers > in Icehouse is a mistake. The timing WRT Icehouse release schedule is > unfortunate but so is shipping immature drivers that have to be supported > and possibly deprecated. Should new drivers that are lacking have some > not-quite-supported status to allow them to be removed in Juno if not > brought up to par? Or moved into cinder/contrib? > > I don't mean to be picking on Cinder here, this seems to be recurring > theme in OpenStack. I think we benefit from strengthening the precedent > that makes it harder to get things in that are not ready even if the timing > is inconvenient. We're seeing this in project incubation and I think we > all benefit in the end. > > dt > Since the cert tests were introduced, new drivers have been required to pass in order to be merged. -Mike Perez ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
On Thu, Feb 13, 2014 at 9:59 AM, Walter A. Boring IV wrote: > On 02/13/2014 02:51 AM, Thierry Carrez wrote: >> >> John Griffith wrote: >>> >>> So we've talked about this a bit and had a number of ideas regarding >>> how to test and show compatibility for third-party drivers in Cinder. >>> This has been an eye opening experience (the number of folks that have >>> NEVER run tempest before, as well as the problems uncovered now that >>> they're trying it). >>> >>> I'm even more convinced now that having vendors run these tests is a >>> good thing and should be required. That being said there's a ton of >>> push back from my proposal to require that results from a successful >>> run of the tempest tests to accompany any new drivers submitted to >>> Cinder. >> >> Could you describe the nature of the pushback ? Is it that the tests are >> too deep and reject valid drivers ? Is it that it's deemed unfair to >> block new drivers while the existing ones aren't better ? Is it that >> it's difficult for them to run those tests and get a report ? Or is it >> because they care more about having their name covered in mainline and >> not so much about having the code working properly ? >> >>> The consensus from the Cinder community for now is that we'll >>> log a bug for each driver after I3, stating that it hasn't passed >>> certification tests. We'll then have a public record showing >>> drivers/vendors that haven't demonstrated functional compatibility, >>> and in order to close those bugs they'll be required to run the tests >>> and submit the results to the bug in Launchpad. >>> >>> So, this seems to be the approach we're taking for Icehouse at least, >>> it's far from ideal IMO, however I think it's still progress and it's >>> definitely exposed some issues with how drivers are currently >>> submitted to Cinder so those are positive things that we can learn >>> from and improve upon in future releases. >>> >>> To add some controversy and keep the original intent of having only >>> known tested and working drivers in the Cinder release, I am going to >>> propose that any driver that has not submitted successful functional >>> testing by RC1 that that driver be removed. I'd at least like to see >>> driver maintainers try... if the test fails a test or two that's >>> something that can be discussed, but it seems that until now most >>> drivers just flat out are not even being tested. >> >> I think there are multiple stages here. >> >> Stage 0: noone knows if drivers work >> Stage 1: we know the (potentially sad) state of the drivers that are in >> the release >> Stage 2: only drivers that pass tests are added, drivers that don't pass >> tests have a gap analysis and a plan to fix it >> Stage 3: drivers that fail tests are removed before release >> Stage 4: 3rd-party testing rigs must run tests on every change in order >> to stay in tree >> >> At the very minimum you should be at stage 1 for the Icehouse release, >> so I agree with your last paragraph. I'd recommend that you start the >> Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for >> the end of the Juno release. >> > I have to agree with Thierry here. I think if we can get drivers to pass > the tests > in the Juno timeframe, then it's fine to remove then during Juno. > I think the idea of having drivers run their code through tempest and work > towards passing all of those tests is a great thing for Cinder and OpenStack > in general. > > What I would do different for the Icehouse release is this: > > If a driver doesn't pass the certification test by IceHouse RC1, then we > have a bug filed > against the driver. I would also put a warning message in the log for that > driver that it > doesn't pass the certification test. I would not remove it from the > codebase. > > Also: >if a driver hasn't even run the certification test by RC1, then we mark > the driver as > uncertified and deprecated in the code and throw an error at driver init > time. > We can have a option in cinder.conf that says > ignore_uncertified_drivers=False. > If an admin wants to ignore the error, they set the flag to True, and we let > the driver init at next startup. > The admin then takes full responsibility for running uncertified code. > > I think removing the drivers outright is premature for Icehouse, since the > certification process is a new thing. > For Juno, we remove any drivers that are still marked as uncertified and > haven't run the tests. > > I think the purpose of the tests is to get vendors to actually run their > code through tempest and > prove to the community that they are willing to show that they are fixing > their code. At the end of the day, > it better serves the community and Cinder if we have many working drivers. > > My $0.02, > Walt > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I'm fine with
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
On Thu, Feb 13, 2014 at 4:51 AM, Thierry Carrez wrote: > John Griffith wrote: > > To add some controversy and keep the original intent of having only > > known tested and working drivers in the Cinder release, I am going to > > propose that any driver that has not submitted successful functional > > testing by RC1 that that driver be removed. I'd at least like to see > > driver maintainers try... if the test fails a test or two that's > > something that can be discussed, but it seems that until now most > > drivers just flat out are not even being tested. > +1 > I think there are multiple stages here. > > Stage 0: noone knows if drivers work > Stage 1: we know the (potentially sad) state of the drivers that are in > the release > Stage 2: only drivers that pass tests are added, drivers that don't pass > tests have a gap analysis and a plan to fix it > Stage 3: drivers that fail tests are removed before release > Stage 4: 3rd-party testing rigs must run tests on every change in order > to stay in tree > > At the very minimum you should be at stage 1 for the Icehouse release, > so I agree with your last paragraph. I'd recommend that you start the > Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for > the end of the Juno release. > Are any of these drivers new for Icehouse? I think adding broken drivers in Icehouse is a mistake. The timing WRT Icehouse release schedule is unfortunate but so is shipping immature drivers that have to be supported and possibly deprecated. Should new drivers that are lacking have some not-quite-supported status to allow them to be removed in Juno if not brought up to par? Or moved into cinder/contrib? I don't mean to be picking on Cinder here, this seems to be recurring theme in OpenStack. I think we benefit from strengthening the precedent that makes it harder to get things in that are not ready even if the timing is inconvenient. We're seeing this in project incubation and I think we all benefit in the end. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
On 02/13/2014 02:51 AM, Thierry Carrez wrote: John Griffith wrote: So we've talked about this a bit and had a number of ideas regarding how to test and show compatibility for third-party drivers in Cinder. This has been an eye opening experience (the number of folks that have NEVER run tempest before, as well as the problems uncovered now that they're trying it). I'm even more convinced now that having vendors run these tests is a good thing and should be required. That being said there's a ton of push back from my proposal to require that results from a successful run of the tempest tests to accompany any new drivers submitted to Cinder. Could you describe the nature of the pushback ? Is it that the tests are too deep and reject valid drivers ? Is it that it's deemed unfair to block new drivers while the existing ones aren't better ? Is it that it's difficult for them to run those tests and get a report ? Or is it because they care more about having their name covered in mainline and not so much about having the code working properly ? The consensus from the Cinder community for now is that we'll log a bug for each driver after I3, stating that it hasn't passed certification tests. We'll then have a public record showing drivers/vendors that haven't demonstrated functional compatibility, and in order to close those bugs they'll be required to run the tests and submit the results to the bug in Launchpad. So, this seems to be the approach we're taking for Icehouse at least, it's far from ideal IMO, however I think it's still progress and it's definitely exposed some issues with how drivers are currently submitted to Cinder so those are positive things that we can learn from and improve upon in future releases. To add some controversy and keep the original intent of having only known tested and working drivers in the Cinder release, I am going to propose that any driver that has not submitted successful functional testing by RC1 that that driver be removed. I'd at least like to see driver maintainers try... if the test fails a test or two that's something that can be discussed, but it seems that until now most drivers just flat out are not even being tested. I think there are multiple stages here. Stage 0: noone knows if drivers work Stage 1: we know the (potentially sad) state of the drivers that are in the release Stage 2: only drivers that pass tests are added, drivers that don't pass tests have a gap analysis and a plan to fix it Stage 3: drivers that fail tests are removed before release Stage 4: 3rd-party testing rigs must run tests on every change in order to stay in tree At the very minimum you should be at stage 1 for the Icehouse release, so I agree with your last paragraph. I'd recommend that you start the Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for the end of the Juno release. I have to agree with Thierry here. I think if we can get drivers to pass the tests in the Juno timeframe, then it's fine to remove then during Juno. I think the idea of having drivers run their code through tempest and work towards passing all of those tests is a great thing for Cinder and OpenStack in general. What I would do different for the Icehouse release is this: If a driver doesn't pass the certification test by IceHouse RC1, then we have a bug filed against the driver. I would also put a warning message in the log for that driver that it doesn't pass the certification test. I would not remove it from the codebase. Also: if a driver hasn't even run the certification test by RC1, then we mark the driver as uncertified and deprecated in the code and throw an error at driver init time. We can have a option in cinder.conf that says ignore_uncertified_drivers=False. If an admin wants to ignore the error, they set the flag to True, and we let the driver init at next startup. The admin then takes full responsibility for running uncertified code. I think removing the drivers outright is premature for Icehouse, since the certification process is a new thing. For Juno, we remove any drivers that are still marked as uncertified and haven't run the tests. I think the purpose of the tests is to get vendors to actually run their code through tempest and prove to the community that they are willing to show that they are fixing their code. At the end of the day, it better serves the community and Cinder if we have many working drivers. My $0.02, Walt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
John Griffith wrote: > So we've talked about this a bit and had a number of ideas regarding > how to test and show compatibility for third-party drivers in Cinder. > This has been an eye opening experience (the number of folks that have > NEVER run tempest before, as well as the problems uncovered now that > they're trying it). > > I'm even more convinced now that having vendors run these tests is a > good thing and should be required. That being said there's a ton of > push back from my proposal to require that results from a successful > run of the tempest tests to accompany any new drivers submitted to > Cinder. Could you describe the nature of the pushback ? Is it that the tests are too deep and reject valid drivers ? Is it that it's deemed unfair to block new drivers while the existing ones aren't better ? Is it that it's difficult for them to run those tests and get a report ? Or is it because they care more about having their name covered in mainline and not so much about having the code working properly ? > The consensus from the Cinder community for now is that we'll > log a bug for each driver after I3, stating that it hasn't passed > certification tests. We'll then have a public record showing > drivers/vendors that haven't demonstrated functional compatibility, > and in order to close those bugs they'll be required to run the tests > and submit the results to the bug in Launchpad. > > So, this seems to be the approach we're taking for Icehouse at least, > it's far from ideal IMO, however I think it's still progress and it's > definitely exposed some issues with how drivers are currently > submitted to Cinder so those are positive things that we can learn > from and improve upon in future releases. > > To add some controversy and keep the original intent of having only > known tested and working drivers in the Cinder release, I am going to > propose that any driver that has not submitted successful functional > testing by RC1 that that driver be removed. I'd at least like to see > driver maintainers try... if the test fails a test or two that's > something that can be discussed, but it seems that until now most > drivers just flat out are not even being tested. I think there are multiple stages here. Stage 0: noone knows if drivers work Stage 1: we know the (potentially sad) state of the drivers that are in the release Stage 2: only drivers that pass tests are added, drivers that don't pass tests have a gap analysis and a plan to fix it Stage 3: drivers that fail tests are removed before release Stage 4: 3rd-party testing rigs must run tests on every change in order to stay in tree At the very minimum you should be at stage 1 for the Icehouse release, so I agree with your last paragraph. I'd recommend that you start the Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for the end of the Juno release. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification
Hey, So we've talked about this a bit and had a number of ideas regarding how to test and show compatibility for third-party drivers in Cinder. This has been an eye opening experience (the number of folks that have NEVER run tempest before, as well as the problems uncovered now that they're trying it). I'm even more convinced now that having vendors run these tests is a good thing and should be required. That being said there's a ton of push back from my proposal to require that results from a successful run of the tempest tests to accompany any new drivers submitted to Cinder. The consensus from the Cinder community for now is that we'll log a bug for each driver after I3, stating that it hasn't passed certification tests. We'll then have a public record showing drivers/vendors that haven't demonstrated functional compatibility, and in order to close those bugs they'll be required to run the tests and submit the results to the bug in Launchpad. So, this seems to be the approach we're taking for Icehouse at least, it's far from ideal IMO, however I think it's still progress and it's definitely exposed some issues with how drivers are currently submitted to Cinder so those are positive things that we can learn from and improve upon in future releases. To add some controversy and keep the original intent of having only known tested and working drivers in the Cinder release, I am going to propose that any driver that has not submitted successful functional testing by RC1 that that driver be removed. I'd at least like to see driver maintainers try... if the test fails a test or two that's something that can be discussed, but it seems that until now most drivers just flat out are not even being tested. Thanks, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
