Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-24 Thread Thierry Carrez
Boris Renski wrote:
> There are a couple of additional things we are working on driver
> verification front that, I believe, it is now time to socialize with the
> dev community:
> 
> 1. In addition to the scare tactic of deprecating the code for those
> that don't test their drivers, we also want to implement a carrot-tactic
> of granting a trademark privilege to those that do. Specifically,
> storage vendors that have achieved Stage 4 or Thierry's ladder below,
> will have the right granted by the openstack foundation to publically
> endorse themselves as "OpenStack Verified Block Storage Vendors." I've
> spoken to the vast majority of the foundation board members about
> this as well as Mark and Jonathan, and, everybody appears to be onboard.
> I plan to formally have a foundation board discussion on this topic
> during the upcoming March 3rd board meeting and would like to gather the
> feedback of the dev community on this. So please feedback away...

The end result of the scare tactic will be that only tested drivers are
kept *in* OpenStack. The others might be compatible with OpenStack, but
they won't be shipped within the main code. So it's quite natural to me
if the "Block Storage vendors", "Network plugin vendors" and "Compute
hypervisor vendors" that fulfill 3rd-party testing requirements can call
their drivers a part of "OpenStack", and have trademark usage rules they
can use for public self-endorsement.

> 2. As a stepping stone to #1, we are working on implementing an
> interactive dashboard (not just for devs, but for OpenStack users as
> well) that will display the results of driver tests against trunk
> and stable dot releases pulled directly from CI even stream (mock
> attached). The way we are thinking of implementing this right now is
> next to each of the drivers in
> https://wiki.openstack.org/wiki/CinderSupportMatrix, specify if the
> compatibility is "self-reported" or "verified." Verified will be a link
> to the dashboard for a particular driver kinda like in the mock.

This sounds especially useful as we go through steps 1-2 of the ladder,
and try to track the extent of our testing for current drivers.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-13 Thread Walter A. Boring IV

On 02/13/2014 09:51 AM, Avishay Traeger wrote:

"Walter A. Boring IV"  wrote on 02/13/2014 06:59:38
PM:

What I would do different for the Icehouse release is this:

If a driver doesn't pass the certification test by IceHouse RC1, then we
have a bug filed
against the driver.   I would also put a warning message in the log for
that driver that it
doesn't pass the certification test.  I would not remove it from the
codebase.

Also:
 if a driver hasn't even run the certification test by RC1, then we
mark the driver as
uncertified and deprecated in the code and throw an error at driver init
time.
We can have a option in cinder.conf that says
ignore_uncertified_drivers=False.
If an admin wants to ignore the error, they set the flag to True, and we
let the driver init at next startup.
The admin then takes full responsibility for running uncertified code.

I think removing the drivers outright is premature for Icehouse,
since the certification process is a new thing.
For Juno, we remove any drivers that are still marked as uncertified and
haven't run the tests.

I think the purpose of the tests is to get vendors to actually run their
code through tempest and
prove to the community that they are willing to show that they are
fixing their code.  At the end of the day,
it better serves the community and Cinder if we have many working

drivers.

My $0.02,
Walt


I like this.  Make that $0.04 now :)


I wrote a bit of code so we had something to discuss if anyone thinks 
it's a good enough

compromise.
https://review.openstack.org/#/c/73464/

Walt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-13 Thread John Griffith
On Thu, Feb 13, 2014 at 10:30 AM, Dean Troyer  wrote:
> On Thu, Feb 13, 2014 at 4:51 AM, Thierry Carrez 
> wrote:
>
>> John Griffith wrote:
>> > To add some controversy and keep the original intent of having only
>> > known tested and working drivers in the Cinder release, I am going to
>> > propose that any driver that has not submitted successful functional
>> > testing by RC1 that that driver be removed.  I'd at least like to see
>> > driver maintainers try... if the test fails a test or two that's
>> > something that can be discussed, but it seems that until now most
>> > drivers just flat out are not even being tested.
>
>
> +1
>
>>
>> I think there are multiple stages here.
>>
>> Stage 0: noone knows if drivers work
>> Stage 1: we know the (potentially sad) state of the drivers that are in
>> the release
>> Stage 2: only drivers that pass tests are added, drivers that don't pass
>> tests have a gap analysis and a plan to fix it
>> Stage 3: drivers that fail tests are removed before release
>> Stage 4: 3rd-party testing rigs must run tests on every change in order
>> to stay in tree
>>
>> At the very minimum you should be at stage 1 for the Icehouse release,
>> so I agree with your last paragraph. I'd recommend that you start the
>> Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for
>> the end of the Juno release.
>
>
> Are any of these drivers new for Icehouse?  I think adding broken drivers in
> Icehouse is a mistake.  The timing WRT Icehouse release schedule is
> unfortunate but so is shipping immature drivers that have to be supported
> and possibly deprecated.  Should new drivers that are lacking have some
> not-quite-supported status to allow them to be removed in Juno if not
> brought up to par?  Or moved into cinder/contrib?

Yes, there are a boatload of new drivers being added.

>
> I don't mean to be picking on Cinder here, this seems to be recurring theme
> in OpenStack.  I think we benefit from strengthening the precedent that
> makes it harder to get things in that are not ready even if the timing is
> inconvenient.  We're seeing this in project incubation and I think we all
> benefit in the end.
>
> dt
>
> --
>
> Dean Troyer
> dtro...@gmail.com
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

I have another tact we can take on this in the interim. I like the
contrib dir idea raised by Dean, a hybrid of that and the original
proposal is we leave the certification optional, but we publish a
certified driver list.  We can also use the contrib idea with that as
well, so the contrib dir would denote drivers that are not officially
certified.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-13 Thread Avishay Traeger
"Walter A. Boring IV"  wrote on 02/13/2014 06:59:38 
PM:
> What I would do different for the Icehouse release is this:
> 
> If a driver doesn't pass the certification test by IceHouse RC1, then we 

> have a bug filed
> against the driver.   I would also put a warning message in the log for 
> that driver that it
> doesn't pass the certification test.  I would not remove it from the 
> codebase.
> 
> Also:
> if a driver hasn't even run the certification test by RC1, then we 
> mark the driver as
> uncertified and deprecated in the code and throw an error at driver init 

> time.
> We can have a option in cinder.conf that says 
> ignore_uncertified_drivers=False.
> If an admin wants to ignore the error, they set the flag to True, and we 

> let the driver init at next startup.
> The admin then takes full responsibility for running uncertified code.
> 
>I think removing the drivers outright is premature for Icehouse, 
> since the certification process is a new thing.
> For Juno, we remove any drivers that are still marked as uncertified and 

> haven't run the tests.
> 
> I think the purpose of the tests is to get vendors to actually run their 

> code through tempest and
> prove to the community that they are willing to show that they are 
> fixing their code.  At the end of the day,
> it better serves the community and Cinder if we have many working 
drivers.
> 
> My $0.02,
> Walt


I like this.  Make that $0.04 now :)

Avishay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-13 Thread Mike Perez
On Thu, Feb 13, 2014 at 9:30 AM, Dean Troyer  wrote:

> Are any of these drivers new for Icehouse?  I think adding broken drivers
> in Icehouse is a mistake.  The timing WRT Icehouse release schedule is
> unfortunate but so is shipping immature drivers that have to be supported
> and possibly deprecated.  Should new drivers that are lacking have some
> not-quite-supported status to allow them to be removed in Juno if not
> brought up to par?  Or moved into cinder/contrib?
>
> I don't mean to be picking on Cinder here, this seems to be recurring
> theme in OpenStack.  I think we benefit from strengthening the precedent
> that makes it harder to get things in that are not ready even if the timing
> is inconvenient.  We're seeing this in project incubation and I think we
> all benefit in the end.
>
> dt
>

Since the cert tests were introduced, new drivers have been required to
pass in order to be merged.


-Mike Perez
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-13 Thread John Griffith
On Thu, Feb 13, 2014 at 9:59 AM, Walter A. Boring IV
 wrote:
> On 02/13/2014 02:51 AM, Thierry Carrez wrote:
>>
>> John Griffith wrote:
>>>
>>> So we've talked about this a bit and had a number of ideas regarding
>>> how to test and show compatibility for third-party drivers in Cinder.
>>> This has been an eye opening experience (the number of folks that have
>>> NEVER run tempest before, as well as the problems uncovered now that
>>> they're trying it).
>>>
>>> I'm even more convinced now that having vendors run these tests is a
>>> good thing and should be required.  That being said there's a ton of
>>> push back from my proposal to require that results from a successful
>>> run of the tempest tests to accompany any new drivers submitted to
>>> Cinder.
>>
>> Could you describe the nature of the pushback ? Is it that the tests are
>> too deep and reject valid drivers ? Is it that it's deemed unfair to
>> block new drivers while the existing ones aren't better ? Is it that
>> it's difficult for them to run those tests and get a report ? Or is it
>> because they care more about having their name covered in mainline and
>> not so much about having the code working properly ?
>>
>>> The consensus from the Cinder community for now is that we'll
>>> log a bug for each driver after I3, stating that it hasn't passed
>>> certification tests.  We'll then have a public record showing
>>> drivers/vendors that haven't demonstrated functional compatibility,
>>> and in order to close those bugs they'll be required to run the tests
>>> and submit the results to the bug in Launchpad.
>>>
>>> So, this seems to be the approach we're taking for Icehouse at least,
>>> it's far from ideal IMO, however I think it's still progress and it's
>>> definitely exposed some issues with how drivers are currently
>>> submitted to Cinder so those are positive things that we can learn
>>> from and improve upon in future releases.
>>>
>>> To add some controversy and keep the original intent of having only
>>> known tested and working drivers in the Cinder release, I am going to
>>> propose that any driver that has not submitted successful functional
>>> testing by RC1 that that driver be removed.  I'd at least like to see
>>> driver maintainers try... if the test fails a test or two that's
>>> something that can be discussed, but it seems that until now most
>>> drivers just flat out are not even being tested.
>>
>> I think there are multiple stages here.
>>
>> Stage 0: noone knows if drivers work
>> Stage 1: we know the (potentially sad) state of the drivers that are in
>> the release
>> Stage 2: only drivers that pass tests are added, drivers that don't pass
>> tests have a gap analysis and a plan to fix it
>> Stage 3: drivers that fail tests are removed before release
>> Stage 4: 3rd-party testing rigs must run tests on every change in order
>> to stay in tree
>>
>> At the very minimum you should be at stage 1 for the Icehouse release,
>> so I agree with your last paragraph. I'd recommend that you start the
>> Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for
>> the end of the Juno release.
>>
> I have to agree with Thierry here.  I think if we can get drivers to pass
> the tests
> in the Juno timeframe, then it's fine to remove then during Juno.
> I think the idea of having drivers run their code through tempest and work
> towards passing all of those tests is a great thing for Cinder and OpenStack
> in general.
>
> What I would do different for the Icehouse release is this:
>
> If a driver doesn't pass the certification test by IceHouse RC1, then we
> have a bug filed
> against the driver.   I would also put a warning message in the log for that
> driver that it
> doesn't pass the certification test.  I would not remove it from the
> codebase.
>
> Also:
>if a driver hasn't even run the certification test by RC1, then we mark
> the driver as
> uncertified and deprecated in the code and throw an error at driver init
> time.
> We can have a option in cinder.conf that says
> ignore_uncertified_drivers=False.
> If an admin wants to ignore the error, they set the flag to True, and we let
> the driver init at next startup.
> The admin then takes full responsibility for running uncertified code.
>
>   I think removing the drivers outright is premature for Icehouse, since the
> certification process is a new thing.
> For Juno, we remove any drivers that are still marked as uncertified and
> haven't run the tests.
>
> I think the purpose of the tests is to get vendors to actually run their
> code through tempest and
> prove to the community that they are willing to show that they are fixing
> their code.  At the end of the day,
> it better serves the community and Cinder if we have many working drivers.
>
> My $0.02,
> Walt
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
I'm fine with 

Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-13 Thread Dean Troyer
On Thu, Feb 13, 2014 at 4:51 AM, Thierry Carrez wrote:

> John Griffith wrote:
> > To add some controversy and keep the original intent of having only
> > known tested and working drivers in the Cinder release, I am going to
> > propose that any driver that has not submitted successful functional
> > testing by RC1 that that driver be removed.  I'd at least like to see
> > driver maintainers try... if the test fails a test or two that's
> > something that can be discussed, but it seems that until now most
> > drivers just flat out are not even being tested.
>

+1


> I think there are multiple stages here.
>
> Stage 0: noone knows if drivers work
> Stage 1: we know the (potentially sad) state of the drivers that are in
> the release
> Stage 2: only drivers that pass tests are added, drivers that don't pass
> tests have a gap analysis and a plan to fix it
> Stage 3: drivers that fail tests are removed before release
> Stage 4: 3rd-party testing rigs must run tests on every change in order
> to stay in tree
>
> At the very minimum you should be at stage 1 for the Icehouse release,
> so I agree with your last paragraph. I'd recommend that you start the
> Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for
> the end of the Juno release.
>

Are any of these drivers new for Icehouse?  I think adding broken drivers
in Icehouse is a mistake.  The timing WRT Icehouse release schedule is
unfortunate but so is shipping immature drivers that have to be supported
and possibly deprecated.  Should new drivers that are lacking have some
not-quite-supported status to allow them to be removed in Juno if not
brought up to par?  Or moved into cinder/contrib?

I don't mean to be picking on Cinder here, this seems to be recurring theme
in OpenStack.  I think we benefit from strengthening the precedent that
makes it harder to get things in that are not ready even if the timing is
inconvenient.  We're seeing this in project incubation and I think we all
benefit in the end.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-13 Thread Walter A. Boring IV

On 02/13/2014 02:51 AM, Thierry Carrez wrote:

John Griffith wrote:

So we've talked about this a bit and had a number of ideas regarding
how to test and show compatibility for third-party drivers in Cinder.
This has been an eye opening experience (the number of folks that have
NEVER run tempest before, as well as the problems uncovered now that
they're trying it).

I'm even more convinced now that having vendors run these tests is a
good thing and should be required.  That being said there's a ton of
push back from my proposal to require that results from a successful
run of the tempest tests to accompany any new drivers submitted to
Cinder.

Could you describe the nature of the pushback ? Is it that the tests are
too deep and reject valid drivers ? Is it that it's deemed unfair to
block new drivers while the existing ones aren't better ? Is it that
it's difficult for them to run those tests and get a report ? Or is it
because they care more about having their name covered in mainline and
not so much about having the code working properly ?


The consensus from the Cinder community for now is that we'll
log a bug for each driver after I3, stating that it hasn't passed
certification tests.  We'll then have a public record showing
drivers/vendors that haven't demonstrated functional compatibility,
and in order to close those bugs they'll be required to run the tests
and submit the results to the bug in Launchpad.

So, this seems to be the approach we're taking for Icehouse at least,
it's far from ideal IMO, however I think it's still progress and it's
definitely exposed some issues with how drivers are currently
submitted to Cinder so those are positive things that we can learn
from and improve upon in future releases.

To add some controversy and keep the original intent of having only
known tested and working drivers in the Cinder release, I am going to
propose that any driver that has not submitted successful functional
testing by RC1 that that driver be removed.  I'd at least like to see
driver maintainers try... if the test fails a test or two that's
something that can be discussed, but it seems that until now most
drivers just flat out are not even being tested.

I think there are multiple stages here.

Stage 0: noone knows if drivers work
Stage 1: we know the (potentially sad) state of the drivers that are in
the release
Stage 2: only drivers that pass tests are added, drivers that don't pass
tests have a gap analysis and a plan to fix it
Stage 3: drivers that fail tests are removed before release
Stage 4: 3rd-party testing rigs must run tests on every change in order
to stay in tree

At the very minimum you should be at stage 1 for the Icehouse release,
so I agree with your last paragraph. I'd recommend that you start the
Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for
the end of the Juno release.

I have to agree with Thierry here.  I think if we can get drivers to 
pass the tests

in the Juno timeframe, then it's fine to remove then during Juno.
I think the idea of having drivers run their code through tempest and work
towards passing all of those tests is a great thing for Cinder and 
OpenStack in general.


What I would do different for the Icehouse release is this:

If a driver doesn't pass the certification test by IceHouse RC1, then we 
have a bug filed
against the driver.   I would also put a warning message in the log for 
that driver that it
doesn't pass the certification test.  I would not remove it from the 
codebase.


Also:
   if a driver hasn't even run the certification test by RC1, then we 
mark the driver as
uncertified and deprecated in the code and throw an error at driver init 
time.
We can have a option in cinder.conf that says 
ignore_uncertified_drivers=False.
If an admin wants to ignore the error, they set the flag to True, and we 
let the driver init at next startup.

The admin then takes full responsibility for running uncertified code.

  I think removing the drivers outright is premature for Icehouse, 
since the certification process is a new thing.
For Juno, we remove any drivers that are still marked as uncertified and 
haven't run the tests.


I think the purpose of the tests is to get vendors to actually run their 
code through tempest and
prove to the community that they are willing to show that they are 
fixing their code.  At the end of the day,

it better serves the community and Cinder if we have many working drivers.

My $0.02,
Walt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-13 Thread Thierry Carrez
John Griffith wrote:
> So we've talked about this a bit and had a number of ideas regarding
> how to test and show compatibility for third-party drivers in Cinder.
> This has been an eye opening experience (the number of folks that have
> NEVER run tempest before, as well as the problems uncovered now that
> they're trying it).
> 
> I'm even more convinced now that having vendors run these tests is a
> good thing and should be required.  That being said there's a ton of
> push back from my proposal to require that results from a successful
> run of the tempest tests to accompany any new drivers submitted to
> Cinder.

Could you describe the nature of the pushback ? Is it that the tests are
too deep and reject valid drivers ? Is it that it's deemed unfair to
block new drivers while the existing ones aren't better ? Is it that
it's difficult for them to run those tests and get a report ? Or is it
because they care more about having their name covered in mainline and
not so much about having the code working properly ?

> The consensus from the Cinder community for now is that we'll
> log a bug for each driver after I3, stating that it hasn't passed
> certification tests.  We'll then have a public record showing
> drivers/vendors that haven't demonstrated functional compatibility,
> and in order to close those bugs they'll be required to run the tests
> and submit the results to the bug in Launchpad.
> 
> So, this seems to be the approach we're taking for Icehouse at least,
> it's far from ideal IMO, however I think it's still progress and it's
> definitely exposed some issues with how drivers are currently
> submitted to Cinder so those are positive things that we can learn
> from and improve upon in future releases.
> 
> To add some controversy and keep the original intent of having only
> known tested and working drivers in the Cinder release, I am going to
> propose that any driver that has not submitted successful functional
> testing by RC1 that that driver be removed.  I'd at least like to see
> driver maintainers try... if the test fails a test or two that's
> something that can be discussed, but it seems that until now most
> drivers just flat out are not even being tested.

I think there are multiple stages here.

Stage 0: noone knows if drivers work
Stage 1: we know the (potentially sad) state of the drivers that are in
the release
Stage 2: only drivers that pass tests are added, drivers that don't pass
tests have a gap analysis and a plan to fix it
Stage 3: drivers that fail tests are removed before release
Stage 4: 3rd-party testing rigs must run tests on every change in order
to stay in tree

At the very minimum you should be at stage 1 for the Icehouse release,
so I agree with your last paragraph. I'd recommend that you start the
Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for
the end of the Juno release.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [OpenStack-Dev] [Cinder] Cinder driver verification

2014-02-12 Thread John Griffith
Hey,

So we've talked about this a bit and had a number of ideas regarding
how to test and show compatibility for third-party drivers in Cinder.
This has been an eye opening experience (the number of folks that have
NEVER run tempest before, as well as the problems uncovered now that
they're trying it).

I'm even more convinced now that having vendors run these tests is a
good thing and should be required.  That being said there's a ton of
push back from my proposal to require that results from a successful
run of the tempest tests to accompany any new drivers submitted to
Cinder.  The consensus from the Cinder community for now is that we'll
log a bug for each driver after I3, stating that it hasn't passed
certification tests.  We'll then have a public record showing
drivers/vendors that haven't demonstrated functional compatibility,
and in order to close those bugs they'll be required to run the tests
and submit the results to the bug in Launchpad.

So, this seems to be the approach we're taking for Icehouse at least,
it's far from ideal IMO, however I think it's still progress and it's
definitely exposed some issues with how drivers are currently
submitted to Cinder so those are positive things that we can learn
from and improve upon in future releases.

To add some controversy and keep the original intent of having only
known tested and working drivers in the Cinder release, I am going to
propose that any driver that has not submitted successful functional
testing by RC1 that that driver be removed.  I'd at least like to see
driver maintainers try... if the test fails a test or two that's
something that can be discussed, but it seems that until now most
drivers just flat out are not even being tested.

Thanks,
John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev