subject:"Re\: \[openstack\-dev\] \[Cinder\] \[stable\] \[all\] Changing stable policy for drivers"

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-09-05 Thread Arkady_Kanevsky

That is the question of how many releases backwards community is willing to 
“support”.
The current answer is 1 year. If customer wants something earlier – do it 
yourself. Or to be precise work with you vendor whose driver you want updated. 
This creates vendor driver mismatch issues for validation of older version. The 
only two options I see are 3rd party distros or OpenStack changing its policy.
Former is being done now.
Thus, it works.
Thanks,
Arkady


From: Duncan Thomas [mailto:duncan.tho...@gmail.com]
Sent: Monday, August 22, 2016 3:45 PM
To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for 
drivers


What is the logic for that? It's a massive duplication of effort, and it leads 
to defacto forks and inconsistencies between clouds - exactly what the 
OpenStack mission is against.

Many/most of the clouds actually in production are already out of upstream 
stable policy. The more convergence we can get on what happens after that the 
better. There are zero advantages I can see to each vendor going it alone.

On 22 Aug 2016 19:31, 
<arkady_kanev...@dell.com<mailto:arkady_kanev...@dell.com>> wrote:
Sorry if touch 3rd rail.
But should backport bug fixes to older releases be done in distros and not 
upstream?

-Original Message-
From: Walter A. Boring IV 
[mailto:walter.bor...@hpe.com<mailto:walter.bor...@hpe.com>]
Sent: Tuesday, August 09, 2016 12:34 PM
To: openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for 
drivers

On 08/08/2016 02:28 PM, Ihar Hrachyshka wrote:
> Duncan Thomas wrote:
>
>> On 8 August 2016 at 21:12, Matthew Treinish
>> wrote:
>> Ignoring all that, this is also contrary to how we perform testing in
>> OpenStack.
>> We don't turn off entire classes of testing we have so we can land
>> patches, that's just a recipe for disaster.
>>
>> But is it more of a disaster (for the consumers) than zero testing,
>> zero review, scattered around the internet
>> if-you're-lucky-with-a-good-wind you'll maybe get the right patch
>> set? Because that's where we are right now, and vendors, distributors
>> and the cinder core team are all saying it's a disaster.
>
> If consumers rely on upstream releases, then they are expected to
> migrate to newer releases after EOL, not switch to a random branch on
> the internet. If they rely on some commercial product, then they
> usually have an extended period of support and certification for their
> drivers, so it’s not a problem for them.
>
> Ihar
This is entirely unrealistic. Force customers to upgrade. Good luck
explaining to a bank that in order to get their cinder driver fix in, they have 
to upgrade their entire OpenStack deployment. Real world customers simply will 
balk at this all day long.

Walt
>
> __
> 
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-22 Thread Duncan Thomas

What is the logic for that? It's a massive duplication of effort, and it
leads to defacto forks and inconsistencies between clouds - exactly what
the OpenStack mission is against.

Many/most of the clouds actually in production are already out of upstream
stable policy. The more convergence we can get on what happens after that
the better. There are zero advantages I can see to each vendor going it
alone.

On 22 Aug 2016 19:31, <arkady_kanev...@dell.com> wrote:

> Sorry if touch 3rd rail.
>
> But should backport bug fixes to older releases be done in distros and not
> upstream?
>
> -Original Message-
> From: Walter A. Boring IV [mailto:walter.bor...@hpe.com]
> Sent: Tuesday, August 09, 2016 12:34 PM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [Cinder] [stable] [all] Changing stable
> policy for drivers
>
> On 08/08/2016 02:28 PM, Ihar Hrachyshka wrote:
> > Duncan Thomas wrote:
> >
> >> On 8 August 2016 at 21:12, Matthew Treinish
> >> wrote:
> >> Ignoring all that, this is also contrary to how we perform testing in
> >> OpenStack.
> >> We don't turn off entire classes of testing we have so we can land
> >> patches, that's just a recipe for disaster.
> >>
> >> But is it more of a disaster (for the consumers) than zero testing,
> >> zero review, scattered around the internet
> >> if-you're-lucky-with-a-good-wind you'll maybe get the right patch
> >> set? Because that's where we are right now, and vendors, distributors
> >> and the cinder core team are all saying it's a disaster.
> >
> > If consumers rely on upstream releases, then they are expected to
> > migrate to newer releases after EOL, not switch to a random branch on
> > the internet. If they rely on some commercial product, then they
> > usually have an extended period of support and certification for their
> > drivers, so it’s not a problem for them.
> >
> > Ihar
> This is entirely unrealistic. Force customers to upgrade. Good luck
> explaining to a bank that in order to get their cinder driver fix in, they
> have to upgrade their entire OpenStack deployment. Real world customers
> simply will balk at this all day long.
>
> Walt
> >
> > __
> > 
> >
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-22 Thread Arkady_Kanevsky

Sorry if touch 3rd rail.
But should backport bug fixes to older releases be done in distros and not 
upstream?

-Original Message-
From: Walter A. Boring IV [mailto:walter.bor...@hpe.com]
Sent: Tuesday, August 09, 2016 12:34 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for 
drivers

On 08/08/2016 02:28 PM, Ihar Hrachyshka wrote:
> Duncan Thomas wrote:
>
>> On 8 August 2016 at 21:12, Matthew Treinish
>> wrote:
>> Ignoring all that, this is also contrary to how we perform testing in
>> OpenStack.
>> We don't turn off entire classes of testing we have so we can land
>> patches, that's just a recipe for disaster.
>>
>> But is it more of a disaster (for the consumers) than zero testing,
>> zero review, scattered around the internet
>> if-you're-lucky-with-a-good-wind you'll maybe get the right patch
>> set? Because that's where we are right now, and vendors, distributors
>> and the cinder core team are all saying it's a disaster.
>
> If consumers rely on upstream releases, then they are expected to
> migrate to newer releases after EOL, not switch to a random branch on
> the internet. If they rely on some commercial product, then they
> usually have an extended period of support and certification for their
> drivers, so it's not a problem for them.
>
> Ihar
This is entirely unrealistic. Force customers to upgrade. Good luck
explaining to a bank that in order to get their cinder driver fix in, they have 
to upgrade their entire OpenStack deployment. Real world customers simply will 
balk at this all day long.

Walt
>
> __
> 
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-12 Thread Thierry Carrez

Duncan Thomas wrote:
> [...]
> To turn the question around, what is the downside of loosing the tag?

The tag does not exist in a vacuum. It describes a behavior that
operators want.

They want a sane deprecation policy so that the blanket is not pulled
from under them without a warning. They want a sane stable policy so
that they can deploy from trusted stable branches and not necessarily
review patch-per-patch to see if one happens to change behavior in an
unwanted way.

> Are people going to suddenly stop deploying cinder? That seems rather
> unlikely.

This sounds a bit like "you'll have to swallow this or just stop
deploying cinder altogether" which is not really a place I want us to
put our users in.

Asking the same questions on the -operators list, telling them that you
won't have a deprecation policy or a stable policy anymore and that if
you don't like it you should drop OpenStack altogether, I suspect you'd
get a pretty sad response.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-12 Thread Duncan Thomas

On 12 August 2016 at 16:09, Thierry Carrez  wrote:

>
> How about: 4. Take 3rd-party drivers to a separate cinder-extra-drivers
> repository/deliverable under the Cinder team, one that would /not/ have
> follows-stable-policy or follows-standard-deprecation tags ? That
> repository would still get core-reviewed by the Cinder team, so you
> would keep the centralized code review value. It would be in a single
> repository, so you would keep most of the "all drivers checked out in
> one place" benefits. But you could have a special stable branch policy
> there and that would also solve that other issue in the thread about
> removing unmaintained drivers without deprecation notices.
>
> Or is there another benefit in shipping everything inside a single
> repository that you didn't mention ?
>

The development process is definitely smoother with everything in one repo.
Cross repo changes (even repos under the same team, like brinck is for
cinder) are painful, because you have to get the change into the 'child'
repo, wait of it to merge, then wait for it to be released in some form
that is usable to the parent project (e.g. a pip release), then finally you
can merge the cinder change.

To turn the question around, what is the downside of loosing the tag? Are
people going to suddenly stop deploying cinder? That seems rather unlikely.

Nobody has yet given a single benefit to shipping a broken driver.

-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-12 Thread Thierry Carrez

Duncan Thomas wrote:
> [...]
> Given this need, what are our options?
> 
> 1. We could do all this outside Openstack infrastructure. There are
> significant downsides to doing so from organisational, maintenance, cost
> etc points of view. Also means that the place vendors go for these
> patches is not obvious, and the process for getting patches in is not
> standard.
> 
> 2. We could have something not named 'stable' that has looser rules than
> stable branches,, maybe just pep8 / unit / cinder in-tree tests. No
> devstack.
> 
> 3. We go with the Neutron model and take drivers out of tree. This is
> not something the cinder core team are in favour of - we see significant
> value in the code review that drivers currently get - the code quality
> improvements between when a driver is submitted and when it is merged
> are sometimes very significant. Also, taking the code out of tree makes
> it difficult to get all the drivers checked out in one place to analyse
> e.g. how a certain driver call is implemented across all the drivers,
> when reasoning or making changes to core code.

How about: 4. Take 3rd-party drivers to a separate cinder-extra-drivers
repository/deliverable under the Cinder team, one that would /not/ have
follows-stable-policy or follows-standard-deprecation tags ? That
repository would still get core-reviewed by the Cinder team, so you
would keep the centralized code review value. It would be in a single
repository, so you would keep most of the "all drivers checked out in
one place" benefits. But you could have a special stable branch policy
there and that would also solve that other issue in the thread about
removing unmaintained drivers without deprecation notices.

Or is there another benefit in shipping everything inside a single
repository that you didn't mention ?

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-12 Thread Duncan Thomas

Is there some docs for it somewhere? Or some quick way of telling that
we've done it and gotten it right?

On 12 Aug 2016 08:17, "Andreas Jaeger"  wrote:

> On 08/12/2016 04:25 AM, Robert Collins wrote:
> > On 11 Aug 2016 3:13 PM, "Ben Swartzlander"  > > wrote:
> >>
> >> ...
> >>
> >> I still don't agree with this stance. Code doesn't just magically stop
> > working. Code breaks when things change which aren't version controlled
> > properly or when you have undeclared dependencies.
> >
> > Well this is why the constraints work was and is being done. It's not
> > 100%rolled out as far as I know though, and stable branch support feels
> > all the gaps.
>
> As announced yesterday:
>
> Constraints work is *now* 100 % rolled out from the infra side, it's up
> to projects to use it fully now,
>
> Andreas
> --
>  Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
>   SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
>GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>HRB 21284 (AG Nürnberg)
> GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-11 Thread Andreas Jaeger

On 08/12/2016 04:25 AM, Robert Collins wrote:
> On 11 Aug 2016 3:13 PM, "Ben Swartzlander"  > wrote:
>>
>> ...
>>
>> I still don't agree with this stance. Code doesn't just magically stop
> working. Code breaks when things change which aren't version controlled
> properly or when you have undeclared dependencies.
> 
> Well this is why the constraints work was and is being done. It's not
> 100%rolled out as far as I know though, and stable branch support feels
> all the gaps.

As announced yesterday:

Constraints work is *now* 100 % rolled out from the infra side, it's up
to projects to use it fully now,

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-11 Thread Robert Collins

On 11 Aug 2016 3:13 PM, "Ben Swartzlander"  wrote:
>
> ...
>
> I still don't agree with this stance. Code doesn't just magically stop
working. Code breaks when things change which aren't version controlled
properly or when you have undeclared dependencies.

Well this is why the constraints work was and is being done. It's not
100%rolled out as far as I know though, and stable branch support feels all
the gaps.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-11 Thread Chris Friesen

On 08/11/2016 04:13 PM, Ben Swartzlander wrote:

On 08/10/2016 01:57 PM, Matthew Treinish wrote:

On Wed, Aug 10, 2016 at 09:52:55AM -0700, Clay Gerrard wrote:

On Wed, Aug 10, 2016 at 7:42 AM, Ben Swartzlander
wrote:

A big source of problems IMO is that tempest doesn't have stable branches.
We use the master branch of tempest to test stable branches of other
projects, and tempest regularly adds new features.

How come not this +1000 just fix this?

Well, mostly because it's actually not a problem and ignores the history on why
tempest is branchless. We actually used to do this pre-icehouse and it actually
made things much worse. What was happening back then was we didn't have enough
activity to keep the stable branches working at all. So we'd go very long
periods where nothing actually could land. We also often wedged ourselves where
master tempest changed to a point where we couldn't sanely backport a fix to
the stable branch. This would often mean that up until right before a stable
release things just couldn't land until someone was actually motivated to try
and dig us out. But, what more often happened was we had to just disable tempest
on the branch, because we didn't have another option. It also turns out that
having different tests across a release boundary meant we weren't actually
validating that the OpenStack APIs were consistent and worked the same. We had
many instances where a projects API just changed between release boundaries,
which violates our API consistency and backwards compatibility guidelines.
Tempest is about verifying the API and just like an other API client it should
work against any OpenStack release.

Doing this has been a huge benefit for making things actually work on the stable
branches. (in fact just thinking back about how broken everything was all the
time back then makes me appreciate it even more) We also test every incoming
tempest change on all the stable branches, and nothing can land unless it works
on all supported branches. It means we have a consistent and stable api across
releases. We do have occasional bugs where a new test or change in tempest
triggers a new race in a project's stable branch. But, that's a real bug and
normally a fix can be backported.(which is the whole point of doing stable
branches) If it can't and the race is bad enough to actively interfere with
things, we have a mechanism to skip the test. (but that's normally a last
resort) Although, these issues tend to come up pretty infrequently in practice,
especially as we slowly ramp up the stability of things over time.

FWIW, a lot of these details are covered in the original spec for implementing
this: (although it's definitely assumes a bit of prior knowledge about the
state of things going on when it was written)

http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html

I still don't agree with this stance. Code doesn't just magically stop working.
Code breaks when things change which aren't version controlled properly or when
you have undeclared dependencies.

Or testcases breaks when you change the timing of your tests, which exposes
latent bugs that were always there but by fluke weren't being hit.

In this case the code was previously broken, we just weren't hitting it in the
testcases.

Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-11 Thread Ben Swartzlander

On 08/10/2016 01:57 PM, Matthew Treinish wrote:

On Wed, Aug 10, 2016 at 09:52:55AM -0700, Clay Gerrard wrote:

On Wed, Aug 10, 2016 at 7:42 AM, Ben Swartzlander
wrote:

A big source of problems IMO is that tempest doesn't have stable branches.
We use the master branch of tempest to test stable branches of other
projects, and tempest regularly adds new features.

How come not this +1000 just fix this?

FWIW, a lot of these details are covered in the original spec for implementing
this: (although it's definitely assumes a bit of prior knowledge about the
state of things going on when it was written)

http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html

I still don't agree with this stance. Code doesn't just magically stop
working. Code breaks when things change which aren't version controlled
properly or when you have undeclared dependencies.

Yes, managing dependencies and doing version control is a lot of work. I
understand that the tempest team is strapped for resources. However
simply declaring that the solution is to stop doing version control is
an epic failure. I read the spec above and it sounds like cry for help
rather than a well thought out idea.

Personally I would be interested in helping improve this situation, but
I think the proper way to improve it is to actually do version control
of everything that matters and use dependency management. If you do this
correctly, then maintaining the old stuff stops being a chore because it
never breaks. By definition, if a stable branch breaks without a change
going into it, you failed at doing dependency management.

I'm not sure how I even could help with the current situation. It feels
like we've dug ourselves into a hole and the plan of record is to get
used to living underground. Does anyone have a will to make thing better
and get to a place where stable branches could reasonably expected to
remain stable for months or years without constant fixing?

-Ben Swartzlander

-Matt Treinish

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Clay Gerrard

On Wed, Aug 10, 2016 at 10:57 AM, Matthew Treinish 
wrote:

>
> http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/
> branchless-tempest.html
>
>
>
This was actually a *great* read, thanks for that link!

-Clay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Clay Gerrard

On Wed, Aug 10, 2016 at 10:57 AM, Matthew Treinish 
wrote:

> We also test every incoming
> tempest change on all the stable branches, and nothing can land unless it
> works
> on all supported branches.


Did not know that, pretty awesome!


>
-Clay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Matthew Treinish

On Wed, Aug 10, 2016 at 09:52:55AM -0700, Clay Gerrard wrote:
> On Wed, Aug 10, 2016 at 7:42 AM, Ben Swartzlander 
> wrote:
> 
> >
> > A big source of problems IMO is that tempest doesn't have stable branches.
> > We use the master branch of tempest to test stable branches of other
> > projects, and tempest regularly adds new features.
> >
> 
> How come not this +1000 just fix this?

Well, mostly because it's actually not a problem and ignores the history on why
tempest is branchless. We actually used to do this pre-icehouse and it actually
made things much worse. What was happening back then was we didn't have enough
activity to keep the stable branches working at all. So we'd go very long
periods where nothing actually could land. We also often wedged ourselves where
master tempest changed to a point where we couldn't sanely backport a fix to
the stable branch. This would often mean that up until right before a stable
release things just couldn't land until someone was actually motivated to try
and dig us out. But, what more often happened was we had to just disable tempest
on the branch, because we didn't have another option. It also turns out that
having different tests across a release boundary meant we weren't actually
validating that the OpenStack APIs were consistent and worked the same. We had
many instances where a projects API just changed between release boundaries,
which violates our API consistency and backwards compatibility guidelines.
Tempest is about verifying the API and just like an other API client it should
work against any OpenStack release.

Doing this has been a huge benefit for making things actually work on the stable
branches. (in fact just thinking back about how broken everything was all the
time back then makes me appreciate it even more) We also test every incoming
tempest change on all the stable branches, and nothing can land unless it works
on all supported branches. It means we have a consistent and stable api across
releases. We do have occasional bugs where a new test or change in tempest
triggers a new race in a project's stable branch. But, that's a real bug and
normally a fix can be backported.(which is the whole point of doing stable
branches) If it can't and the race is bad enough to actively interfere with
things, we have a mechanism to skip the test. (but that's normally a last
resort) Although, these issues tend to come up pretty infrequently in practice,
especially as we slowly ramp up the stability of things over time.

FWIW, a lot of these details are covered in the original spec for implementing
this: (although it's definitely assumes a bit of prior knowledge about the
state of things going on when it was written)

http://specs.openstack.org/openstack/qa-specs/specs/tempest/implemented/branchless-tempest.html

-Matt Treinish

signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Clay Gerrard

On Wed, Aug 10, 2016 at 10:21 AM, Matthew Treinish 
wrote:

> But, to keep the gate running
> involves a lot of coordination between multiple projects that are tightly
> coupled. Things like an entire extra set of job definitions in zuul, a
> branch on
> global requirements, a devstack branch, extra devstack-gate logic, a bunch
> of
> extra config options for skips in tempest, extra node types, etc. Keeping
> all
> those things working together is a big part of what stable maint actually
> entails.

that actually makes more sense (sorry I missed any earlier explanation) -
I'm reading this as there is only ever one CI *system* running at a time,
and that system needs to know a bunch about how to *setup* a test job on an
old branches - not that any of the old versions of code or tests or even
the history of the CI system that existed and was able to test them at the
time is GONE - its just that the current deployed system needs to move on...

> That's why at the EOL we tag
> the branch tip and then delete it. Leaving the branch around advertises
> that
> we're in a position to accept new patches to it, which we aren't after the
> EOL.
>
>
Oh wow... so it *is* GONE ;)

And really "we can't test it so no-one can" might be a big part of the
issue that was brought up in the earlier thread.  Maybe trying to support
stable branches longer than 18 months is *not* something can to be broadly
supported inside of OpenStack (there seemed to be some interest in the
etherpad going out to 24 months some day, even though older branches would
have less and less support for new testing capabilities).  But I think the
heart of this thread is "we appreciate the complexity and effort that it
takes to deliver what we have for older branches.  [full stop]  We need a
way to extend some minimal life support into older releases in a way that
is compatible with the current policy.  [full stop] "

Would it be *too* confusing to have "End of Full OpenStack Supported
Official Testing/Life" != "End of a projects commitment to people running
clouds using our software to try and help them be successful"?  Without
having to define unilaterally for every installation that the only option
for success is upgrade to the next about to be abandoned in 6-18 mo major
release?

I think ideally we'd be looking for a way to let them have their cake
without extra work.

OTOH, forking to support old branches seems just as reasonable to me as
well (that's what we do)...

However, I fully admit, I'm probably thinking about it wrong.  :D

-Clay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Matthew Treinish

On Wed, Aug 10, 2016 at 09:56:09AM -0700, Clay Gerrard wrote:
> On Mon, Aug 8, 2016 at 8:31 AM, Matthew Treinish 
> wrote:
> 
> > When we EOL a branch all of the infrastructure for running any ci against
> > it goes away.
> 
> 
> But... like... version control?  I mean I'm sure it's more complicated than
> that or you wouldn't have said this - but I don't understand, sorry.
> 
> Can you elaborate on this?
> 

I did in other parts of the thread. The thing is you're only thinking about the
CI system as involving a single project and repo. But, to keep the gate running
involves a lot of coordination between multiple projects that are tightly
coupled. Things like an entire extra set of job definitions in zuul, a branch on
global requirements, a devstack branch, extra devstack-gate logic, a bunch of
extra config options for skips in tempest, extra node types, etc. Keeping all
those things working together is a big part of what stable maint actually
entails. When we EOL a branch most of the mechanics involved are a matter of
cleaning up all of those pieces everywhere because we don't have the bandwidth
or resources to continue keeping it all working. That's why at the EOL we tag 
the branch tip and then delete it. Leaving the branch around advertises that
we're in a position to accept new patches to it, which we aren't after the EOL.

-Matt Treinish

signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Clay Gerrard

On Mon, Aug 8, 2016 at 8:31 AM, Matthew Treinish 
wrote:

> When we EOL a branch all of the infrastructure for running any ci against
> it goes away.

But... like... version control?  I mean I'm sure it's more complicated than
that or you wouldn't have said this - but I don't understand, sorry.

Can you elaborate on this?

-Clay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Clay Gerrard

On Wed, Aug 10, 2016 at 7:42 AM, Ben Swartzlander 
wrote:

>
> A big source of problems IMO is that tempest doesn't have stable branches.
> We use the master branch of tempest to test stable branches of other
> projects, and tempest regularly adds new features.
>

How come not this +1000 just fix this?
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Luigi Toscano

On Wednesday, 10 August 2016 12:00:41 CEST Ben Swartzlander wrote:
> On 08/10/2016 11:33 AM, Luigi Toscano wrote:
> > On Wednesday, 10 August 2016 17:00:36 CEST Ihar Hrachyshka wrote:
> >> Luigi Toscano  wrote:
> >>> On Wednesday, 10 August 2016 10:42:41 CEST Ben Swartzlander wrote:
>  On 08/10/2016 04:33 AM, Duncan Thomas wrote:
> > So I tried to get into helping with the cinder stable tree for a
> > while,
> > and while I wasn't very successful (lack of time and an inability to
> > convince my employer it should be a priority), one thing I did notice
> > it
> > that much of the breakage seemed to come from outside cinder - many of
> > the libraries we depend on make backwards incompatible changes by
> > accident, for example. Would it be possible to have a
> > long-term-support
> > branch where we pinned the max version of everything for the gate,
> > pips
> > and devtstack? I'd have thought (and I'm very willing to be corrected)
> > that would make the stable gate, well, stable, such that it required
> > far
> > less work to keep it able to run a basic devstack test plus unit
> > tests.
> > 
> > Does that sound at all sane?
>  
>  A big source of problems IMO is that tempest doesn't have stable
>  branches. We use the master branch of tempest to test stable branches
>  of
>  other projects, and tempest regularly adds new features. This
>  guarantees
>  instability if you rely on tempest anywhere in your gate (and cinder
>  does).
> >>> 
> >>> Orthogonal to the discussion, but: this is not due to the lack of stable
> >>> branch, but that part of the Tempest API are not stable yet. This is
> >>> being
> >>> addressed right now (in scope for Newton).
> >>> Once the Tempest stable API are used, no breakages should happen.
> >> 
> >> Well, it’s only partially true. But what happens when you add a new test
> >> to
> >> tempest/master? It gets executed on all branches, and maybe some of them
> >> are failing it. We can argue that it’s probably a bug revealed, but it
> >> nevertheless requires attention from stable maintainers to solve.
> > 
> > The new test should work on all support branches. As tester I find a lot
> > of
> > advantages of maintaining a unified set of tests than fighting with
> > backports of tests.
> 
> I'm sure it makes YOUR life easier to not have to deal with backports.
> The rest of us who do maintain stable branches though don't appreciate
> it when tempest adds a new feature or a new dependency which breaks the
> stable gate jobs. This happens a few times each release, and tends to
> get fixed within a day or two, which is barely tolerable.

Again, if a new test uncover the bug, why blame the test and not the bug?

If you don't want to fix bugs in stable branches, why keep them?

I don't understand...

> 
> The fact that it happens at all though says that we're "doing it wrong"
> w.r.t. testing of stable branches, and completely explains why we find
> it so challenging to support stuff more than 12 months old. If
> everything we used had stable branches and proper dependency management,
> then it would easy to keep gate job running for years.

Again, if a test uncover a bugs, let fix the bug.

> 
> > There are mechanisms to skip tests based on the cloud capabilities.
> > So this should not be an issue, and if a bug is found that should
> > definitely be viewed as a good thing.
> 
> It's not new tests that cause the problem, because those would be easy
> to skip. It's changes to tempest core which force changes elsewhere.

As I mentioned before, that's something temporary, due to the change of scope 
of Tempest (the core six, the plugin systems) and the stabilization of that's 
ongoing for Newton, so *right now*.
We will forget quickly about all of this once its fixed (more or less like the 
old complains about Neutron - rememeber?). 
Patches always welcome to speed up the Tempest API stabilization.

-- 
Luigi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Mike Perez

On 19:42 Aug 09, Ben Swartzlander wrote:
> Mike, you must have left the midcycle by the time this topic came up. On the
> issue of out-of-tree drivers, I specifically offered this proposal (a
> community managed mechanism for distributing driver bugfix backports) as an
> compromise alternative to try to address the needs of both camps. Everyone
> who was in the room at the time (plus DuncanT who wasn't) agreed that if we
> had that (a way to deal with backports) that they wouldn't want drivers out
> of the tree anymore.
> 
> Your point of view wasn't represented so go ahead and explain why, if we did
> have a reasonable way for bugfixes to get backported to the releases
> customers actually run (leaving that mechanism unspecified for the time
> being), that you would still want the drivers out of the tree.

There is no need to repeat. Read the earlier oppositions in this thread,
thanks.

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Ben Swartzlander


On 08/10/2016 11:33 AM, Luigi Toscano wrote:

On Wednesday, 10 August 2016 17:00:36 CEST Ihar Hrachyshka wrote:

Luigi Toscano  wrote:

On Wednesday, 10 August 2016 10:42:41 CEST Ben Swartzlander wrote:

On 08/10/2016 04:33 AM, Duncan Thomas wrote:

So I tried to get into helping with the cinder stable tree for a while,
and while I wasn't very successful (lack of time and an inability to
convince my employer it should be a priority), one thing I did notice it
that much of the breakage seemed to come from outside cinder - many of
the libraries we depend on make backwards incompatible changes by
accident, for example. Would it be possible to have a long-term-support
branch where we pinned the max version of everything for the gate, pips
and devtstack? I'd have thought (and I'm very willing to be corrected)
that would make the stable gate, well, stable, such that it required far
less work to keep it able to run a basic devstack test plus unit tests.

Does that sound at all sane?


A big source of problems IMO is that tempest doesn't have stable
branches. We use the master branch of tempest to test stable branches of
other projects, and tempest regularly adds new features. This guarantees
instability if you rely on tempest anywhere in your gate (and cinder
does).


Orthogonal to the discussion, but: this is not due to the lack of stable
branch, but that part of the Tempest API are not stable yet. This is being
addressed right now (in scope for Newton).
Once the Tempest stable API are used, no breakages should happen.


Well, it’s only partially true. But what happens when you add a new test to
tempest/master? It gets executed on all branches, and maybe some of them
are failing it. We can argue that it’s probably a bug revealed, but it
nevertheless requires attention from stable maintainers to solve.


The new test should work on all support branches. As tester I find a lot of
advantages of maintaining a unified set of tests than fighting with backports
of tests.


I'm sure it makes YOUR life easier to not have to deal with backports. 
The rest of us who do maintain stable branches though don't appreciate 
it when tempest adds a new feature or a new dependency which breaks the 
stable gate jobs. This happens a few times each release, and tends to 
get fixed within a day or two, which is barely tolerable.


The fact that it happens at all though says that we're "doing it wrong" 
w.r.t. testing of stable branches, and completely explains why we find 
it so challenging to support stuff more than 12 months old. If 
everything we used had stable branches and proper dependency management, 
then it would easy to keep gate job running for years.



There are mechanisms to skip tests based on the cloud capabilities.
So this should not be an issue, and if a bug is found that should definitely
be viewed as a good thing.


It's not new tests that cause the problem, because those would be easy 
to skip. It's changes to tempest core which force changes elsewhere.


-Ben



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Luigi Toscano

On Wednesday, 10 August 2016 17:00:36 CEST Ihar Hrachyshka wrote:
> Luigi Toscano  wrote:
> > On Wednesday, 10 August 2016 10:42:41 CEST Ben Swartzlander wrote:
> >> On 08/10/2016 04:33 AM, Duncan Thomas wrote:
> >>> So I tried to get into helping with the cinder stable tree for a while,
> >>> and while I wasn't very successful (lack of time and an inability to
> >>> convince my employer it should be a priority), one thing I did notice it
> >>> that much of the breakage seemed to come from outside cinder - many of
> >>> the libraries we depend on make backwards incompatible changes by
> >>> accident, for example. Would it be possible to have a long-term-support
> >>> branch where we pinned the max version of everything for the gate, pips
> >>> and devtstack? I'd have thought (and I'm very willing to be corrected)
> >>> that would make the stable gate, well, stable, such that it required far
> >>> less work to keep it able to run a basic devstack test plus unit tests.
> >>> 
> >>> Does that sound at all sane?
> >> 
> >> A big source of problems IMO is that tempest doesn't have stable
> >> branches. We use the master branch of tempest to test stable branches of
> >> other projects, and tempest regularly adds new features. This guarantees
> >> instability if you rely on tempest anywhere in your gate (and cinder
> >> does).
> > 
> > Orthogonal to the discussion, but: this is not due to the lack of stable
> > branch, but that part of the Tempest API are not stable yet. This is being
> > addressed right now (in scope for Newton).
> > Once the Tempest stable API are used, no breakages should happen.
> 
> Well, it’s only partially true. But what happens when you add a new test to
> tempest/master? It gets executed on all branches, and maybe some of them
> are failing it. We can argue that it’s probably a bug revealed, but it
> nevertheless requires attention from stable maintainers to solve.

The new test should work on all support branches. As tester I find a lot of 
advantages of maintaining a unified set of tests than fighting with backports 
of tests.
There are mechanisms to skip tests based on the cloud capabilities.
So this should not be an issue, and if a bug is found that should definitely 
be viewed as a good thing.

-- 
Luigi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Hayes, Graham

On 10/08/2016 16:04, Ihar Hrachyshka wrote:
> Luigi Toscano  wrote:
>
>> On Wednesday, 10 August 2016 10:42:41 CEST Ben Swartzlander wrote:
>>> On 08/10/2016 04:33 AM, Duncan Thomas wrote:
 So I tried to get into helping with the cinder stable tree for a while,
 and while I wasn't very successful (lack of time and an inability to
 convince my employer it should be a priority), one thing I did notice it
 that much of the breakage seemed to come from outside cinder - many of
 the libraries we depend on make backwards incompatible changes by
 accident, for example. Would it be possible to have a long-term-support
 branch where we pinned the max version of everything for the gate, pips
 and devtstack? I'd have thought (and I'm very willing to be corrected)
 that would make the stable gate, well, stable, such that it required far
 less work to keep it able to run a basic devstack test plus unit tests.

 Does that sound at all sane?
>>>
>>> A big source of problems IMO is that tempest doesn't have stable
>>> branches. We use the master branch of tempest to test stable branches of
>>> other projects, and tempest regularly adds new features. This guarantees
>>> instability if you rely on tempest anywhere in your gate (and cinder
>>> does).
>>
>> Orthogonal to the discussion, but: this is not due to the lack of stable
>> branch, but that part of the Tempest API are not stable yet. This is being
>> addressed right now (in scope for Newton).
>> Once the Tempest stable API are used, no breakages should happen.
>
> Well, it’s only partially true. But what happens when you add a new test to
> tempest/master? It gets executed on all branches, and maybe some of them
> are failing it. We can argue that it’s probably a bug revealed, but it
> nevertheless requires attention from stable maintainers to solve.

And would a lot of bugs that a new test could expose be non
back-portable? AS they could cause API changes, which is banned in
the back-port policy.

Def-Core faced similar issues with new tests failing previously
validated clouds.

> Ihar
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Ihar Hrachyshka


Luigi Toscano  wrote:


On Wednesday, 10 August 2016 10:42:41 CEST Ben Swartzlander wrote:

On 08/10/2016 04:33 AM, Duncan Thomas wrote:

So I tried to get into helping with the cinder stable tree for a while,
and while I wasn't very successful (lack of time and an inability to
convince my employer it should be a priority), one thing I did notice it
that much of the breakage seemed to come from outside cinder - many of
the libraries we depend on make backwards incompatible changes by
accident, for example. Would it be possible to have a long-term-support
branch where we pinned the max version of everything for the gate, pips
and devtstack? I'd have thought (and I'm very willing to be corrected)
that would make the stable gate, well, stable, such that it required far
less work to keep it able to run a basic devstack test plus unit tests.

Does that sound at all sane?


A big source of problems IMO is that tempest doesn't have stable
branches. We use the master branch of tempest to test stable branches of
other projects, and tempest regularly adds new features. This guarantees
instability if you rely on tempest anywhere in your gate (and cinder  
does).


Orthogonal to the discussion, but: this is not due to the lack of stable
branch, but that part of the Tempest API are not stable yet. This is being
addressed right now (in scope for Newton).
Once the Tempest stable API are used, no breakages should happen.


Well, it’s only partially true. But what happens when you add a new test to  
tempest/master? It gets executed on all branches, and maybe some of them  
are failing it. We can argue that it’s probably a bug revealed, but it  
nevertheless requires attention from stable maintainers to solve.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Luigi Toscano

On Wednesday, 10 August 2016 10:42:41 CEST Ben Swartzlander wrote:
> On 08/10/2016 04:33 AM, Duncan Thomas wrote:
> > So I tried to get into helping with the cinder stable tree for a while,
> > and while I wasn't very successful (lack of time and an inability to
> > convince my employer it should be a priority), one thing I did notice it
> > that much of the breakage seemed to come from outside cinder - many of
> > the libraries we depend on make backwards incompatible changes by
> > accident, for example. Would it be possible to have a long-term-support
> > branch where we pinned the max version of everything for the gate, pips
> > and devtstack? I'd have thought (and I'm very willing to be corrected)
> > that would make the stable gate, well, stable, such that it required far
> > less work to keep it able to run a basic devstack test plus unit tests.
> > 
> > Does that sound at all sane?
> 
> A big source of problems IMO is that tempest doesn't have stable
> branches. We use the master branch of tempest to test stable branches of
> other projects, and tempest regularly adds new features. This guarantees
> instability if you rely on tempest anywhere in your gate (and cinder does).

Orthogonal to the discussion, but: this is not due to the lack of stable 
branch, but that part of the Tempest API are not stable yet. This is being 
addressed right now (in scope for Newton).
Once the Tempest stable API are used, no breakages should happen.

Ciao
-- 
Luigi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Ben Swartzlander

On 08/10/2016 04:33 AM, Duncan Thomas wrote:

So I tried to get into helping with the cinder stable tree for a while,
and while I wasn't very successful (lack of time and an inability to
convince my employer it should be a priority), one thing I did notice it
that much of the breakage seemed to come from outside cinder - many of
the libraries we depend on make backwards incompatible changes by
accident, for example. Would it be possible to have a long-term-support
branch where we pinned the max version of everything for the gate, pips
and devtstack? I'd have thought (and I'm very willing to be corrected)
that would make the stable gate, well, stable, such that it required far
less work to keep it able to run a basic devstack test plus unit tests.

Does that sound at all sane?

A big source of problems IMO is that tempest doesn't have stable 
branches. We use the master branch of tempest to test stable branches of 
other projects, and tempest regularly adds new features. This guarantees 
instability if you rely on tempest anywhere in your gate (and cinder does).

-Ben Swartzlander

(I'm aware there are community standards for stable currently, but a lot
of this thread is the tail of standards wagging the dog of our goals.
Lets figure out what we want to achieve, and figure out how we can do
that without causing either too much extra work or an unnecessary fall
off in quality, rather than saying we can't do anything because of how
we do things now.)

On 10 August 2016 at 08:54, Tony Breeds > wrote:

On Tue, Aug 09, 2016 at 09:16:02PM -0700, John Griffith wrote:
> Sorry, I wasn't a part of the sessions in Austin on the topic of long
> terms support of Cinder drivers.  There's a lot going on during the 
summits
> these days.

For the record the session in Austin, that I think Matt was
referencing,  were
about stable life-cycles. not cinder specific.

> Yeah, ok... I do see your point here, and as I mentioned I have had this
> conversation with you and others over he years and I don't
disagree.  I also don't have the ability to "force"
> said parties to do things differently.  So when I try and help customers
> that are having issues my only recourse is an out of tree patch, which 
then
> when said distro notices or finds out they don't want to support the
> customer any longer based on the code no longer being "their blessed
> code".  The fact is that the distros hold the power in these situations, 
if
> they happen to own the OS release and the storage then it works out great
> for them, not so much for anybody else.

Right we can't 'force' the distros to participate (if we could we
wouldn't be
having this discussion).  The community has a process and all we can
do is
encourage distros and the like to participate in that process as it
really is
best for them, and us.

> So is the consensus here that the only viable solution is for people to
> invest in keeping the stable branches in general supported longer?  How
> does that work for projects that are interested and have people willing to
> do the work vs projects that don't have the people willing to do the work?
> In other words, Cinder has a somewhat unique problem that Nova, Glance and
> Keystone don't have.  So for Cinder to try and follow the policies,
> processes and philosophies you outlined does that mean that as a project
> Cinder has to try and bend the will of "ALL" of the projects to make this
> happen?  Doesn't seem very realistic to me.

So the 'Cinder' team wont need to do all the will bending, that's
for the
Stable team to do with the support of *everyone* that cares about
the outcome.
That probably doens't fill you with hope, but that is the reality.

> Just one last point and I'll move on from the topic.  I'm not sure where
> this illusion that we're testing all the drivers so well is coming from.
> Sure, we require the steps and facade of 3'rd party CI, but dig a bit
> deeper and you soon find that we're not really testing as much as some
> might think here.

That's probbaly true but if we created a 'mitaka-drivers' branch of
cinder the
gate CI would rapidly degernate to a noop any unit/functional tests
would be
*entirely* 3rd party.

Yours Tony.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

--
--
Duncan Thomas

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Tony Breeds

On Wed, Aug 10, 2016 at 11:33:52AM +0300, Duncan Thomas wrote:
> So I tried to get into helping with the cinder stable tree for a while, and
> while I wasn't very successful (lack of time and an inability to convince
> my employer it should be a priority), one thing I did notice it that much
> of the breakage seemed to come from outside cinder - many of the libraries
> we depend on make backwards incompatible changes by accident, for example.
> Would it be possible to have a long-term-support branch where we pinned the
> max version of everything for the gate, pips and devtstack? I'd have
> thought (and I'm very willing to be corrected) that would make the stable
> gate, well, stable, such that it required far less work to keep it able to
> run a basic devstack test plus unit tests.

upper-constraints helps a lot with that.  Did you time looking at the cinder
stable branches pre-date that?

Note we still don't have 100% coverage of projects using upper-constraints,
something we're closing on slowly and everytime we do it we backport it to all
the stbale branches so bit by bit it's getting much better.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Ihar Hrachyshka


Duncan Thomas  wrote:

So I tried to get into helping with the cinder stable tree for a while,  
and while I wasn't very successful (lack of time and an inability to  
convince my employer it should be a priority), one thing I did notice it  
that much of the breakage seemed to come from outside cinder - many of  
the libraries we depend on make backwards incompatible changes by  
accident, for example. Would it be possible to have a long-term-support  
branch where we pinned the max version of everything for the gate, pips  
and devtstack? I'd have thought (and I'm very willing to be corrected)  
that would make the stable gate, well, stable, such that it required far  
less work to keep it able to run a basic devstack test plus unit tests.




The solution for that is in place: upper-constraints.txt, they are not  
bumped with no particular reason in stable/* branches. So just make sure  
that your project uses those for gating, and you should be safe from 99% of  
breakages. Neutron successfully uses the system for all stable branches,  
and believe me, it’s makes a huge difference.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Ihar Hrachyshka


Jeremy Stanley  wrote:


On 2016-08-09 15:56:57 -0700 (-0700), Mike Perez wrote:
As others have said and as being a Cinder stable core myself, the  
status-quo
and this proposal itself are terrible practices because there is no  
testing

behind it, thereby it not being up to the community QA standards set.

[...]

In fairness to Sean, this thread stared because he was asking in
#openstack-infra for help creating some long-lived driver fix
branches because he felt it was against stable branch policy to
backport bugfixes for drivers. Since this was an unprecedented
request, I recommended he first raise the topic on this list to find
out if this is a common problem across other projects and whether
stable branch policy should be revised to permit driver fixes.



I think if infrastructure feels ok to leave this branch that is *not* named  
as stable/* for a longer time, then it should be ok, since stable program  
does not maintain that branch.



There was a brief discussion of what to do if the Cinder team wanted
driver fixes to EOL stable series, and I still firmly believe effort
there is better expended attempting to help extend stable branch
support since "convenience to package maintainers" (what he said
this plan was trying to solve) is the primary reason we provide
those branches to begin with.

So I guess what I'm asking: If stable branches exist as a place for
package maintainers to collaborate on a common set of backported
fixes, and are not actually usable to that end, why do we continue
to provide them?


Those branches *are* usable, for as long as upstream stable program  
participants and infrastructure feels like they can deliver fixes with  
advertised quality standards. Once we feel that we cannot proceed doing it,  
we stop supporting any more bug fixes, irrelevant whether it’s for drivers  
or core code. At that time, it’s up to consumers (distributions, vendors)  
to come up with another out-of-upstream solution on the way forward.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-10 Thread Duncan Thomas

So I tried to get into helping with the cinder stable tree for a while, and
while I wasn't very successful (lack of time and an inability to convince
my employer it should be a priority), one thing I did notice it that much
of the breakage seemed to come from outside cinder - many of the libraries
we depend on make backwards incompatible changes by accident, for example.
Would it be possible to have a long-term-support branch where we pinned the
max version of everything for the gate, pips and devtstack? I'd have
thought (and I'm very willing to be corrected) that would make the stable
gate, well, stable, such that it required far less work to keep it able to
run a basic devstack test plus unit tests.

Does that sound at all sane?

(I'm aware there are community standards for stable currently, but a lot of
this thread is the tail of standards wagging the dog of our goals. Lets
figure out what we want to achieve, and figure out how we can do that
without causing either too much extra work or an unnecessary fall off in
quality, rather than saying we can't do anything because of how we do
things now.)




On 10 August 2016 at 08:54, Tony Breeds  wrote:

> On Tue, Aug 09, 2016 at 09:16:02PM -0700, John Griffith wrote:
> > Sorry, I wasn't a part of the sessions in Austin on the topic of long
> > terms support of Cinder drivers.  There's a lot going on during the
> summits
> > these days.
>
> For the record the session in Austin, that I think Matt was referencing,
> were
> about stable life-cycles. not cinder specific.
>
> > Yeah, ok... I do see your point here, and as I mentioned I have had this
> > conversation with you and others over he years and I don't disagree.  I
> also don't have the ability to "force"
> > said parties to do things differently.  So when I try and help customers
> > that are having issues my only recourse is an out of tree patch, which
> then
> > when said distro notices or finds out they don't want to support the
> > customer any longer based on the code no longer being "their blessed
> > code".  The fact is that the distros hold the power in these situations,
> if
> > they happen to own the OS release and the storage then it works out great
> > for them, not so much for anybody else.
>
> Right we can't 'force' the distros to participate (if we could we wouldn't
> be
> having this discussion).  The community has a process and all we can do is
> encourage distros and the like to participate in that process as it really
> is
> best for them, and us.
>
> > So is the consensus here that the only viable solution is for people to
> > invest in keeping the stable branches in general supported longer?  How
> > does that work for projects that are interested and have people willing
> to
> > do the work vs projects that don't have the people willing to do the
> work?
> > In other words, Cinder has a somewhat unique problem that Nova, Glance
> and
> > Keystone don't have.  So for Cinder to try and follow the policies,
> > processes and philosophies you outlined does that mean that as a project
> > Cinder has to try and bend the will of "ALL" of the projects to make this
> > happen?  Doesn't seem very realistic to me.
>
> So the 'Cinder' team wont need to do all the will bending, that's for the
> Stable team to do with the support of *everyone* that cares about the
> outcome.
> That probably doens't fill you with hope, but that is the reality.
>
> > Just one last point and I'll move on from the topic.  I'm not sure where
> > this illusion that we're testing all the drivers so well is coming from.
> > Sure, we require the steps and facade of 3'rd party CI, but dig a bit
> > deeper and you soon find that we're not really testing as much as some
> > might think here.
>
> That's probbaly true but if we created a 'mitaka-drivers' branch of cinder
> the
> gate CI would rapidly degernate to a noop any unit/functional tests would
> be
> *entirely* 3rd party.
>
> Yours Tony.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Tony Breeds

On Tue, Aug 09, 2016 at 09:16:02PM -0700, John Griffith wrote:
> Sorry, I wasn't a part of the sessions in Austin on the topic of long
> terms support of Cinder drivers.  There's a lot going on during the summits
> these days.

For the record the session in Austin, that I think Matt was referencing,  were
about stable life-cycles. not cinder specific.

> Yeah, ok... I do see your point here, and as I mentioned I have had this
> conversation with you and others over he years and I don't disagree.  I also 
> don't have the ability to "force"
> said parties to do things differently.  So when I try and help customers
> that are having issues my only recourse is an out of tree patch, which then
> when said distro notices or finds out they don't want to support the
> customer any longer based on the code no longer being "their blessed
> code".  The fact is that the distros hold the power in these situations, if
> they happen to own the OS release and the storage then it works out great
> for them, not so much for anybody else.

Right we can't 'force' the distros to participate (if we could we wouldn't be
having this discussion).  The community has a process and all we can do is
encourage distros and the like to participate in that process as it really is
best for them, and us.

> So is the consensus here that the only viable solution is for people to
> invest in keeping the stable branches in general supported longer?  How
> does that work for projects that are interested and have people willing to
> do the work vs projects that don't have the people willing to do the work?
> In other words, Cinder has a somewhat unique problem that Nova, Glance and
> Keystone don't have.  So for Cinder to try and follow the policies,
> processes and philosophies you outlined does that mean that as a project
> Cinder has to try and bend the will of "ALL" of the projects to make this
> happen?  Doesn't seem very realistic to me.

So the 'Cinder' team wont need to do all the will bending, that's for the
Stable team to do with the support of *everyone* that cares about the outcome.
That probably doens't fill you with hope, but that is the reality.

> Just one last point and I'll move on from the topic.  I'm not sure where
> this illusion that we're testing all the drivers so well is coming from.
> Sure, we require the steps and facade of 3'rd party CI, but dig a bit
> deeper and you soon find that we're not really testing as much as some
> might think here.

That's probbaly true but if we created a 'mitaka-drivers' branch of cinder the
gate CI would rapidly degernate to a noop any unit/functional tests would be
*entirely* 3rd party.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread John Griffith

On Tue, Aug 9, 2016 at 10:26 PM, Matthew Treinish 
wrote:

> On Tue, Aug 09, 2016 at 09:16:02PM -0700, John Griffith wrote:
> > On Tue, Aug 9, 2016 at 7:21 PM, Matthew Treinish 
> > wrote:
> >
> > > On Tue, Aug 09, 2016 at 05:28:52PM -0700, John Griffith wrote:
> > > > On Tue, Aug 9, 2016 at 4:53 PM, Sean McGinnis  >
> > > wrote:
> > > >
> > > > > .
> > > > > >
> > > > > > Mike, you must have left the midcycle by the time this topic came
> > > > > > up. On the issue of out-of-tree drivers, I specifically offered
> this
> > > > > > proposal (a community managed mechanism for distributing driver
> > > > > > bugfix backports) as an compromise alternative to try to address
> the
> > > > > > needs of both camps. Everyone who was in the room at the time
> (plus
> > > > > > DuncanT who wasn't) agreed that if we had that (a way to deal
> with
> > > > > > backports) that they wouldn't want drivers out of the tree
> anymore.
> > > > > >
> > > > > > Your point of view wasn't represented so go ahead and explain
> why,
> > > > > > if we did have a reasonable way for bugfixes to get backported to
> > > > > > the releases customers actually run (leaving that mechanism
> > > > > > unspecified for the time being), that you would still want the
> > > > > > drivers out of the tree.
> > > > > >
> > > > > > -Ben Swartzlander
> > > > >
> > > > > The conversation about this started around the 30 minute point
> here if
> > > > > anyone is interested in more of the background discussion on this:
> > > > >
> > > > > https://www.youtube.com/watch?v=g3MEDFp08t4
> > > > >
> > > > > 
> > > __
> > > > > OpenStack Development Mailing List (not for usage questions)
> > > > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> > > unsubscribe
> > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > > >
> > > >
> > > > I don't think anybody is whining at all here, we had a fairly
> productive
> > > > discussion at the mid-cycle surrounding this topic and I do think
> there
> > > are
> > > > some valid advantages to this approach regardless of the QA question.
> > > Note
> > > > that it's been pointed out we weren't talking about or considering
> > > > advertising this *special* branch as tested by the standard means or
> gate
> > > > CI etc.
> > > >
> > > > We did discuss this though mostly in the context of helping the
> package
> > > > maintainers and distributions.  The fact is that many of us currently
> > > offer
> > > > backports of fixes in our own various github accounts.  That's fine
> and
> > > it
> > > > works well for many.  The problem we were trying to address however
> is
> > > that
> > > > this practice is rather problematic for the distros.  For example
> RHEL,
> > > > Helion or Mirantis are most certainly not going to run around cherry
> > > > picking change sets from random github repos scattered around.
> > > >
> > > > The context of the discussion was that by having a long lived
> *driver*
> > > > (emphasis on driver) branch there would be a single location and an
> > > *easy*
> > > > method of contact and communication regarding fixes to drivers that
> may
> > > be
> > > > available for stable branches that are no longer supported.  This
> puts
> > > the
> > > > burden of QA/Testing mostly on the vendors and distros, which I
> think is
> > > > fine.  They can either choose to work with the Vendor and verify the
> > > > versions for backport on a regular basis, or they can choose to
> ignore
> > > them
> > > > and NOT provide them to their customers.
> > > >
> > > > I don't think this is an awful idea, and it's very far from the
> "drivers
> > > > out of tree" discussion.  The feedback from the distro maintainers
> during
> > > > the week was that they would gladly welcome a model where they could
> pull
> > > > updates from a single driver branch on a regular basis or as needed
> for
> > > > customers that are on *unsupported* releases and for whom a fix
> exists.
> > > > Note that support cycles are not the same for the distros as they
> are of
> > > > the upstream community.  This is in no way proposing a change to the
> > > > existing support time frames or processes we have now, and in that
> way it
> > > > differs significantly from proposals and discussions we've had in the
> > > past.
> > > >
> > > > The basic idea here was to eliminate the proliferation of custom
> backport
> > > > patches scattered all over the web, and to ease the burden for
> distros
> > > and
> > > > vendors in supporting their customers.  I think there may be some
> > > concepts
> > > > to iron out and I certainly understand some of the comments regarding
> > > being
> > > > disingenuous regarding what we're advertising.  I think that's a
> > > > misunderstanding of the intent however, the proposal is not to
> extend the
> > > > support life of stable from an

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Matthew Treinish

On Tue, Aug 09, 2016 at 09:16:02PM -0700, John Griffith wrote:
> On Tue, Aug 9, 2016 at 7:21 PM, Matthew Treinish 
> wrote:
> 
> > On Tue, Aug 09, 2016 at 05:28:52PM -0700, John Griffith wrote:
> > > On Tue, Aug 9, 2016 at 4:53 PM, Sean McGinnis 
> > wrote:
> > >
> > > > .
> > > > >
> > > > > Mike, you must have left the midcycle by the time this topic came
> > > > > up. On the issue of out-of-tree drivers, I specifically offered this
> > > > > proposal (a community managed mechanism for distributing driver
> > > > > bugfix backports) as an compromise alternative to try to address the
> > > > > needs of both camps. Everyone who was in the room at the time (plus
> > > > > DuncanT who wasn't) agreed that if we had that (a way to deal with
> > > > > backports) that they wouldn't want drivers out of the tree anymore.
> > > > >
> > > > > Your point of view wasn't represented so go ahead and explain why,
> > > > > if we did have a reasonable way for bugfixes to get backported to
> > > > > the releases customers actually run (leaving that mechanism
> > > > > unspecified for the time being), that you would still want the
> > > > > drivers out of the tree.
> > > > >
> > > > > -Ben Swartzlander
> > > >
> > > > The conversation about this started around the 30 minute point here if
> > > > anyone is interested in more of the background discussion on this:
> > > >
> > > > https://www.youtube.com/watch?v=g3MEDFp08t4
> > > >
> > > > 
> > __
> > > > OpenStack Development Mailing List (not for usage questions)
> > > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> > unsubscribe
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > >
> > >
> > > I don't think anybody is whining at all here, we had a fairly productive
> > > discussion at the mid-cycle surrounding this topic and I do think there
> > are
> > > some valid advantages to this approach regardless of the QA question.
> > Note
> > > that it's been pointed out we weren't talking about or considering
> > > advertising this *special* branch as tested by the standard means or gate
> > > CI etc.
> > >
> > > We did discuss this though mostly in the context of helping the package
> > > maintainers and distributions.  The fact is that many of us currently
> > offer
> > > backports of fixes in our own various github accounts.  That's fine and
> > it
> > > works well for many.  The problem we were trying to address however is
> > that
> > > this practice is rather problematic for the distros.  For example RHEL,
> > > Helion or Mirantis are most certainly not going to run around cherry
> > > picking change sets from random github repos scattered around.
> > >
> > > The context of the discussion was that by having a long lived *driver*
> > > (emphasis on driver) branch there would be a single location and an
> > *easy*
> > > method of contact and communication regarding fixes to drivers that may
> > be
> > > available for stable branches that are no longer supported.  This puts
> > the
> > > burden of QA/Testing mostly on the vendors and distros, which I think is
> > > fine.  They can either choose to work with the Vendor and verify the
> > > versions for backport on a regular basis, or they can choose to ignore
> > them
> > > and NOT provide them to their customers.
> > >
> > > I don't think this is an awful idea, and it's very far from the "drivers
> > > out of tree" discussion.  The feedback from the distro maintainers during
> > > the week was that they would gladly welcome a model where they could pull
> > > updates from a single driver branch on a regular basis or as needed for
> > > customers that are on *unsupported* releases and for whom a fix exists.
> > > Note that support cycles are not the same for the distros as they are of
> > > the upstream community.  This is in no way proposing a change to the
> > > existing support time frames or processes we have now, and in that way it
> > > differs significantly from proposals and discussions we've had in the
> > past.
> > >
> > > The basic idea here was to eliminate the proliferation of custom backport
> > > patches scattered all over the web, and to ease the burden for distros
> > and
> > > vendors in supporting their customers.  I think there may be some
> > concepts
> > > to iron out and I certainly understand some of the comments regarding
> > being
> > > disingenuous regarding what we're advertising.  I think that's a
> > > misunderstanding of the intent however, the proposal is not to extend the
> > > support life of stable from an upstream or community perspective but
> > > instead the proposal is geared at consolidation and tracking of drivers.
> >
> > I fully understood the proposal but I still think you're optimizing for the
> > wrong thing.
> 
> Ok, that's fair. It seemed like there might be some confusion with some of
> the comments that were made.

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread John Griffith

On Tue, Aug 9, 2016 at 7:21 PM, Matthew Treinish 
wrote:

> On Tue, Aug 09, 2016 at 05:28:52PM -0700, John Griffith wrote:
> > On Tue, Aug 9, 2016 at 4:53 PM, Sean McGinnis 
> wrote:
> >
> > > .
> > > >
> > > > Mike, you must have left the midcycle by the time this topic came
> > > > up. On the issue of out-of-tree drivers, I specifically offered this
> > > > proposal (a community managed mechanism for distributing driver
> > > > bugfix backports) as an compromise alternative to try to address the
> > > > needs of both camps. Everyone who was in the room at the time (plus
> > > > DuncanT who wasn't) agreed that if we had that (a way to deal with
> > > > backports) that they wouldn't want drivers out of the tree anymore.
> > > >
> > > > Your point of view wasn't represented so go ahead and explain why,
> > > > if we did have a reasonable way for bugfixes to get backported to
> > > > the releases customers actually run (leaving that mechanism
> > > > unspecified for the time being), that you would still want the
> > > > drivers out of the tree.
> > > >
> > > > -Ben Swartzlander
> > >
> > > The conversation about this started around the 30 minute point here if
> > > anyone is interested in more of the background discussion on this:
> > >
> > > https://www.youtube.com/watch?v=g3MEDFp08t4
> > >
> > > 
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> > I don't think anybody is whining at all here, we had a fairly productive
> > discussion at the mid-cycle surrounding this topic and I do think there
> are
> > some valid advantages to this approach regardless of the QA question.
> Note
> > that it's been pointed out we weren't talking about or considering
> > advertising this *special* branch as tested by the standard means or gate
> > CI etc.
> >
> > We did discuss this though mostly in the context of helping the package
> > maintainers and distributions.  The fact is that many of us currently
> offer
> > backports of fixes in our own various github accounts.  That's fine and
> it
> > works well for many.  The problem we were trying to address however is
> that
> > this practice is rather problematic for the distros.  For example RHEL,
> > Helion or Mirantis are most certainly not going to run around cherry
> > picking change sets from random github repos scattered around.
> >
> > The context of the discussion was that by having a long lived *driver*
> > (emphasis on driver) branch there would be a single location and an
> *easy*
> > method of contact and communication regarding fixes to drivers that may
> be
> > available for stable branches that are no longer supported.  This puts
> the
> > burden of QA/Testing mostly on the vendors and distros, which I think is
> > fine.  They can either choose to work with the Vendor and verify the
> > versions for backport on a regular basis, or they can choose to ignore
> them
> > and NOT provide them to their customers.
> >
> > I don't think this is an awful idea, and it's very far from the "drivers
> > out of tree" discussion.  The feedback from the distro maintainers during
> > the week was that they would gladly welcome a model where they could pull
> > updates from a single driver branch on a regular basis or as needed for
> > customers that are on *unsupported* releases and for whom a fix exists.
> > Note that support cycles are not the same for the distros as they are of
> > the upstream community.  This is in no way proposing a change to the
> > existing support time frames or processes we have now, and in that way it
> > differs significantly from proposals and discussions we've had in the
> past.
> >
> > The basic idea here was to eliminate the proliferation of custom backport
> > patches scattered all over the web, and to ease the burden for distros
> and
> > vendors in supporting their customers.  I think there may be some
> concepts
> > to iron out and I certainly understand some of the comments regarding
> being
> > disingenuous regarding what we're advertising.  I think that's a
> > misunderstanding of the intent however, the proposal is not to extend the
> > support life of stable from an upstream or community perspective but
> > instead the proposal is geared at consolidation and tracking of drivers.
>
> I fully understood the proposal but I still think you're optimizing for the
> wrong thing.

Ok, that's fair. It seemed like there might be some confusion with some of
the comments that were made.
 


> We have a community process for doing backports and maintaining
> released versions of OpenStack code. The fundamental problem here is
> actually
> that the parties you've identified aren't actively involved in stable
> branch
> maintenance.

Yes, I

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Tony Breeds

On Wed, Aug 10, 2016 at 01:39:55AM +, Jeremy Stanley wrote:

> So I guess what I'm asking: If stable branches exist as a place for
> package maintainers to collaborate on a common set of backported
> fixes, and are not actually usable to that end, why do we continue
> to provide them?

I don't this has a binary answer.  It's a scale.  The branches we have do have
value but they just don't last long enough to cover the '5 year support
contract' use case, but I think we're basically on the same page there.

If we could support release for longer, we would.

Yours Tony.

signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Tony Breeds

On Tue, Aug 09, 2016 at 10:21:19PM -0400, Matthew Treinish wrote:

> I fully understood the proposal but I still think you're optimizing for the
> wrong thing. We have a community process for doing backports and maintaining
> released versions of OpenStack code. The fundamental problem here is actually
> that the parties you've identified aren't actively involved in stable branch
> maintenance. The stable maint team and policy was primarily created as a
> solution to the exact problem you outlined above, that it providing a place 
> for
> vendors, distros, etc to collaborate on backports and stable branch maint.
> while following our communities process. Regardless of framing it as being 
> only
> for drivers it doesn't change that you're talking about the same thing. (this 
> is
> why in-tree vs out-of-tree was coming up from others, because if it was
> out-of-tree then you don't have to respect the stable policy)
> 
> What I was getting at before was if instead of ignoring all the previous
> conversations we've had about this exact topic (including 2 sessions in 
> Austin)
> and people actually stepped up and got involved we wouldn't be having this
> discussion today. More than likely we'd have enough people to actually 
> maintain
> a stable branch for longer than we can today. But, instead it feels like our
> community process for code review and actually testing proposed backports is 
> too
> much of a burden for any of these parties you've identified to bother getting
> involved. So we're instead in a situation where we have a proposal to actively
> circumvent our community policy and procedures for maintaining stable 
> branches.
> 
> Creating a free space where vendors can push whatever they want without any
> gating is not only contrary to our stable branch policy but also goes against
> some of the fundamental tenants of OpenStack. It's not something I think we
> should ever be doing.
> 
> > 
> > If this isn't something we can come to an agreement on as a community, then
> > I'd suggest we just create our own repo on github outside of upstream and
> > have it serve the same purpose.
> > 
> 
> If you're dead set on doing this then I think that's your only recourse, 
> because
> what you're attempting to do is something outside our community processes and
> I don't think it has any place in upstream. But, I really think it'd be much
> better if instead of going off into a corner that all of the people who have
> complained about the state of our current stable support windows and stable
> policy actually got involved in this space. That way over time we can make
> the stable branches something where we can have longer support windows. 
> 
> -Matt Treinish

I appologise for weighing in so late.  This is a complex issue with no answer 
today :(

I think think Matt's email neatly sums up my position.  The stable policy is
the way it is to balance support requirements and community effort.  Any effort
to extend/alter the stable life-cycle needs to start *now* with bodies on the
ground in cinder, infra and the stable team working together to enable this
goal.  A patch to the policy isn't really going to cut it.

Even splitting the drivers out wont work long term, without the effort on
stable support.

I've advocated for the last 12months to lengthen the support cycles, and will
do again as soon as I feel the balance tip towards success.

In short come to the stable team ... we have cookies[1]

Yours Tony.

[1] Collectable at summits, and select mid-cycles


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Matthew Treinish

On Wed, Aug 10, 2016 at 01:39:55AM +, Jeremy Stanley wrote:
> On 2016-08-09 15:56:57 -0700 (-0700), Mike Perez wrote:
> > As others have said and as being a Cinder stable core myself, the status-quo
> > and this proposal itself are terrible practices because there is no testing
> > behind it, thereby it not being up to the community QA standards set.
> [...]
> 
> In fairness to Sean, this thread stared because he was asking in
> #openstack-infra for help creating some long-lived driver fix
> branches because he felt it was against stable branch policy to
> backport bugfixes for drivers. Since this was an unprecedented
> request, I recommended he first raise the topic on this list to find
> out if this is a common problem across other projects and whether
> stable branch policy should be revised to permit driver fixes.
> 
> There was a brief discussion of what to do if the Cinder team wanted
> driver fixes to EOL stable series, and I still firmly believe effort
> there is better expended attempting to help extend stable branch
> support since "convenience to package maintainers" (what he said
> this plan was trying to solve) is the primary reason we provide
> those branches to begin with.
> 
> So I guess what I'm asking: If stable branches exist as a place for
> package maintainers to collaborate on a common set of backported
> fixes, and are not actually usable to that end, why do we continue
> to provide them? Should we just stop testing stable branches
> altogether since their primary value would (as is suggested) be
> served even without our testing efforts? Ceasing any attempts to
> test backports post-release would certainly free up a lot of our
> current upstream effort and resources we could redirect into other
> priorities. Or is it just stable branch changes for drivers we
> shouldn't bother testing?

Well, at a bare minimum we need the previous release around to test upgrades
which is very important. But, this exact argument has come up in the past when
we've had this exact discussion before. (it's been at least once a cycle for as
long as I can remember) I might have actually proposed having only one stable
branch at a time during one of the past summits. But, every time it's been
proposed in the past people come out of the woodwork and say there is value in
continuing them, so we've continued maintaining them. I do agree though, that it
does feel like there is a disconnect between downstream consumers and upstream
when we get proposals like this while at the same time we have had recent quite
lengthy discussions where we decided not to extend our support windows because
it's not feasible given the level of activity.

As for not testing stable changes for drivers I fundamentally disagree with any
approach that puts us in a situation where we are landing patches in an
OpenStack project that does not have any testing. This is a core part of doing
development "the OpenStack way", (to quote the governance repo) if the driver
code is part of the project then we need to be testing it.

-Matt Treinish

signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Matthew Treinish

On Tue, Aug 09, 2016 at 05:28:52PM -0700, John Griffith wrote:
> On Tue, Aug 9, 2016 at 4:53 PM, Sean McGinnis  wrote:
> 
> > .
> > >
> > > Mike, you must have left the midcycle by the time this topic came
> > > up. On the issue of out-of-tree drivers, I specifically offered this
> > > proposal (a community managed mechanism for distributing driver
> > > bugfix backports) as an compromise alternative to try to address the
> > > needs of both camps. Everyone who was in the room at the time (plus
> > > DuncanT who wasn't) agreed that if we had that (a way to deal with
> > > backports) that they wouldn't want drivers out of the tree anymore.
> > >
> > > Your point of view wasn't represented so go ahead and explain why,
> > > if we did have a reasonable way for bugfixes to get backported to
> > > the releases customers actually run (leaving that mechanism
> > > unspecified for the time being), that you would still want the
> > > drivers out of the tree.
> > >
> > > -Ben Swartzlander
> >
> > The conversation about this started around the 30 minute point here if
> > anyone is interested in more of the background discussion on this:
> >
> > https://www.youtube.com/watch?v=g3MEDFp08t4
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> I don't think anybody is whining at all here, we had a fairly productive
> discussion at the mid-cycle surrounding this topic and I do think there are
> some valid advantages to this approach regardless of the QA question.  Note
> that it's been pointed out we weren't talking about or considering
> advertising this *special* branch as tested by the standard means or gate
> CI etc.
> 
> We did discuss this though mostly in the context of helping the package
> maintainers and distributions.  The fact is that many of us currently offer
> backports of fixes in our own various github accounts.  That's fine and it
> works well for many.  The problem we were trying to address however is that
> this practice is rather problematic for the distros.  For example RHEL,
> Helion or Mirantis are most certainly not going to run around cherry
> picking change sets from random github repos scattered around.
> 
> The context of the discussion was that by having a long lived *driver*
> (emphasis on driver) branch there would be a single location and an *easy*
> method of contact and communication regarding fixes to drivers that may be
> available for stable branches that are no longer supported.  This puts the
> burden of QA/Testing mostly on the vendors and distros, which I think is
> fine.  They can either choose to work with the Vendor and verify the
> versions for backport on a regular basis, or they can choose to ignore them
> and NOT provide them to their customers.
> 
> I don't think this is an awful idea, and it's very far from the "drivers
> out of tree" discussion.  The feedback from the distro maintainers during
> the week was that they would gladly welcome a model where they could pull
> updates from a single driver branch on a regular basis or as needed for
> customers that are on *unsupported* releases and for whom a fix exists.
> Note that support cycles are not the same for the distros as they are of
> the upstream community.  This is in no way proposing a change to the
> existing support time frames or processes we have now, and in that way it
> differs significantly from proposals and discussions we've had in the past.
> 
> The basic idea here was to eliminate the proliferation of custom backport
> patches scattered all over the web, and to ease the burden for distros and
> vendors in supporting their customers.  I think there may be some concepts
> to iron out and I certainly understand some of the comments regarding being
> disingenuous regarding what we're advertising.  I think that's a
> misunderstanding of the intent however, the proposal is not to extend the
> support life of stable from an upstream or community perspective but
> instead the proposal is geared at consolidation and tracking of drivers.

I fully understood the proposal but I still think you're optimizing for the
wrong thing. We have a community process for doing backports and maintaining
released versions of OpenStack code. The fundamental problem here is actually
that the parties you've identified aren't actively involved in stable branch
maintenance. The stable maint team and policy was primarily created as a
solution to the exact problem you outlined above, that it providing a place for
vendors, distros, etc to collaborate on backports and stable branch maint.
while following our communities process. Regardless of framing it as being only
for drivers it doesn't change that you're talking about the same thing. (this is
why in-tree vs

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Jeremy Stanley

On 2016-08-09 15:56:57 -0700 (-0700), Mike Perez wrote:
> As others have said and as being a Cinder stable core myself, the status-quo
> and this proposal itself are terrible practices because there is no testing
> behind it, thereby it not being up to the community QA standards set.
[...]

In fairness to Sean, this thread stared because he was asking in
#openstack-infra for help creating some long-lived driver fix
branches because he felt it was against stable branch policy to
backport bugfixes for drivers. Since this was an unprecedented
request, I recommended he first raise the topic on this list to find
out if this is a common problem across other projects and whether
stable branch policy should be revised to permit driver fixes.

There was a brief discussion of what to do if the Cinder team wanted
driver fixes to EOL stable series, and I still firmly believe effort
there is better expended attempting to help extend stable branch
support since "convenience to package maintainers" (what he said
this plan was trying to solve) is the primary reason we provide
those branches to begin with.

So I guess what I'm asking: If stable branches exist as a place for
package maintainers to collaborate on a common set of backported
fixes, and are not actually usable to that end, why do we continue
to provide them? Should we just stop testing stable branches
altogether since their primary value would (as is suggested) be
served even without our testing efforts? Ceasing any attempts to
test backports post-release would certainly free up a lot of our
current upstream effort and resources we could redirect into other
priorities. Or is it just stable branch changes for drivers we
shouldn't bother testing?
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread John Griffith

On Tue, Aug 9, 2016 at 4:53 PM, Sean McGinnis  wrote:

> .
> >
> > Mike, you must have left the midcycle by the time this topic came
> > up. On the issue of out-of-tree drivers, I specifically offered this
> > proposal (a community managed mechanism for distributing driver
> > bugfix backports) as an compromise alternative to try to address the
> > needs of both camps. Everyone who was in the room at the time (plus
> > DuncanT who wasn't) agreed that if we had that (a way to deal with
> > backports) that they wouldn't want drivers out of the tree anymore.
> >
> > Your point of view wasn't represented so go ahead and explain why,
> > if we did have a reasonable way for bugfixes to get backported to
> > the releases customers actually run (leaving that mechanism
> > unspecified for the time being), that you would still want the
> > drivers out of the tree.
> >
> > -Ben Swartzlander
>
> The conversation about this started around the 30 minute point here if
> anyone is interested in more of the background discussion on this:
>
> https://www.youtube.com/watch?v=g3MEDFp08t4
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

I don't think anybody is whining at all here, we had a fairly productive
discussion at the mid-cycle surrounding this topic and I do think there are
some valid advantages to this approach regardless of the QA question.  Note
that it's been pointed out we weren't talking about or considering
advertising this *special* branch as tested by the standard means or gate
CI etc.

We did discuss this though mostly in the context of helping the package
maintainers and distributions.  The fact is that many of us currently offer
backports of fixes in our own various github accounts.  That's fine and it
works well for many.  The problem we were trying to address however is that
this practice is rather problematic for the distros.  For example RHEL,
Helion or Mirantis are most certainly not going to run around cherry
picking change sets from random github repos scattered around.

The context of the discussion was that by having a long lived *driver*
(emphasis on driver) branch there would be a single location and an *easy*
method of contact and communication regarding fixes to drivers that may be
available for stable branches that are no longer supported.  This puts the
burden of QA/Testing mostly on the vendors and distros, which I think is
fine.  They can either choose to work with the Vendor and verify the
versions for backport on a regular basis, or they can choose to ignore them
and NOT provide them to their customers.

I don't think this is an awful idea, and it's very far from the "drivers
out of tree" discussion.  The feedback from the distro maintainers during
the week was that they would gladly welcome a model where they could pull
updates from a single driver branch on a regular basis or as needed for
customers that are on *unsupported* releases and for whom a fix exists.
Note that support cycles are not the same for the distros as they are of
the upstream community.  This is in no way proposing a change to the
existing support time frames or processes we have now, and in that way it
differs significantly from proposals and discussions we've had in the past.

The basic idea here was to eliminate the proliferation of custom backport
patches scattered all over the web, and to ease the burden for distros and
vendors in supporting their customers.  I think there may be some concepts
to iron out and I certainly understand some of the comments regarding being
disingenuous regarding what we're advertising.  I think that's a
misunderstanding of the intent however, the proposal is not to extend the
support life of stable from an upstream or community perspective but
instead the proposal is geared at consolidation and tracking of drivers.

If this isn't something we can come to an agreement on as a community, then
I'd suggest we just create our own repo on github outside of upstream and
have it serve the same purpose.

Thanks,
John 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Sean McGinnis

.
> 
> Mike, you must have left the midcycle by the time this topic came
> up. On the issue of out-of-tree drivers, I specifically offered this
> proposal (a community managed mechanism for distributing driver
> bugfix backports) as an compromise alternative to try to address the
> needs of both camps. Everyone who was in the room at the time (plus
> DuncanT who wasn't) agreed that if we had that (a way to deal with
> backports) that they wouldn't want drivers out of the tree anymore.
> 
> Your point of view wasn't represented so go ahead and explain why,
> if we did have a reasonable way for bugfixes to get backported to
> the releases customers actually run (leaving that mechanism
> unspecified for the time being), that you would still want the
> drivers out of the tree.
> 
> -Ben Swartzlander

The conversation about this started around the 30 minute point here if
anyone is interested in more of the background discussion on this:

https://www.youtube.com/watch?v=g3MEDFp08t4 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Ben Swartzlander


On 08/09/2016 06:56 PM, Mike Perez wrote:

On 10:31 Aug 06, Sean McGinnis wrote:


I'm open and welcome to any feedback on this. Unless there are any major
concerns raised, I will at least instruct any Cinder stable cores to
start allowing these bugfix patches through past the security only
phase.


As others have said and as being a Cinder stable core myself, the status-quo
and this proposal itself are terrible practices because there is no testing
behind it, thereby it not being up to the community QA standards set. I will be
issuing -2 on these changes in the stable branch regardless of your
instructions until the policy has changed.


I agree we can't drop the testing standards on the "stable" branches. 
I'm not in favor of that. I'd rather use differently-named branches with 
different and well-documented policies. Ideally the branches would be 
named something like driver-fixes/newton and driver-fixes/ocata, etc, to 
avoid confusion with the stable branches.


-Ben Swartzlander


If you want to change that, work with the stable team on the various options
provided. This tangent of people whining on the mailing list and in
#openstack-cinder is not going to accomplish anything.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Sean McGinnis

On Tue, Aug 09, 2016 at 03:56:57PM -0700, Mike Perez wrote:

> If you want to change that, work with the stable team on the various options
> provided. This tangent of people whining on the mailing list and in
> #openstack-cinder is not going to accomplish anything.

That's what we're doing here

> 
> -- 
> Mike Perez
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Ben Swartzlander


On 08/09/2016 05:45 PM, Mike Perez wrote:

On 19:40 Aug 08, Duncan Thomas wrote:

On 8 August 2016 at 18:31, Matthew Treinish  wrote:



This argument comes up at least once a cycle and there is a reason we
don't do
this. When we EOL a branch all of the infrastructure for running any ci
against
it goes away. This means devstack support, job definitions, tempest skip
checks,
etc. Leaving the branch around advertises that you can still submit
patches to
it which you can't anymore. As a community we've very clearly said that we
don't
land any code without ensuring it passes tests first, and we do not
maintain any
of the infrastructure for doing that after an EOL.



Ok, to turn the question around, we (the cinder team) have recognised a
definite and strong need to have somewhere for vendors to share patches on
versions of Cinder older than the stable branch policy allows.

Given this need, what are our options?

1. We could do all this outside Openstack infrastructure. There are
significant downsides to doing so from organisational, maintenance, cost
etc points of view. Also means that the place vendors go for these patches
is not obvious, and the process for getting patches in is not standard.

2. We could have something not named 'stable' that has looser rules than
stable branches,, maybe just pep8 / unit / cinder in-tree tests. No
devstack.

3. We go with the Neutron model and take drivers out of tree. This is not
something the cinder core team are in favour of - we see significant value
in the code review that drivers currently get - the code quality
improvements between when a driver is submitted and when it is merged are
sometimes very significant. Also, taking the code out of tree makes it
difficult to get all the drivers checked out in one place to analyse e.g.
how a certain driver call is implemented across all the drivers, when
reasoning or making changes to core code.


Just to set the record straight here, some Cinder core members are in favor of
out of tree.


Mike, you must have left the midcycle by the time this topic came up. On 
the issue of out-of-tree drivers, I specifically offered this proposal 
(a community managed mechanism for distributing driver bugfix backports) 
as an compromise alternative to try to address the needs of both camps. 
Everyone who was in the room at the time (plus DuncanT who wasn't) 
agreed that if we had that (a way to deal with backports) that they 
wouldn't want drivers out of the tree anymore.


Your point of view wasn't represented so go ahead and explain why, if we 
did have a reasonable way for bugfixes to get backported to the releases 
customers actually run (leaving that mechanism unspecified for the time 
being), that you would still want the drivers out of the tree.


-Ben Swartzlander


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Mike Perez

On 10:31 Aug 06, Sean McGinnis wrote:

> I'm open and welcome to any feedback on this. Unless there are any major
> concerns raised, I will at least instruct any Cinder stable cores to
> start allowing these bugfix patches through past the security only
> phase.

As others have said and as being a Cinder stable core myself, the status-quo
and this proposal itself are terrible practices because there is no testing
behind it, thereby it not being up to the community QA standards set. I will be
issuing -2 on these changes in the stable branch regardless of your
instructions until the policy has changed.

If you want to change that, work with the stable team on the various options
provided. This tangent of people whining on the mailing list and in
#openstack-cinder is not going to accomplish anything.

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Mike Perez

On 19:40 Aug 08, Duncan Thomas wrote:
> On 8 August 2016 at 18:31, Matthew Treinish  wrote:
> 
> >
> > This argument comes up at least once a cycle and there is a reason we
> > don't do
> > this. When we EOL a branch all of the infrastructure for running any ci
> > against
> > it goes away. This means devstack support, job definitions, tempest skip
> > checks,
> > etc. Leaving the branch around advertises that you can still submit
> > patches to
> > it which you can't anymore. As a community we've very clearly said that we
> > don't
> > land any code without ensuring it passes tests first, and we do not
> > maintain any
> > of the infrastructure for doing that after an EOL.
> >
> >
> Ok, to turn the question around, we (the cinder team) have recognised a
> definite and strong need to have somewhere for vendors to share patches on
> versions of Cinder older than the stable branch policy allows.
> 
> Given this need, what are our options?
> 
> 1. We could do all this outside Openstack infrastructure. There are
> significant downsides to doing so from organisational, maintenance, cost
> etc points of view. Also means that the place vendors go for these patches
> is not obvious, and the process for getting patches in is not standard.
> 
> 2. We could have something not named 'stable' that has looser rules than
> stable branches,, maybe just pep8 / unit / cinder in-tree tests. No
> devstack.
> 
> 3. We go with the Neutron model and take drivers out of tree. This is not
> something the cinder core team are in favour of - we see significant value
> in the code review that drivers currently get - the code quality
> improvements between when a driver is submitted and when it is merged are
> sometimes very significant. Also, taking the code out of tree makes it
> difficult to get all the drivers checked out in one place to analyse e.g.
> how a certain driver call is implemented across all the drivers, when
> reasoning or making changes to core code.

Just to set the record straight here, some Cinder core members are in favor of
out of tree.

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Chris Friesen


On 08/09/2016 02:10 PM, Ben Swartzlander wrote:


The best example of why this is good is Linux. If you tell the Linux people to
take their drivers out of the tree I can guarantee you they'll laugh you out of
the room. The reasons for their stance are many and I won't recount them here
(unless you want me to).


Just to play devil's advocate, the latest Intel ethernet drivers are generally 
found at https://sourceforge.net/projects/e1000


These drivers often have fixes/features that haven't made it into the kernel 
yet, and are compilable against a variety of kernel versions rather than just 
the most recent.


Of course the same developers also maintain in-tree kernel drivers, but changes 
to the in-tree drivers only affect the currently-maintained subset of kernel 
branches.


Other vendors also maintain out-of-tree drivers as well as in-tree drivers, so 
the situation is exactly analogous to Cinder.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Walter A. Boring IV


On 08/09/2016 11:52 AM, Ihar Hrachyshka wrote:

Walter A. Boring IV  wrote:


On 08/08/2016 02:28 PM, Ihar Hrachyshka wrote:

Duncan Thomas  wrote:

On 8 August 2016 at 21:12, Matthew Treinish  
wrote:
Ignoring all that, this is also contrary to how we perform testing 
in OpenStack.
We don't turn off entire classes of testing we have so we can land 
patches,

that's just a recipe for disaster.

But is it more of a disaster (for the consumers) than zero testing, 
zero review, scattered around the internet 
if-you're-lucky-with-a-good-wind you'll maybe get the right patch 
set? Because that's where we are right now, and vendors, 
distributors and the cinder core team are all saying it's a disaster.


If consumers rely on upstream releases, then they are expected to 
migrate to newer releases after EOL, not switch to a random branch 
on the internet. If they rely on some commercial product, then they 
usually have an extended period of support and certification for 
their drivers, so it’s not a problem for them.


Ihar
This is entirely unrealistic.  Force customers to upgrade. Good luck 
explaining to a bank that in order to get their cinder driver fix in, 
they have to upgrade their entire OpenStack deployment. Real world 
customers simply will balk at this all day long.


Real world customers will pay for engineering to support their 
software, either their own or of one of OpenStack vendors. There is no 
free lunch from upstream here.


  Our customers are already paying us to support them and it's what we 
are doing.  Nobody is asking for a free lunch from upstream.  We are 
simply asking for a way to have a centralized repository that each 
vendor uses to support their drivers.


The problem is how to get customers patches against older drivers and 
then support following that.  We have no place to centrally place our 
patches against our driver other than our forked github account for 
older releases.   This is exactly what the rest of the Cinder driver 
vendors are doing, and is what we are trying to avoid.  The problem even 
gets worse when a customer has a LeftHand array and a SolidFire and/or 
Netapp and/or Pure array.  The customer will have to get fixes from each 
separate repository and monitor each of those for changes in the 
future.   Which fork to they follow?  This is utter chaos from a 
customer perspective as well as a distributor's perspective and is 
terrible for OpenStack users/deployers.



Walt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Ben Swartzlander


On 08/09/2016 03:01 PM, Ihar Hrachyshka wrote:

Walter A. Boring IV  wrote:




I think "currently active stable branches" is key there. These branches
would no longer be "currently active". They would get an EOL tag
when it
reaches the end of the support phases. We just wouldn't delete the
branch.

This argument comes up at least once a cycle and there is a reason we
don't do
this. When we EOL a branch all of the infrastructure for running any
ci against
it goes away. This means devstack support, job definitions, tempest
skip checks,
etc. Leaving the branch around advertises that you can still submit
patches to
it which you can't anymore. As a community we've very clearly said
that we don't
land any code without ensuring it passes tests first, and we do not
maintain any
of the infrastructure for doing that after an EOL.


And it's this exact policy that has lead us to this mess we are in
today.   As a vendor that has customers that use OpenStack, we have to
support very old releases.  Customers in the wild do not like to
upgrade once they get OpenStack up and running because it's very
difficult, time consuming and dangerous to do.  We have customers
still running Icehouse and they will most likely won't upgrade any
time soon.  Banks hate upgrading software after they have customers
running on it.   This is a community wide problem that needs to be
addressed.

Because of this problem, (not being able to backport bug fixes in our
drivers), we have been left with forking Cinder on our own github to
put our driver fixes there.   This is a terrible practice for the
OpenStack community in general, and terrible for customers/users of
OpenStack, as we have N driver vendors that have N different
mechanisms for getting bug fixes to their customers.  I believe this
is a major problem for users of OpenStack and it needs to be addressed.


Right. And so the proper solution in line with OpenStack practices would
be to allow vendors to own their plugins and maintain them, potentially
for extensive time. The fact the Cinder team is unwilling [as it seems
like from what I read in other replies to the thread] to provide that
kind of extensibility to vendors is unfortunate. BTW do we have a
write-up of reasons behind that?


I'm not sure what OpenStack practices you're referring to. The only 
project I'm aware of which does out of tree drivers is Neutron, and 
Neutron is a very different kind of project architecture that's not very 
comparable to Cinder. Most projects keep their drivers in tree.


The best example of why this is good is Linux. If you tell the Linux 
people to take their drivers out of the tree I can guarantee you they'll 
laugh you out of the room. The reasons for their stance are many and I 
won't recount them here (unless you want me to).



At the Cinder midcycle, we came up with a solution that would satisfy
Cinder customers, as Sean planned out.


All of them? I am not sure on that one. Some consumers may put some
trust into stable branches, and may be surprised by patches landing
there without upstream CI in place, undermining the promise the project
made by adopting the stable:follows-policy tag.


The main customers for bugfixes are OpenStack distros (like RHOS).

Only a tiny minority of users deploy from upstream and those users tend 
to be sophisticated enough to do their own backports or to pull from 
vendor's forked repos anyways.


What we're trying to achieve is to make the lives of the distro 
maintainers easier by creating a clearing house for bugfix backports 
which is owned and operating by the upstream community.


Regarding the stable:follows-policy tag, I never proposed using the 
stable branches -- I proposed creating different branches specifically 
to avoid the surprise you're alluding to. The infra team suggested that 
maybe reusing the stable branches was a better option.



We acknowledge that it's a driver maintainer's responsibility to make
sure they test any changes that get into the stable branches, because
there is no infra support for running CI against the patches of old
stable branches. I think that risk is far better than the existing
reality of N cinder forks floating around github.   It's just no way
to ship software to actual customers.


If Cinder would give you a right hooking entry point for your plugin,
you would not need to fork the whole thing just to extend your small
isolated bit. You would live on upstream infrastructure while stable/*
is alive, then move to your own git repo somewhere on the internet to
provide more bug fixes [as some vendors do for neutron drivers].


In theory you're right. We never touch stuff outside the drivers when we 
backport fixes. However, to maintain proper git histories and to be able 
to run the unit tests, etc, it's logistically simpler to fork the whole 
repo. It's trivially easy to verify that the all of the differences 
between a fork and its parent are limited to just the driver 
subdirectory, so in

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Hayes, Graham

On 09/08/2016 19:58, Ihar Hrachyshka wrote:
> Walter A. Boring IV  wrote:
>
>> On 08/08/2016 02:28 PM, Ihar Hrachyshka wrote:
>>> Duncan Thomas  wrote:
>>>
 On 8 August 2016 at 21:12, Matthew Treinish  wrote:
 Ignoring all that, this is also contrary to how we perform testing in
 OpenStack.
 We don't turn off entire classes of testing we have so we can land
 patches,
 that's just a recipe for disaster.

 But is it more of a disaster (for the consumers) than zero testing,
 zero review, scattered around the internet
 if-you're-lucky-with-a-good-wind you'll maybe get the right patch set?
 Because that's where we are right now, and vendors, distributors and
 the cinder core team are all saying it's a disaster.
>>>
>>> If consumers rely on upstream releases, then they are expected to
>>> migrate to newer releases after EOL, not switch to a random branch on
>>> the internet. If they rely on some commercial product, then they usually
>>> have an extended period of support and certification for their drivers,
>>> so it’s not a problem for them.
>>>
>>> Ihar
>> This is entirely unrealistic.  Force customers to upgrade.   Good luck
>> explaining to a bank that in order to get their cinder driver fix in,
>> they have to upgrade their entire OpenStack deployment. Real world
>> customers simply will balk at this all day long.
>
> Real world customers will pay for engineering to support their software,
> either their own or of one of OpenStack vendors. There is no free lunch
> from upstream here.

Sure - that may well be the case.

But if a few OpenStack vendors are willing to collaborate on the work,
would it not be better to centralise it, instead of each vendor forking
and doing the same fixes?

No so much a free lunch, as a spreading the cost of a lunch.

> Ihar
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Ihar Hrachyshka


Walter A. Boring IV  wrote:




I think "currently active stable branches" is key there. These branches
would no longer be "currently active". They would get an EOL tag when it
reaches the end of the support phases. We just wouldn't delete the
branch.
This argument comes up at least once a cycle and there is a reason we  
don't do
this. When we EOL a branch all of the infrastructure for running any ci  
against
it goes away. This means devstack support, job definitions, tempest skip  
checks,
etc. Leaving the branch around advertises that you can still submit  
patches to
it which you can't anymore. As a community we've very clearly said that  
we don't
land any code without ensuring it passes tests first, and we do not  
maintain any

of the infrastructure for doing that after an EOL.


And it's this exact policy that has lead us to this mess we are in  
today.   As a vendor that has customers that use OpenStack, we have to  
support very old releases.  Customers in the wild do not like to upgrade  
once they get OpenStack up and running because it's very difficult, time  
consuming and dangerous to do.  We have customers still running Icehouse  
and they will most likely won't upgrade any time soon.  Banks hate  
upgrading software after they have customers running on it.   This is a  
community wide problem that needs to be addressed.


Because of this problem, (not being able to backport bug fixes in our  
drivers), we have been left with forking Cinder on our own github to put  
our driver fixes there.   This is a terrible practice for the OpenStack  
community in general, and terrible for customers/users of OpenStack, as  
we have N driver vendors that have N different mechanisms for getting bug  
fixes to their customers.  I believe this is a major problem for users of  
OpenStack and it needs to be addressed.


Right. And so the proper solution in line with OpenStack practices would be  
to allow vendors to own their plugins and maintain them, potentially for  
extensive time. The fact the Cinder team is unwilling [as it seems like  
from what I read in other replies to the thread] to provide that kind of  
extensibility to vendors is unfortunate. BTW do we have a write-up of  
reasons behind that?


At the Cinder midcycle, we came up with a solution that would satisfy  
Cinder customers, as Sean planned out.


All of them? I am not sure on that one. Some consumers may put some trust  
into stable branches, and may be surprised by patches landing there without  
upstream CI in place, undermining the promise the project made by adopting  
the stable:follows-policy tag.


We acknowledge that it's a driver maintainer's responsibility to make  
sure they test any changes that get into the stable branches, because  
there is no infra support for running CI against the patches of old  
stable branches. I think that risk is far better than the existing  
reality of N cinder forks floating around github.   It's just no way to  
ship software to actual customers.


If Cinder would give you a right hooking entry point for your plugin, you  
would not need to fork the whole thing just to extend your small isolated  
bit. You would live on upstream infrastructure while stable/* is alive,  
then move to your own git repo somewhere on the internet to provide more  
bug fixes [as some vendors do for neutron drivers].


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Ihar Hrachyshka


Walter A. Boring IV  wrote:


On 08/08/2016 02:28 PM, Ihar Hrachyshka wrote:

Duncan Thomas  wrote:


On 8 August 2016 at 21:12, Matthew Treinish  wrote:
Ignoring all that, this is also contrary to how we perform testing in  
OpenStack.
We don't turn off entire classes of testing we have so we can land  
patches,

that's just a recipe for disaster.

But is it more of a disaster (for the consumers) than zero testing,  
zero review, scattered around the internet  
if-you're-lucky-with-a-good-wind you'll maybe get the right patch set?  
Because that's where we are right now, and vendors, distributors and  
the cinder core team are all saying it's a disaster.


If consumers rely on upstream releases, then they are expected to  
migrate to newer releases after EOL, not switch to a random branch on  
the internet. If they rely on some commercial product, then they usually  
have an extended period of support and certification for their drivers,  
so it’s not a problem for them.


Ihar
This is entirely unrealistic.  Force customers to upgrade.   Good luck  
explaining to a bank that in order to get their cinder driver fix in,  
they have to upgrade their entire OpenStack deployment. Real world  
customers simply will balk at this all day long.


Real world customers will pay for engineering to support their software,  
either their own or of one of OpenStack vendors. There is no free lunch  
from upstream here.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Walter A. Boring IV


On 08/08/2016 02:28 PM, Ihar Hrachyshka wrote:

Duncan Thomas  wrote:

On 8 August 2016 at 21:12, Matthew Treinish  
wrote:
Ignoring all that, this is also contrary to how we perform testing in 
OpenStack.
We don't turn off entire classes of testing we have so we can land 
patches,

that's just a recipe for disaster.

But is it more of a disaster (for the consumers) than zero testing, 
zero review, scattered around the internet 
if-you're-lucky-with-a-good-wind you'll maybe get the right patch 
set? Because that's where we are right now, and vendors, distributors 
and the cinder core team are all saying it's a disaster.


If consumers rely on upstream releases, then they are expected to 
migrate to newer releases after EOL, not switch to a random branch on 
the internet. If they rely on some commercial product, then they 
usually have an extended period of support and certification for their 
drivers, so it’s not a problem for them.


Ihar
This is entirely unrealistic.  Force customers to upgrade.   Good luck 
explaining to a bank that in order to get their cinder driver fix in, 
they have to upgrade their entire OpenStack deployment. Real world 
customers simply will balk at this all day long.


Walt


__ 


OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-09 Thread Walter A. Boring IV




I think "currently active stable branches" is key there. These branches
would no longer be "currently active". They would get an EOL tag when it
reaches the end of the support phases. We just wouldn't delete the
branch.

This argument comes up at least once a cycle and there is a reason we don't do
this. When we EOL a branch all of the infrastructure for running any ci against
it goes away. This means devstack support, job definitions, tempest skip checks,
etc. Leaving the branch around advertises that you can still submit patches to
it which you can't anymore. As a community we've very clearly said that we don't
land any code without ensuring it passes tests first, and we do not maintain any
of the infrastructure for doing that after an EOL.


And it's this exact policy that has lead us to this mess we are in 
today.   As a vendor that has customers that use OpenStack, we have to 
support very old releases.  Customers in the wild do not like to upgrade 
once they get OpenStack up and running because it's very difficult, time 
consuming and dangerous to do.  We have customers still running Icehouse 
and they will most likely won't upgrade any time soon.  Banks hate 
upgrading software after they have customers running on it.   This is a 
community wide problem that needs to be addressed.


Because of this problem, (not being able to backport bug fixes in our 
drivers), we have been left with forking Cinder on our own github to put 
our driver fixes there.   This is a terrible practice for the OpenStack 
community in general, and terrible for customers/users of OpenStack, as 
we have N driver vendors that have N different mechanisms for getting 
bug fixes to their customers.  I believe this is a major problem for 
users of OpenStack and it needs to be addressed.
At the Cinder midcycle, we came up with a solution that would satisfy 
Cinder customers, as Sean planned out.  We acknowledge that it's a 
driver maintainer's responsibility to make sure they test any changes 
that get into the stable branches, because there is no infra support for 
running CI against the patches of old stable branches. I think that risk 
is far better than the existing reality of N cinder forks floating 
around github.   It's just no way to ship software to actual customers.


$0.02,
Walt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Ihar Hrachyshka


Duncan Thomas  wrote:


On 8 August 2016 at 21:12, Matthew Treinish  wrote:
Ignoring all that, this is also contrary to how we perform testing in  
OpenStack.

We don't turn off entire classes of testing we have so we can land patches,
that's just a recipe for disaster.

But is it more of a disaster (for the consumers) than zero testing, zero  
review, scattered around the internet if-you're-lucky-with-a-good-wind  
you'll maybe get the right patch set? Because that's where we are right  
now, and vendors, distributors and the cinder core team are all saying  
it's a disaster.


If consumers rely on upstream releases, then they are expected to migrate  
to newer releases after EOL, not switch to a random branch on the internet.  
If they rely on some commercial product, then they usually have an extended  
period of support and certification for their drivers, so it’s not a  
problem for them.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Duncan Thomas

On 8 August 2016 at 21:12, Matthew Treinish  wrote:

> Ignoring all that, this is also contrary to how we perform testing in
> OpenStack.
> We don't turn off entire classes of testing we have so we can land patches,
> that's just a recipe for disaster.
>

But is it more of a disaster (for the consumers) than zero testing, zero
review, scattered around the internet if-you're-lucky-with-a-good-wind
you'll maybe get the right patch set? Because that's where we are right
now, and vendors, distributors and the cinder core team are all saying it's
a disaster.

-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Matthew Treinish

On Mon, Aug 08, 2016 at 07:40:56PM +0300, Duncan Thomas wrote:
> On 8 August 2016 at 18:31, Matthew Treinish  wrote:
> 
> >
> > This argument comes up at least once a cycle and there is a reason we
> > don't do
> > this. When we EOL a branch all of the infrastructure for running any ci
> > against
> > it goes away. This means devstack support, job definitions, tempest skip
> > checks,
> > etc. Leaving the branch around advertises that you can still submit
> > patches to
> > it which you can't anymore. As a community we've very clearly said that we
> > don't
> > land any code without ensuring it passes tests first, and we do not
> > maintain any
> > of the infrastructure for doing that after an EOL.
> >
> >
> Ok, to turn the question around, we (the cinder team) have recognised a
> definite and strong need to have somewhere for vendors to share patches on
> versions of Cinder older than the stable branch policy allows.
> 
> Given this need, what are our options?
> 
> 1. We could do all this outside Openstack infrastructure. There are
> significant downsides to doing so from organisational, maintenance, cost
> etc points of view. Also means that the place vendors go for these patches
> is not obvious, and the process for getting patches in is not standard.

This is probably your only viable option. As a community we've hit this boundary
many times. Everyone claims to want longer support windows but when it comes
down to it there is very little activity in making things work on stable
branches. Our support window is at it's maximum viable length now, and that's
unlikely to change anytime soon. We had a discussion on this exact topic at
summit:

https://etherpad.openstack.org/p/r.ddabf5c865d6f77740bcfbc112ed391c

> 
> 2. We could have something not named 'stable' that has looser rules than
> stable branches,, maybe just pep8 / unit / cinder in-tree tests. No
> devstack.

This is not an option, as I said before this isn't feasible. All the
infrastructure for running jobs on the old branches goes away. It's much more
than you realize is actually there. Including things like global requirements,
job definitions, and old node types. A lot of work goes into keeping all of
this running, and it's all interconnected. There is a reason we EOL a branch,
it's not to be vindictive, it's because keeping it running is too much work for
the small number of people who fix things. (and to a lesser extent an increased
burden our CI resources)

Ignoring all that, this is also contrary to how we perform testing in OpenStack.
We don't turn off entire classes of testing we have so we can land patches,
that's just a recipe for disaster.

-Matt Treinish

> 
> 3. We go with the Neutron model and take drivers out of tree. This is not
> something the cinder core team are in favour of - we see significant value
> in the code review that drivers currently get - the code quality
> improvements between when a driver is submitted and when it is merged are
> sometimes very significant. Also, taking the code out of tree makes it
> difficult to get all the drivers checked out in one place to analyse e.g.
> how a certain driver call is implemented across all the drivers, when
> reasoning or making changes to core code.
> 
> Given we've identified a clear need, and have repeated rejected one
> solution (take drivers out of tree - it has been discussed at every summit
> and midcycle for 3+ cycles), what positive suggestions can people make?
> 

signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Ben Swartzlander


On 08/08/2016 12:36 PM, Jeremy Stanley wrote:

On 2016-08-08 13:03:51 +0200 (+0200), Ihar Hrachyshka wrote:

Sean McGinnis  wrote:

[...]

The suggestion was to just change our stable policy in regards to driver
bugfix backports. No need to create and maintain more branches. No need
to set up gate jobs and things like that.


Unless you manage to get it approved for the global policy

[...]

That was the gist of my suggestion to Sean as far as bringing this
discussion to the ML as a first option. Basically, if lots of
projects see their driver maintainers and downstream distros forking
their stable branches to add driver updates for support of newer
hardware, then see if the current OpenStack Stable Branch policy
should be amended to say that bug fixes and newer hardware support
specifically in driver source code (as long as it doesn't touch the
core service in that repo) are acceptable.

As far as the tangent this thread has taken on changing when we
delete stable branches, I feel like the only solution there is
working with the Stable Branch team to find ways to properly extend
support for the branches in question (including keeping them
properly tested). There have been ongoing efforts to make stable
branch testing less problematic, so it's possible over time we'll be
able to increase the support duration for them. Stating that it's
okay to keep them open for changes with no testing sets a terrible
precedent.


The proposal isn't "no testing". The proposal is that the gate tests 
would be minimal. We would rely heavily on the 3rd party CI system to 
actually test the patch and tell us that nothing is broken. If the 3rd 
party CI systems can't be relied on for this purpose then they're 
useless IMO.


Yes a human would have to recognize that the patch affects a particular 
vendor and know which CI system to look at before putting his +2 on. 
This is an unfortunate effect of not having 3rd party CI vote.


-Ben Swartzlander


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Ben Swartzlander


On 08/08/2016 12:40 PM, Duncan Thomas wrote:

On 8 August 2016 at 18:31, Matthew Treinish > wrote:


This argument comes up at least once a cycle and there is a reason
we don't do
this. When we EOL a branch all of the infrastructure for running any
ci against
it goes away. This means devstack support, job definitions, tempest
skip checks,
etc. Leaving the branch around advertises that you can still submit
patches to
it which you can't anymore. As a community we've very clearly said
that we don't
land any code without ensuring it passes tests first, and we do not
maintain any
of the infrastructure for doing that after an EOL.


Ok, to turn the question around, we (the cinder team) have recognised a
definite and strong need to have somewhere for vendors to share patches
on versions of Cinder older than the stable branch policy allows.

Given this need, what are our options?

1. We could do all this outside Openstack infrastructure. There are
significant downsides to doing so from organisational, maintenance, cost
etc points of view. Also means that the place vendors go for these
patches is not obvious, and the process for getting patches in is not
standard.

2. We could have something not named 'stable' that has looser rules than
stable branches,, maybe just pep8 / unit / cinder in-tree tests. No
devstack.


This. (2) is what I thought we were proposing from the beginning. Add a 
requirement for 3rd party CI from the affected vendor to pass and I 
think it works and benefits everyone.


-Ben Swartzlander


3. We go with the Neutron model and take drivers out of tree. This is
not something the cinder core team are in favour of - we see significant
value in the code review that drivers currently get - the code quality
improvements between when a driver is submitted and when it is merged
are sometimes very significant. Also, taking the code out of tree makes
it difficult to get all the drivers checked out in one place to analyse
e.g. how a certain driver call is implemented across all the drivers,
when reasoning or making changes to core code.

Given we've identified a clear need, and have repeated rejected one
solution (take drivers out of tree - it has been discussed at every
summit and midcycle for 3+ cycles), what positive suggestions can people
make?

--
Duncan Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Philipp Marek

> Ok, to turn the question around, we (the cinder team) have recognised a
> definite and strong need to have somewhere for vendors to share patches on
> versions of Cinder older than the stable branch policy allows.
> 
> Given this need, what are our options?
> 
> 1. We could do all this outside Openstack infrastructure. There are
> significant downsides to doing so from organisational, maintenance, cost
> etc points of view. Also means that the place vendors go for these patches
> is not obvious, and the process for getting patches in is not standard.
And if some people have 2 or more different storages, they might need to 
run separate Cinder processes, because the vendors' stable tree diverge and 
cannot easily be merged again.

> 2. We could have something not named 'stable' that has looser rules than
> stable branches,, maybe just pep8 / unit / cinder in-tree tests. No
> devstack.
+1 from me.
Name it "long-term-driver-only-updates" or so ;)


> 3. We go with the Neutron model and take drivers out of tree. This is not
> something the cinder core team are in favour of - we see significant value
> in the code review that drivers currently get - the code quality
> improvements between when a driver is submitted and when it is merged are
> sometimes very significant. Also, taking the code out of tree makes it
> difficult to get all the drivers checked out in one place to analyse e.g.
> how a certain driver call is implemented across all the drivers, when
> reasoning or making changes to core code.
-1

> Given we've identified a clear need, and have repeated rejected one
> solution (take drivers out of tree - it has been discussed at every summit
> and midcycle for 3+ cycles), what positive suggestions can people make?
Number 2 - a centralized branch (ie. in openstack.org) that *only* takes 
driver updates (and doesn't need CI).
If a driver is broken, too bad for that vendor - must have been the last 
driver update, as the Cinder code didn't change since EOL...


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Duncan Thomas

On 8 August 2016 at 18:31, Matthew Treinish  wrote:

>
> This argument comes up at least once a cycle and there is a reason we
> don't do
> this. When we EOL a branch all of the infrastructure for running any ci
> against
> it goes away. This means devstack support, job definitions, tempest skip
> checks,
> etc. Leaving the branch around advertises that you can still submit
> patches to
> it which you can't anymore. As a community we've very clearly said that we
> don't
> land any code without ensuring it passes tests first, and we do not
> maintain any
> of the infrastructure for doing that after an EOL.
>
>
Ok, to turn the question around, we (the cinder team) have recognised a
definite and strong need to have somewhere for vendors to share patches on
versions of Cinder older than the stable branch policy allows.

Given this need, what are our options?

1. We could do all this outside Openstack infrastructure. There are
significant downsides to doing so from organisational, maintenance, cost
etc points of view. Also means that the place vendors go for these patches
is not obvious, and the process for getting patches in is not standard.

2. We could have something not named 'stable' that has looser rules than
stable branches,, maybe just pep8 / unit / cinder in-tree tests. No
devstack.

3. We go with the Neutron model and take drivers out of tree. This is not
something the cinder core team are in favour of - we see significant value
in the code review that drivers currently get - the code quality
improvements between when a driver is submitted and when it is merged are
sometimes very significant. Also, taking the code out of tree makes it
difficult to get all the drivers checked out in one place to analyse e.g.
how a certain driver call is implemented across all the drivers, when
reasoning or making changes to core code.

Given we've identified a clear need, and have repeated rejected one
solution (take drivers out of tree - it has been discussed at every summit
and midcycle for 3+ cycles), what positive suggestions can people make?

-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Jeremy Stanley

On 2016-08-08 13:03:51 +0200 (+0200), Ihar Hrachyshka wrote:
> Sean McGinnis  wrote:
[...]
> > The suggestion was to just change our stable policy in regards to driver
> > bugfix backports. No need to create and maintain more branches. No need
> > to set up gate jobs and things like that.
> 
> Unless you manage to get it approved for the global policy
[...]

That was the gist of my suggestion to Sean as far as bringing this
discussion to the ML as a first option. Basically, if lots of
projects see their driver maintainers and downstream distros forking
their stable branches to add driver updates for support of newer
hardware, then see if the current OpenStack Stable Branch policy
should be amended to say that bug fixes and newer hardware support
specifically in driver source code (as long as it doesn't touch the
core service in that repo) are acceptable.

As far as the tangent this thread has taken on changing when we
delete stable branches, I feel like the only solution there is
working with the Stable Branch team to find ways to properly extend
support for the branches in question (including keeping them
properly tested). There have been ongoing efforts to make stable
branch testing less problematic, so it's possible over time we'll be
able to increase the support duration for them. Stating that it's
okay to keep them open for changes with no testing sets a terrible
precedent.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Matthew Treinish

On Mon, Aug 08, 2016 at 09:47:53AM -0500, Sean McGinnis wrote:
> > 
> > Unless you manage to get it approved for the global policy, I think
> > you will effectively make your stable:follows-policy tag obsolete,
> > and then it should be removed from your project. Read the
> > requirements:
> > 
> > https://governance.openstack.org/reference/tags/stable_follows-policy.html#requirements
> > 
> > Support phases are part of the stable policy, and so if you don’t
> > mostly adhere to their definitions, you should not carry the tag.
> > Which is fine with me, it’s up to Cinder team to decide whether it’s
> > worth it.
> 
> I think "currently active stable branches" is key there. These branches
> would no longer be "currently active". They would get an EOL tag when it
> reaches the end of the support phases. We just wouldn't delete the
> branch.

This argument comes up at least once a cycle and there is a reason we don't do
this. When we EOL a branch all of the infrastructure for running any ci against
it goes away. This means devstack support, job definitions, tempest skip checks,
etc. Leaving the branch around advertises that you can still submit patches to
it which you can't anymore. As a community we've very clearly said that we don't
land any code without ensuring it passes tests first, and we do not maintain any
of the infrastructure for doing that after an EOL. 

> 
> Again, this is only for driver code. We would not allow backports to the
> core Cinder codebase.

This distinction does not actually matter, you're still trying to backport code
without the ability to run tests in the gate. The fact that it's part of a
driver doesn't really change anything.

-Matt Treinish

signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Sean McGinnis

> 
> Unless you manage to get it approved for the global policy, I think
> you will effectively make your stable:follows-policy tag obsolete,
> and then it should be removed from your project. Read the
> requirements:
> 
> https://governance.openstack.org/reference/tags/stable_follows-policy.html#requirements
> 
> Support phases are part of the stable policy, and so if you don’t
> mostly adhere to their definitions, you should not carry the tag.
> Which is fine with me, it’s up to Cinder team to decide whether it’s
> worth it.

I think "currently active stable branches" is key there. These branches
would no longer be "currently active". They would get an EOL tag when it
reaches the end of the support phases. We just wouldn't delete the
branch.

Again, this is only for driver code. We would not allow backports to the
core Cinder codebase.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Sean McGinnis

On Sat, Aug 06, 2016 at 08:00:06PM -0400, Ben Swartzlander wrote:
> On 08/06/2016 06:11 PM, Jeremy Stanley wrote:
> >On 2016-08-06 17:51:02 -0400 (-0400), Ben Swartzlander wrote:
> >[...]
> >>when it's no longer to run dsvm jobs on them (because those jobs
> >>WILL eventually break as infra stops maintaining support for very
> >>old releases) then we simply remove those jobs and rely on vendor
> >>CI + minimal upstream tests (pep8, unit tests).
> >
> >This suggestion has been resisted in the past as it's not up to our
> >community's QA standards, and implying there is "support" when we
> >can no longer test that changes don't cause breakage is effectively
> >dishonest. In the past we've held that if a branch is no longer
> >testable, then there's not much reason to collaborate on code
> >reviewing proposed backports in the first place. If we're reducing
> >these branches to merely a holding place for "fixes" that "might
> >work" it doesn't sound particularly beneficial.
> 
> Well this was the whole point, and the reason I suggested using a
> different branch other than stable/release. Keeping the branches
> open for driver bugfix backports is only valuable if we can go 5
> releases back.

We can debate how far back they should go, but I think initially we can
limit it to less that 5. I haven't done anywhere near a thorough survey,
but my impression is most distros so far have tried to stay with three
releases of the current one.

> 
> I agree the level of QA we can do gets less as releases get older,
> and nobody expects the Infra team to keep devstack-gate running on
> such old releases. However vendors and distros DO support such old
> releases and the proposal to create these branches is largely to
> simplify the distributions of bugfixes from vendors to customers and
> distros.
> 
> Compare this proposal to the status quo, which is that several
> vendors effectively maintain forks of Cinder on github or other
> public repos just to have a place to distribute bugfixes on old
> releases. Distros either need to know about these repos or do the
> backports from master themselves when taking bugfixes into old
> releases.

I think this is a key point. The need for this is to have a common and
known place for these driver fixes to be backported and made available
to whomever needs them. This isn't necessarily saying we are going to
thoroughly test and guarantee that these backports are 100% OK.

>From my understanding, part of what has failed with keeping stable
branches around longer has been the lack of involvement from distros and
vendors to keep those branches up to date. After the EOL date, these
backported driver fixes would be 100% on the vendors to have tested
their drivers and made sure they work. We as a community would just be
provider a place for those updates to go and a known place for those
that need the fixes to find them.

> 
> -Ben
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Ihar Hrachyshka


Jeremy Stanley  wrote:


On 2016-08-06 17:51:02 -0400 (-0400), Ben Swartzlander wrote:
[...]

when it's no longer to run dsvm jobs on them (because those jobs
WILL eventually break as infra stops maintaining support for very
old releases) then we simply remove those jobs and rely on vendor
CI + minimal upstream tests (pep8, unit tests).


This suggestion has been resisted in the past as it's not up to our
community's QA standards, and implying there is "support" when we
can no longer test that changes don't cause breakage is effectively
dishonest. In the past we've held that if a branch is no longer
testable, then there's not much reason to collaborate on code
reviewing proposed backports in the first place. If we're reducing
these branches to merely a holding place for "fixes" that "might
work" it doesn't sound particularly beneficial.


Right. At this point the branch does not really rely on openstack  
infrastructure services (particularly CI), and so there is little reason to  
keep the code on OpenStack premises.


A similar suggestion was proposed earlier for extending life for stable  
branches while deprovisioning gate jobs, allowing interested parties to  
collaborate on eg. CVE fixes in this branch. It was not supported in the  
past, because CI is an integral part of the promise of ‘safe fixes’ that we  
give as part of our stable policy.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-08 Thread Ihar Hrachyshka


Sean McGinnis  wrote:


This may mostly be a Cinder concern, but putting it out there to get
wider input.

For some time now there has been some debate about moving third party
drivers in Cinder to be out of tree. I won't go into that too much,
other than to point out one of the major drivers for this desire that
was brought up at our recent Cinder midcycle.

It turned out at least part of the desire to move drivers out of tree
came down to the difficulty in getting bug fixes out to end users that
were on older stable versions, whether because that's what their distro
was still using, or because of some other internal constraint that
prevented them from upgrading.

A lot of times what several vendors ended up doing is forking Cinder to
their own github repo and keeping that in sync with backports, plus
including driver fixes they needed to get out to their end users. This
has a few drawbacks:


If you would at least provide a public (more or less) stable driver API for  
vendors to use, like neutron does, then your vendors would not need to fork  
the whole Cinder tree. Instead, they would 1) work with community on bug  
fixes while stable/* is supported; 2) once stable/* is EOL, fork it into  
their own repo (on their own premises!) and maintain it from there.  
Consumers will then decide whether they trust the vendor shipped code as  
much as upstream maintained version of it that is now EOL.


Why don't vendors feel like maintaining their drivers out of tree? Is it  
technically possible? Is it too much of a burden?




1- this is more work for the vendor to keep this fork up to date
2- end users don't necessarily know where to go to find these without
   calling in to a support desk (that then troubleshoots a known issue
   and hopefully eventually ends up contacting the folks internally that
   actually work on Cinder that know it's been fixed and where to get
   the updates). Generally a bad taste for someone using Cinder and
   OpenStack.
3- Distros that package stable branches aren't able to pick up these
   changes, even if they are picking up stable branch updates for
   security fixes
4- We end up with a lot of patches proposed against security only stable
   branches that we need to either leave or abandon, just so a vendor
   can point end users to the patch to be able to grab the code changes

Proposed Solution
-

So part of our discussion at the midcycle was a desire to open up stable
restrictions for getting these driver bugfixes backported. At the time,
we had discussed having new branches created off of the stable branches
specifically for driver bugfixes. Something like:

stable/mitaka > stable/mitaka-drivers


How would distributions that care about quality determine which one to ship  
in their products? If the former, for as long as it’s supported by  
upstream, then how/when/whether distros are expected to transition to the  
latter branch?




After talking to the infra team, this really did sound like overkill.
The suggestion was to just change our stable policy in regards to driver
bugfix backports. No need to create and maintain more branches. No need
to set up gate jobs and things like that.



Unless you manage to get it approved for the global policy, I think you  
will effectively make your stable:follows-policy tag obsolete, and then it  
should be removed from your project. Read the requirements:


https://governance.openstack.org/reference/tags/stable_follows-policy.html#requirements

Support phases are part of the stable policy, and so if you don’t mostly  
adhere to their definitions, you should not carry the tag. Which is fine  
with me, it’s up to Cinder team to decide whether it’s worth it.



So this is a divergence from our official policy. I want to propose
we officially make a change to our stable policy to call out that
drivers bugfixes (NOT new driver features) be allowed at any time.



If that would be pushed as a global OpenStack policy, I would voice my  
concerns.


I think Neutron model is much more viable, with vendors untangled from core  
neutron release cycles, and effectively controlling their own destiny by  
relying on (more or less) stable plugin/driver API.


Then each vendor will be able to determine whether carrying new bug fixes  
is more important for them than having the stable:follows-policy tag for  
their deliverable, without compromising the promise the core project  
(Cinder) made with the tag applied.



If that's not OK with other project teams that support any kind of third
party drivers, I will just implement this policy specific to Cinder
unless there is a very strong objection, with good logic behind it, why
this should not be allowed.



Support phases are signalling consumers what to expect from new patch/minor  
releases. Without following the global policy, you leave consumers puzzled  
as to whether the next patch release from a  
widely-advertised-to-be-CVE-only branch will break

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-06 Thread Ben Swartzlander


On 08/06/2016 06:11 PM, Jeremy Stanley wrote:

On 2016-08-06 17:51:02 -0400 (-0400), Ben Swartzlander wrote:
[...]

when it's no longer to run dsvm jobs on them (because those jobs
WILL eventually break as infra stops maintaining support for very
old releases) then we simply remove those jobs and rely on vendor
CI + minimal upstream tests (pep8, unit tests).


This suggestion has been resisted in the past as it's not up to our
community's QA standards, and implying there is "support" when we
can no longer test that changes don't cause breakage is effectively
dishonest. In the past we've held that if a branch is no longer
testable, then there's not much reason to collaborate on code
reviewing proposed backports in the first place. If we're reducing
these branches to merely a holding place for "fixes" that "might
work" it doesn't sound particularly beneficial.


Well this was the whole point, and the reason I suggested using a 
different branch other than stable/release. Keeping the branches open 
for driver bugfix backports is only valuable if we can go 5 releases back.


I agree the level of QA we can do gets less as releases get older, and 
nobody expects the Infra team to keep devstack-gate running on such old 
releases. However vendors and distros DO support such old releases and 
the proposal to create these branches is largely to simplify the 
distributions of bugfixes from vendors to customers and distros.


Compare this proposal to the status quo, which is that several vendors 
effectively maintain forks of Cinder on github or other public repos 
just to have a place to distribute bugfixes on old releases. Distros 
either need to know about these repos or do the backports from master 
themselves when taking bugfixes into old releases.


-Ben


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-06 Thread Jeremy Stanley

On 2016-08-06 17:51:02 -0400 (-0400), Ben Swartzlander wrote:
[...]
> when it's no longer to run dsvm jobs on them (because those jobs
> WILL eventually break as infra stops maintaining support for very
> old releases) then we simply remove those jobs and rely on vendor
> CI + minimal upstream tests (pep8, unit tests).

This suggestion has been resisted in the past as it's not up to our
community's QA standards, and implying there is "support" when we
can no longer test that changes don't cause breakage is effectively
dishonest. In the past we've held that if a branch is no longer
testable, then there's not much reason to collaborate on code
reviewing proposed backports in the first place. If we're reducing
these branches to merely a holding place for "fixes" that "might
work" it doesn't sound particularly beneficial.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-06 Thread Ben Swartzlander


On 08/06/2016 11:31 AM, Sean McGinnis wrote:

This may mostly be a Cinder concern, but putting it out there to get
wider input.

For some time now there has been some debate about moving third party
drivers in Cinder to be out of tree. I won't go into that too much,
other than to point out one of the major drivers for this desire that
was brought up at our recent Cinder midcycle.

It turned out at least part of the desire to move drivers out of tree
came down to the difficulty in getting bug fixes out to end users that
were on older stable versions, whether because that's what their distro
was still using, or because of some other internal constraint that
prevented them from upgrading.

A lot of times what several vendors ended up doing is forking Cinder to
their own github repo and keeping that in sync with backports, plus
including driver fixes they needed to get out to their end users. This
has a few drawbacks:

1- this is more work for the vendor to keep this fork up to date
2- end users don't necessarily know where to go to find these without
   calling in to a support desk (that then troubleshoots a known issue
   and hopefully eventually ends up contacting the folks internally that
   actually work on Cinder that know it's been fixed and where to get
   the updates). Generally a bad taste for someone using Cinder and
   OpenStack.
3- Distros that package stable branches aren't able to pick up these
   changes, even if they are picking up stable branch updates for
   security fixes
4- We end up with a lot of patches proposed against security only stable
   branches that we need to either leave or abandon, just so a vendor
   can point end users to the patch to be able to grab the code changes

Proposed Solution
-

So part of our discussion at the midcycle was a desire to open up stable
restrictions for getting these driver bugfixes backported. At the time,
we had discussed having new branches created off of the stable branches
specifically for driver bugfixes. Something like:

stable/mitaka > stable/mitaka-drivers

After talking to the infra team, this really did sound like overkill.
The suggestion was to just change our stable policy in regards to driver
bugfix backports. No need to create and maintain more branches. No need
to set up gate jobs and things like that.

So this is a divergence from our official policy. I want to propose
we officially make a change to our stable policy to call out that
drivers bugfixes (NOT new driver features) be allowed at any time.

If that's not OK with other project teams that support any kind of third
party drivers, I will just implement this policy specific to Cinder
unless there is a very strong objection, with good logic behind it, why
this should not be allowed.

This would address a lot of the concerns at least within Cinder and
allow us to better support users stuck on older releases.

I'm open and welcome to any feedback on this. Unless there are any major
concerns raised, I will at least instruct any Cinder stable cores to
start allowing these bugfix patches through past the security only
phase.


The only issue I see with this modified proposal is that it doesn't 
address the lifetime of the stable branches. If the plan is to use the 
normal stable branch instead of making a special branch, then we also 
need to find a way to keep stable branches around for practically 
forever (way longer than the typical 12 months).


Those of us dealing with bugfix backports for customers inevitably are 
looking at going 3, 4, or 5 releases back with the backports. Therefore 
I'd suggest modifying the policy to keep the stable branches around more 
or less forever, and when it's no longer to run dsvm jobs on them 
(because those jobs WILL eventually break as infra stops maintaining 
support for very old releases) then we simply remove those jobs and rely 
on vendor CI + minimal upstream tests (pep8, unit tests).


-Ben


Thanks!

Sean McGinnis (smcginnis)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-06 Thread Duncan Thomas

+1 from me

Sound like the best solution to at least part of the problem that was
causing people to want to pull the drivers out of tree

On 6 Aug 2016 18:49, "Philipp Marek"  wrote:

> > I want to propose
> > we officially make a change to our stable policy to call out that
> > drivers bugfixes (NOT new driver features) be allowed at any time.
> Emphatically +1 from me.
>
>
> With the small addendum that "bugfixes" should include compatibility
> changes for libraries used.
>
>
> Thanks for bringing that up!
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [stable] [all] Changing stable policy for drivers

2016-08-06 Thread Philipp Marek

> I want to propose
> we officially make a change to our stable policy to call out that
> drivers bugfixes (NOT new driver features) be allowed at any time.
Emphatically +1 from me.


With the small addendum that "bugfixes" should include compatibility
changes for libraries used.


Thanks for bringing that up!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

72 matches

Mail list logo