Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Matt Riedemann]

On 11/9/2015 10:15 PM, Matthew Treinish wrote:

On Mon, Nov 09, 2015 at 05:05:36PM +1100, Tony Breeds wrote:

On Fri, Nov 06, 2015 at 10:12:21AM -0800, Clint Byrum wrote:


The argument in the original post, I think, is that we should not
stand in the way of the vendors continuing to collaborate on stable
maintenance in the upstream context after the EOL date. We already have
distro vendors doing work in the stable branches, but at EOL we push
them off to their respective distro-specific homes.


That is indeed a better summary than I started with.

I have a half formed idea that creates a state between the current EOL (where
we delete the branches) to what we have today (where we have full CI/VMT/Release
management.


So this idea has come up before and it has been rejected for good reasons. If
vendors want to collaborate, but not to actually work to keep things working in
the gate is that really collaboration? It's just vendors pushing whatever random
backports they want into a shared repo. There is a reason we do all the testing
and it's deeply encoded into the openstack culture. If managing to keep things
verifiably working with the same set of test jobs when a branch was created is
too much of a burden for people then there isn't a reason to keep it around.

This is the crux of why we have shorter branch support windows. No matter how
much complaining people do about how they want LTS releases or longer support
windows, and we'll have world peace when they can use Grizzly with upstream
support for 17 years it doesn't change that barely anyone ever steps up to work
on keeping the gate working on stable branches.


Heh, speaking of Grizzly, I was just looking at backporting 
https://review.openstack.org/#/c/219301/ to Grizzly yesterday since we 
still support it. There is an unholy trifecta of things in that change 
that aren't in Grizzly: nova objects, an API method to get filtered 
resources from the DB, and a database migration in the change that added 
the virt driver method used as part of the fix (all added in Havana).


So fixing that in Grizzly is basically going to be a Frankenstein change 
to avoid the REST API backport and DB migration backport. Good times.


Why am I pointing this out? Just to give an example of the kind of mess 
that is involved in maintaining a branch that old.




Tony, as someone who has done a good job coming up to speed on fixing issues on 
the
stable branches you know this firsthand. We're always end up talking to the same
handful of people to debug issues. We're also almost always in firefighting mode
and regular failure rates on stable just keep going up when we look away. People
also burn out quickly debugging these issues all the time. Personally, I know I
don't keep an eye on things nearly as closely as I did before.



There is a massive ammount of trust involved in that simple statement and I 
don't
underestimate that.

This would provide a place where interested vendors can work together.
We could disable grenade in juno which isn't awesome but removes the gotta land
patch in juno, kilo and liberty to unblock the master gate.
We could reduce the VMT impact by nominating a point of contact for juno and
granting that person access to the embargoed bugs.  Similarly we could
trust/delegate a certain about of the release management to that team (I say
team but so far we've only got one volunteer)

We can't ignore the fact that fixing things in Juno *may* still require fixes
in Kilo (and later releases) esp. given the mess that is requirements in
stable/juno


As much as I'd like everyone to get on the CD train, I think it might
make sense to enable the vendors to not diverge, but instead let them
show up with people and commitment and say "Hey we're going to keep
Juno/Mitaka/etc alive!".

So perhaps what would make sense is defining a process by which they can
make that happen.

Note that it's not just backporters though. It's infra resources too.


Sure.  CI and Python 2.6 I have a little understanding of.  I guess I can
extrapolate the additional burden on {system,project}-config.  I willingly
admit I don't have a detailed feel for what I'm asking here.


It's more than just that too, there is additional resources on Tempest and other
QA projects like devstack and grenade too. Tempest is branchless, so to keep
branches around longer you have to have additional jobs running on tempest to
ensure incoming changes work across all releases. In Tokyo we were discussing
doing the same thing on the client repos too because they make similar
guarantees about backwards compat but we never test it. There is a ton of extra
load generated by keeping things around for longer, it's not to be taken
lightly. Especially given the historical lack of contribution in this space.
This is honestly why our 1 experiment in a longer support ended in failure,
nobody stepped up to support the extra branch. To even attempt it again we need
proof that things have improved, which it clearly

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Matthew Treinish]

On Mon, Nov 09, 2015 at 05:05:36PM +1100, Tony Breeds wrote:
> On Fri, Nov 06, 2015 at 10:12:21AM -0800, Clint Byrum wrote:
> 
> > The argument in the original post, I think, is that we should not
> > stand in the way of the vendors continuing to collaborate on stable
> > maintenance in the upstream context after the EOL date. We already have
> > distro vendors doing work in the stable branches, but at EOL we push
> > them off to their respective distro-specific homes.
> 
> That is indeed a better summary than I started with.
> 
> I have a half formed idea that creates a state between the current EOL (where
> we delete the branches) to what we have today (where we have full 
> CI/VMT/Release
> management.

So this idea has come up before and it has been rejected for good reasons. If
vendors want to collaborate, but not to actually work to keep things working in
the gate is that really collaboration? It's just vendors pushing whatever random
backports they want into a shared repo. There is a reason we do all the testing
and it's deeply encoded into the openstack culture. If managing to keep things
verifiably working with the same set of test jobs when a branch was created is
too much of a burden for people then there isn't a reason to keep it around.

This is the crux of why we have shorter branch support windows. No matter how
much complaining people do about how they want LTS releases or longer support
windows, and we'll have world peace when they can use Grizzly with upstream
support for 17 years it doesn't change that barely anyone ever steps up to work
on keeping the gate working on stable branches.

Tony, as someone who has done a good job coming up to speed on fixing issues on 
the
stable branches you know this firsthand. We're always end up talking to the same
handful of people to debug issues. We're also almost always in firefighting mode
and regular failure rates on stable just keep going up when we look away. People
also burn out quickly debugging these issues all the time. Personally, I know I
don't keep an eye on things nearly as closely as I did before.

> 
> There is a massive ammount of trust involved in that simple statement and I 
> don't
> underestimate that.
> 
> This would provide a place where interested vendors can work together.
> We could disable grenade in juno which isn't awesome but removes the gotta 
> land
> patch in juno, kilo and liberty to unblock the master gate.
> We could reduce the VMT impact by nominating a point of contact for juno and
> granting that person access to the embargoed bugs.  Similarly we could
> trust/delegate a certain about of the release management to that team (I say
> team but so far we've only got one volunteer)
> 
> We can't ignore the fact that fixing things in Juno *may* still require fixes
> in Kilo (and later releases) esp. given the mess that is requirements in
> stable/juno
>  
> > As much as I'd like everyone to get on the CD train, I think it might
> > make sense to enable the vendors to not diverge, but instead let them
> > show up with people and commitment and say "Hey we're going to keep
> > Juno/Mitaka/etc alive!".
> > 
> > So perhaps what would make sense is defining a process by which they can
> > make that happen.
> > 
> > Note that it's not just backporters though. It's infra resources too.
> 
> Sure.  CI and Python 2.6 I have a little understanding of.  I guess I can
> extrapolate the additional burden on {system,project}-config.  I willingly
> admit I don't have a detailed feel for what I'm asking here.

It's more than just that too, there is additional resources on Tempest and other
QA projects like devstack and grenade too. Tempest is branchless, so to keep
branches around longer you have to have additional jobs running on tempest to
ensure incoming changes work across all releases. In Tokyo we were discussing
doing the same thing on the client repos too because they make similar
guarantees about backwards compat but we never test it. There is a ton of extra
load generated by keeping things around for longer, it's not to be taken
lightly. Especially given the historical lack of contribution in this space.
This is honestly why our 1 experiment in a longer support ended in failure,
nobody stepped up to support the extra branch. To even attempt it again we need
proof that things have improved, which it clearly hasn't.

-Matt Treinish


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Robert Collins]

On 10 November 2015 at 08:22, matt  wrote:
> tons from what i've seen.  there are a LOT of havana and even earlier stuff
> out there.  essex is still out there in the wild.

>From the Mitaka keynotes we know there are substantial sized public
clouds still in production running Diablo...

-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Matt Riedemann]

On 11/9/2015 12:21 AM, Tony Breeds wrote:

On Fri, Nov 06, 2015 at 06:42:05PM +, Jeremy Stanley wrote:

On 2015-11-06 10:12:21 -0800 (-0800), Clint Byrum wrote:
[...]

Note that it's not just backporters though. It's infra resources too.


Aye, there's the rub. We don't just EOL these branches for fun or
because we hate old things or because ooh shiny squirrel. We EOL
them at a cadence where the community has demonstrated it loses its
ability to keep them healthy and testable (evaluated based on
performance over recent prior cycles because we have to warn
downstreams well in advance as to when they should expect upstream
support to cease).


Sure and Juno is in bad shape, as Matt will attest but I think it can be fixed
;P


Downstream maintainers regularly claim they will step up their
assistance upstream to keep stable branches alive if only we'll
extend the lifespan on them, so we tried that with Icehouse and,
based on our experience there, scaled back the lifespan of Juno
again accordingly. Keep in mind that extending support of stable
branches necessarily implies supporting a larger _number_ of stable
branches in parallel. If we switched from 12 months after release to
18 then we're maintaining at least 3 stable branches at any point in
time. If we extend it to 24 months then that's 4 stable branches.


Yes I agree and frankly I'm disappointed that the support I received in Tokyo
hasn't arrived on this thread (yet).  As to the number of stable branches,
I'm nervous about this.  I don't really want an additional period of
life-support for Juno to mandate a similar commitment for Kilo.

I realise it's a slippery slope.


To those who suggest solving this by claiming one is a LTS release
every couple years, you're implying a vastly different upgrade model
than we have now. If we declare Juno is a LTS and leave it supported
another 12 months, then 6 months from now when we EOL stable/kilo
we'll be telling deployers that they have to upgrade from supported
stable/juno through unsupported stable/kilo to supported
stable/icehouse before running stable/mitaka. Or else you're saying
you intend to fix the current inability of our projects to skip
intermediate releases entirely during upgrades (a great idea, and so
I'm thrilled by those of you who intend to make it a reality, we can
revisit the LTS discussion once you finish that).


I carefully *didn't* suggest and OpenStack LTS :)

Yours Tony.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Given the requirements whack-a-mole in stable/juno, and to an extent in 
stable/kilo, I just don't think it's really worth keeping alive what we 
have for stable/juno right now.


I think once we end of life Kilo we'll have a much better idea of how 
stable/liberty is going with upper-constraints. If we're not constantly 
putting out gate wedge issues due to capped requirements, that makes for 
a happier stable maint / QA / dev team that is then more willing to keep 
things around longer because they just work (unit test jobs, grenade 
jobs and tempest/dsvm jobs).


Keeping the branches around longer if they are working isn't so bad - 
you only consume CI resources as needed when changes on stable are 
proposed and updated, which is like the experimental queue on master. 
There is the nightly periodic jobs but that's just once a day.


As already noted, the other hit is branchless Tempest running stable 
compat jobs, which is a problem for anyone trying to get Tempest changes 
to land. Think of the race bugs you have to recheck against just for 
master, then multiply that times however many stable branches you have 
to maintain and that's what a Tempest change has to pass through. We 
could also branch Tempest at a certain point though. Tempest has tags 
that correspond to release milestones for the core projects. Internally 
we create branches for Tempest to align with the branches that are EOL 
upstream so we can fix requirements issues as needed (like capping 
requirements on grizzly or havana, for example).


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Tony Breeds]

On Fri, Nov 06, 2015 at 06:42:05PM +, Jeremy Stanley wrote:
> On 2015-11-06 10:12:21 -0800 (-0800), Clint Byrum wrote:
> [...]
> > Note that it's not just backporters though. It's infra resources too.
> 
> Aye, there's the rub. We don't just EOL these branches for fun or
> because we hate old things or because ooh shiny squirrel. We EOL
> them at a cadence where the community has demonstrated it loses its
> ability to keep them healthy and testable (evaluated based on
> performance over recent prior cycles because we have to warn
> downstreams well in advance as to when they should expect upstream
> support to cease).

Sure and Juno is in bad shape, as Matt will attest but I think it can be fixed
;P
 
> Downstream maintainers regularly claim they will step up their
> assistance upstream to keep stable branches alive if only we'll
> extend the lifespan on them, so we tried that with Icehouse and,
> based on our experience there, scaled back the lifespan of Juno
> again accordingly. Keep in mind that extending support of stable
> branches necessarily implies supporting a larger _number_ of stable
> branches in parallel. If we switched from 12 months after release to
> 18 then we're maintaining at least 3 stable branches at any point in
> time. If we extend it to 24 months then that's 4 stable branches.

Yes I agree and frankly I'm disappointed that the support I received in Tokyo
hasn't arrived on this thread (yet).  As to the number of stable branches,
I'm nervous about this.  I don't really want an additional period of
life-support for Juno to mandate a similar commitment for Kilo.

I realise it's a slippery slope.
 
> To those who suggest solving this by claiming one is a LTS release
> every couple years, you're implying a vastly different upgrade model
> than we have now. If we declare Juno is a LTS and leave it supported
> another 12 months, then 6 months from now when we EOL stable/kilo
> we'll be telling deployers that they have to upgrade from supported
> stable/juno through unsupported stable/kilo to supported
> stable/icehouse before running stable/mitaka. Or else you're saying
> you intend to fix the current inability of our projects to skip
> intermediate releases entirely during upgrades (a great idea, and so
> I'm thrilled by those of you who intend to make it a reality, we can
> revisit the LTS discussion once you finish that).

I carefully *didn't* suggest and OpenStack LTS :)

Yours Tony.


pgpjTuwEKm3Of.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Tony Breeds]

On Fri, Nov 06, 2015 at 10:12:21AM -0800, Clint Byrum wrote:

> The argument in the original post, I think, is that we should not
> stand in the way of the vendors continuing to collaborate on stable
> maintenance in the upstream context after the EOL date. We already have
> distro vendors doing work in the stable branches, but at EOL we push
> them off to their respective distro-specific homes.

That is indeed a better summary than I started with.

I have a half formed idea that creates a state between the current EOL (where
we delete the branches) to what we have today (where we have full CI/VMT/Release
management.

There is a massive ammount of trust involved in that simple statement and I 
don't
underestimate that.

This would provide a place where interested vendors can work together.
We could disable grenade in juno which isn't awesome but removes the gotta land
patch in juno, kilo and liberty to unblock the master gate.
We could reduce the VMT impact by nominating a point of contact for juno and
granting that person access to the embargoed bugs.  Similarly we could
trust/delegate a certain about of the release management to that team (I say
team but so far we've only got one volunteer)

We can't ignore the fact that fixing things in Juno *may* still require fixes
in Kilo (and later releases) esp. given the mess that is requirements in
stable/juno
 
> As much as I'd like everyone to get on the CD train, I think it might
> make sense to enable the vendors to not diverge, but instead let them
> show up with people and commitment and say "Hey we're going to keep
> Juno/Mitaka/etc alive!".
> 
> So perhaps what would make sense is defining a process by which they can
> make that happen.
> 
> Note that it's not just backporters though. It's infra resources too.

Sure.  CI and Python 2.6 I have a little understanding of.  I guess I can
extrapolate the additional burden on {system,project}-config.  I willingly
admit I don't have a detailed feel for what I'm asking here.

Yours Tony.


pgpcnM3p3XpLX.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Mark Baker]

Certainly the aim is to support upgrades between LTS releases.
Getting a meaningful keynote slot at an OpenStack summit is more of a
challenge.

On 6 Nov 2015 9:27 pm, "Jonathan Proulx"  wrote:
>
> On Fri, Nov 06, 2015 at 05:28:13PM +, Mark Baker wrote:
> :Worth mentioning that OpenStack releases that come out at the same time
as
> :Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka) are
> :supported for 5 years by Canonical so are already kind of an LTS. Support
> :in this context means patches, updates and commercial support (for a
fee).
> :For paying customers 3 years of patches, updates and commercial support
for
> :April releases, (Kilo, O, Q etc..) is also available.
>
> 
> And Canonical will support a live upgarde directly from Essex to
> Icehouse and Icehouse to Mitaka?
>
> I'd love to see Shuttleworth do that that as a live keynote, but only
> on a system with at least hundres on nodes and many VMs...
> 
>
> That's where LTS falls down conceptually we're struggling to make
> single release upgrades work at this point.
>
> I do agree LTS for release would be great but honestly OpenStack isn't
> Mature enough for that yet.
>
> -Jon
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Jonathan Proulx]

On Fri, Nov 06, 2015 at 05:28:13PM +, Mark Baker wrote:
:Worth mentioning that OpenStack releases that come out at the same time as
:Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka) are
:supported for 5 years by Canonical so are already kind of an LTS. Support
:in this context means patches, updates and commercial support (for a fee).
:For paying customers 3 years of patches, updates and commercial support for
:April releases, (Kilo, O, Q etc..) is also available.


And Canonical will support a live upgarde directly from Essex to
Icehouse and Icehouse to Mitaka?

I'd love to see Shuttleworth do that that as a live keynote, but only
on a system with at least hundres on nodes and many VMs...


That's where LTS falls down conceptually we're struggling to make
single release upgrades work at this point.

I do agree LTS for release would be great but honestly OpenStack isn't
Mature enough for that yet.

-Jon

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Donald Talton]

I agree, but to use your argument: how hard would it be to setup a small group 
to do this for the community? I’m sure there would be a few people interested 
in maintaining it…

From: matt [mailto:m...@nycresistor.com]
Sent: Friday, November 06, 2015 1:18 PM
To: Fox, Kevin M
Cc: Jesse Keating; OpenStack Development Mailing List (not for usage 
questions); openstack-operat...@lists.openstack.org
Subject: Re: [Openstack-operators] [openstack-dev] [stable][all] Keeping Juno 
"alive" for longer.

backporting patches isn't too terribly hard to be honest.  you could probably 
hire a consultant to do it if need be.  mirantis would probably quote you a 
price.

On Fri, Nov 6, 2015 at 3:10 PM, Fox, Kevin M 
mailto:kevin@pnnl.gov>> wrote:
Kind of related, as an op, we see a lot of 3rd party repositories that recently 
only supported rhel5 move to finally supporting rhel6 because rhel7 came out 
and rhel5 went to long term support contract only. This caused us to have to 
support rhel5 way longer then we would have liked. Now, we're stuck at 6 
instead of 7. :/

Some number of users will stick with juno until it is EOL and then move. 
Sometimes its because its a desire to not make a change. Sometimes its 
considered a good thing by the ops that they finally have a "good enough" 
excuse (EOL) to move forward "finally" (sigh of relief). :)

Thanks,
Kevin

From: Jesse Keating [j...@bluebox.net]
Sent: Friday, November 06, 2015 10:14 AM
To: Dan Smith
Cc: OpenStack Development Mailing List (not for usage questions); 
openstack-operat...@lists.openstack.org
Subject: Re: [Openstack-operators] [openstack-dev] [stable][all] Keeping Juno 
"alive" for longer.
We (Blue Box, an IBM company) do have a lot of installs on Juno, however we'll 
be aggressively moving to Kilo, so we are not interested in keeping Juno alive.


- jlk

On Fri, Nov 6, 2015 at 9:37 AM, Dan Smith 
mailto:d...@danplanet.com>> wrote:
> Worth mentioning that OpenStack releases that come out at the same time
> as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
> are supported for 5 years by Canonical so are already kind of an LTS.
> Support in this context means patches, updates and commercial support
> (for a fee).
> For paying customers 3 years of patches, updates and commercial support
> for April releases, (Kilo, O, Q etc..) is also available.

Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
maintaining an older release for so long is a good use of people or CI
resources, especially given how hard it can be for us to keep even
recent stable releases working and maintained.

--Dan

___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


This email and any files transmitted with it are confidential, proprietary and 
intended solely for the individual or entity to whom they are addressed. If you 
have received this email in error please delete it immediately.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[matt]

backporting patches isn't too terribly hard to be honest.  you could
probably hire a consultant to do it if need be.  mirantis would probably
quote you a price.

On Fri, Nov 6, 2015 at 3:10 PM, Fox, Kevin M  wrote:

> Kind of related, as an op, we see a lot of 3rd party repositories that
> recently only supported rhel5 move to finally supporting rhel6 because
> rhel7 came out and rhel5 went to long term support contract only. This
> caused us to have to support rhel5 way longer then we would have liked.
> Now, we're stuck at 6 instead of 7. :/
>
> Some number of users will stick with juno until it is EOL and then move.
> Sometimes its because its a desire to not make a change. Sometimes its
> considered a good thing by the ops that they finally have a "good enough"
> excuse (EOL) to move forward "finally" (sigh of relief). :)
>
> Thanks,
> Kevin
>
> --
> *From:* Jesse Keating [j...@bluebox.net]
> *Sent:* Friday, November 06, 2015 10:14 AM
> *To:* Dan Smith
> *Cc:* OpenStack Development Mailing List (not for usage questions);
> openstack-operat...@lists.openstack.org
> *Subject:* Re: [Openstack-operators] [openstack-dev] [stable][all]
> Keeping Juno "alive" for longer.
>
> We (Blue Box, an IBM company) do have a lot of installs on Juno, however
> we'll be aggressively moving to Kilo, so we are not interested in keeping
> Juno alive.
>
>
> - jlk
>
> On Fri, Nov 6, 2015 at 9:37 AM, Dan Smith  wrote:
>
>> > Worth mentioning that OpenStack releases that come out at the same time
>> > as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
>> > are supported for 5 years by Canonical so are already kind of an LTS.
>> > Support in this context means patches, updates and commercial support
>> > (for a fee).
>> > For paying customers 3 years of patches, updates and commercial support
>> > for April releases, (Kilo, O, Q etc..) is also available.
>>
>> Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
>> maintaining an older release for so long is a good use of people or CI
>> resources, especially given how hard it can be for us to keep even
>> recent stable releases working and maintained.
>>
>> --Dan
>>
>> ___
>> OpenStack-operators mailing list
>> openstack-operat...@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
> ___
> OpenStack-operators mailing list
> openstack-operat...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Fox, Kevin M]

Kind of related, as an op, we see a lot of 3rd party repositories that recently 
only supported rhel5 move to finally supporting rhel6 because rhel7 came out 
and rhel5 went to long term support contract only. This caused us to have to 
support rhel5 way longer then we would have liked. Now, we're stuck at 6 
instead of 7. :/

Some number of users will stick with juno until it is EOL and then move. 
Sometimes its because its a desire to not make a change. Sometimes its 
considered a good thing by the ops that they finally have a "good enough" 
excuse (EOL) to move forward "finally" (sigh of relief). :)

Thanks,
Kevin


From: Jesse Keating [j...@bluebox.net]
Sent: Friday, November 06, 2015 10:14 AM
To: Dan Smith
Cc: OpenStack Development Mailing List (not for usage questions); 
openstack-operat...@lists.openstack.org
Subject: Re: [Openstack-operators] [openstack-dev] [stable][all] Keeping Juno 
"alive" for longer.

We (Blue Box, an IBM company) do have a lot of installs on Juno, however we'll 
be aggressively moving to Kilo, so we are not interested in keeping Juno alive.


- jlk

On Fri, Nov 6, 2015 at 9:37 AM, Dan Smith 
mailto:d...@danplanet.com>> wrote:
> Worth mentioning that OpenStack releases that come out at the same time
> as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
> are supported for 5 years by Canonical so are already kind of an LTS.
> Support in this context means patches, updates and commercial support
> (for a fee).
> For paying customers 3 years of patches, updates and commercial support
> for April releases, (Kilo, O, Q etc..) is also available.

Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
maintaining an older release for so long is a good use of people or CI
resources, especially given how hard it can be for us to keep even
recent stable releases working and maintained.

--Dan

___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Doug Hellmann]

Excerpts from Joshua Harlow's message of 2015-11-06 11:11:02 -0800:
> Clint Byrum wrote:
> > Excerpts from Doug Hellmann's message of 2015-11-06 10:28:41 -0800:
> >> Excerpts from Clint Byrum's message of 2015-11-06 10:12:21 -0800:
> >>> Excerpts from Dan Smith's message of 2015-11-06 09:37:44 -0800:
> > Worth mentioning that OpenStack releases that come out at the same time
> > as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
> > are supported for 5 years by Canonical so are already kind of an LTS.
> > Support in this context means patches, updates and commercial support
> > (for a fee).
> > For paying customers 3 years of patches, updates and commercial support
> > for April releases, (Kilo, O, Q etc..) is also available.
>  Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
>  maintaining an older release for so long is a good use of people or CI
>  resources, especially given how hard it can be for us to keep even
>  recent stable releases working and maintained.
> 
> >>> The argument in the original post, I think, is that we should not
> >>> stand in the way of the vendors continuing to collaborate on stable
> >>> maintenance in the upstream context after the EOL date. We already have
> >>> distro vendors doing work in the stable branches, but at EOL we push
> >>> them off to their respective distro-specific homes.
> >>>
> >>> As much as I'd like everyone to get on the CD train, I think it might
> >>> make sense to enable the vendors to not diverge, but instead let them
> >>> show up with people and commitment and say "Hey we're going to keep
> >>> Juno/Mitaka/etc alive!".
> >>>
> >>> So perhaps what would make sense is defining a process by which they can
> >>> make that happen.
> >> Do we need a new process? Aren't the existing stable maintenance
> >> and infrastructure teams clearly defined?
> >>
> >> We have this discussion whenever a release is about to go EOL, and
> >> the result is more or less the same each time. The burden of
> >> maintaining stable branches for longer than we do is currently
> >> greater than the resources being applied upstream to do that
> >> maintenance. Until that changes, I don't realistically see us being
> >> able to increase the community's commitment. That's not a lack of
> >> willingness, just an assessment of our current resources.
> >
> > I tend to agree with you. I only bring up a new process because I wonder
> > if the distro vendors would even be interested in collaborating on this,
> > or if this is just sort of "what they do" and we should accept that
> > they're going to do it outside upstream no matter how easy we make it.
> >
> > If we do believe that, and are OK with that, then we should not extend
> > EOL's, and we should make sure users understand that when they choose
> > the source of their OpenStack software.
> 
> Except for the fact that you are now forcing deployers that may or may 
> not be ok with paying for paid support to now pay for it... What is the 
> adoption rate/expected adoption rate of someone transitioning there 
> current cloud (which they did not pay support for) to a paid support model?

I'm not sure where that's coming from. We're not talking about
shortening any of the timelines we set for releases so far, only
sticking with them rather than extending them.

> 
> Does that require them to redeploy/convert there whole cloud using 
> vendors provided packages/deployment model... If so, jeez, that sounds 
> iffy...
> 
> And if a large majority of deployers aren't able to do that conversion 
> (or aren't willing to pay for support) and those same deployers are 
> willing to provide developers/others to ensure the old branches continue 
> to work and they know the issues of CI and they are willing to stay 
> on-top of that (a old-branch-dictator/leader may be needed to ensure 
> this?) then meh, I think we as a community should just let those 
> deployers have at it (ensuring they keep on working on the old branches 
> via what 'old-branch-dictator/leader/group' says is broken/needs fixing...)

That's basically my position. If someone actually shows up to do
the stable branch work, we can discuss changing timelines. So far
we have not had a stampede of new assistance, as far as I can tell.
We do still have a small committed group working on it, but I'm not
prepared to ask them to take on more of a commitment without
additional help.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Clint Byrum]

Excerpts from Joshua Harlow's message of 2015-11-06 11:11:02 -0800:
> Clint Byrum wrote:
> > Excerpts from Doug Hellmann's message of 2015-11-06 10:28:41 -0800:
> >> Excerpts from Clint Byrum's message of 2015-11-06 10:12:21 -0800:
> >>> Excerpts from Dan Smith's message of 2015-11-06 09:37:44 -0800:
> > Worth mentioning that OpenStack releases that come out at the same time
> > as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
> > are supported for 5 years by Canonical so are already kind of an LTS.
> > Support in this context means patches, updates and commercial support
> > (for a fee).
> > For paying customers 3 years of patches, updates and commercial support
> > for April releases, (Kilo, O, Q etc..) is also available.
>  Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
>  maintaining an older release for so long is a good use of people or CI
>  resources, especially given how hard it can be for us to keep even
>  recent stable releases working and maintained.
> 
> >>> The argument in the original post, I think, is that we should not
> >>> stand in the way of the vendors continuing to collaborate on stable
> >>> maintenance in the upstream context after the EOL date. We already have
> >>> distro vendors doing work in the stable branches, but at EOL we push
> >>> them off to their respective distro-specific homes.
> >>>
> >>> As much as I'd like everyone to get on the CD train, I think it might
> >>> make sense to enable the vendors to not diverge, but instead let them
> >>> show up with people and commitment and say "Hey we're going to keep
> >>> Juno/Mitaka/etc alive!".
> >>>
> >>> So perhaps what would make sense is defining a process by which they can
> >>> make that happen.
> >> Do we need a new process? Aren't the existing stable maintenance
> >> and infrastructure teams clearly defined?
> >>
> >> We have this discussion whenever a release is about to go EOL, and
> >> the result is more or less the same each time. The burden of
> >> maintaining stable branches for longer than we do is currently
> >> greater than the resources being applied upstream to do that
> >> maintenance. Until that changes, I don't realistically see us being
> >> able to increase the community's commitment. That's not a lack of
> >> willingness, just an assessment of our current resources.
> >
> > I tend to agree with you. I only bring up a new process because I wonder
> > if the distro vendors would even be interested in collaborating on this,
> > or if this is just sort of "what they do" and we should accept that
> > they're going to do it outside upstream no matter how easy we make it.
> >
> > If we do believe that, and are OK with that, then we should not extend
> > EOL's, and we should make sure users understand that when they choose
> > the source of their OpenStack software.
> 
> Except for the fact that you are now forcing deployers that may or may 
> not be ok with paying for paid support to now pay for it... What is the 
> adoption rate/expected adoption rate of someone transitioning there 
> current cloud (which they did not pay support for) to a paid support model?
> 
> Does that require them to redeploy/convert there whole cloud using 
> vendors provided packages/deployment model... If so, jeez, that sounds 
> iffy...
> 
> And if a large majority of deployers aren't able to do that conversion 
> (or aren't willing to pay for support) and those same deployers are 
> willing to provide developers/others to ensure the old branches continue 
> to work and they know the issues of CI and they are willing to stay 
> on-top of that (a old-branch-dictator/leader may be needed to ensure 
> this?) then meh, I think we as a community should just let those 
> deployers have at it (ensuring they keep on working on the old branches 
> via what 'old-branch-dictator/leader/group' says is broken/needs fixing...)

Right, what I think where this leads though is that those who have
developers converge on CD, and those who have no developers have to pay
for support anyway. Running without developers and without a support
entity that can actually fix things is an interesting combination and
I'd be very curious to hear if there are any deployers having a positive
experience working that way.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Doug Hellmann]

Excerpts from Clint Byrum's message of 2015-11-06 10:50:23 -0800:
> Excerpts from Doug Hellmann's message of 2015-11-06 10:28:41 -0800:
> > Excerpts from Clint Byrum's message of 2015-11-06 10:12:21 -0800:
> > > Excerpts from Dan Smith's message of 2015-11-06 09:37:44 -0800:
> > > > > Worth mentioning that OpenStack releases that come out at the same 
> > > > > time
> > > > > as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + 
> > > > > Mitaka)
> > > > > are supported for 5 years by Canonical so are already kind of an LTS.
> > > > > Support in this context means patches, updates and commercial support
> > > > > (for a fee).
> > > > > For paying customers 3 years of patches, updates and commercial 
> > > > > support
> > > > > for April releases, (Kilo, O, Q etc..) is also available.
> > > > 
> > > > Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
> > > > maintaining an older release for so long is a good use of people or CI
> > > > resources, especially given how hard it can be for us to keep even
> > > > recent stable releases working and maintained.
> > > > 
> > > 
> > > The argument in the original post, I think, is that we should not
> > > stand in the way of the vendors continuing to collaborate on stable
> > > maintenance in the upstream context after the EOL date. We already have
> > > distro vendors doing work in the stable branches, but at EOL we push
> > > them off to their respective distro-specific homes.
> > > 
> > > As much as I'd like everyone to get on the CD train, I think it might
> > > make sense to enable the vendors to not diverge, but instead let them
> > > show up with people and commitment and say "Hey we're going to keep
> > > Juno/Mitaka/etc alive!".
> > > 
> > > So perhaps what would make sense is defining a process by which they can
> > > make that happen.
> > 
> > Do we need a new process? Aren't the existing stable maintenance
> > and infrastructure teams clearly defined?
> > 
> > We have this discussion whenever a release is about to go EOL, and
> > the result is more or less the same each time. The burden of
> > maintaining stable branches for longer than we do is currently
> > greater than the resources being applied upstream to do that
> > maintenance. Until that changes, I don't realistically see us being
> > able to increase the community's commitment. That's not a lack of
> > willingness, just an assessment of our current resources.
> 
> I tend to agree with you. I only bring up a new process because I wonder
> if the distro vendors would even be interested in collaborating on this,
> or if this is just sort of "what they do" and we should accept that
> they're going to do it outside upstream no matter how easy we make it.
> 
> If we do believe that, and are OK with that, then we should not extend
> EOL's, and we should make sure users understand that when they choose
> the source of their OpenStack software.

OK, sure. If we can improve the process, then we should discuss that. We
did accommodate distro requests to continue tagging stable releases for
Liberty, but I'm not sure that compromise was made as the result of
promises of more resources.

Thierry did bring up the idea that the stable maintenance team should
stand alone, rather than being part of the release management team. That
would give the team its own PTL, and give them more autonomy about
deciding stable processes. I support the idea, but no one has come
forward and offered to drive it, yet.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Joshua Harlow]

Clint Byrum wrote:

Excerpts from Doug Hellmann's message of 2015-11-06 10:28:41 -0800:

Excerpts from Clint Byrum's message of 2015-11-06 10:12:21 -0800:

Excerpts from Dan Smith's message of 2015-11-06 09:37:44 -0800:

Worth mentioning that OpenStack releases that come out at the same time
as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
are supported for 5 years by Canonical so are already kind of an LTS.
Support in this context means patches, updates and commercial support
(for a fee).
For paying customers 3 years of patches, updates and commercial support
for April releases, (Kilo, O, Q etc..) is also available.

Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
maintaining an older release for so long is a good use of people or CI
resources, especially given how hard it can be for us to keep even
recent stable releases working and maintained.


The argument in the original post, I think, is that we should not
stand in the way of the vendors continuing to collaborate on stable
maintenance in the upstream context after the EOL date. We already have
distro vendors doing work in the stable branches, but at EOL we push
them off to their respective distro-specific homes.

As much as I'd like everyone to get on the CD train, I think it might
make sense to enable the vendors to not diverge, but instead let them
show up with people and commitment and say "Hey we're going to keep
Juno/Mitaka/etc alive!".

So perhaps what would make sense is defining a process by which they can
make that happen.

Do we need a new process? Aren't the existing stable maintenance
and infrastructure teams clearly defined?

We have this discussion whenever a release is about to go EOL, and
the result is more or less the same each time. The burden of
maintaining stable branches for longer than we do is currently
greater than the resources being applied upstream to do that
maintenance. Until that changes, I don't realistically see us being
able to increase the community's commitment. That's not a lack of
willingness, just an assessment of our current resources.


I tend to agree with you. I only bring up a new process because I wonder
if the distro vendors would even be interested in collaborating on this,
or if this is just sort of "what they do" and we should accept that
they're going to do it outside upstream no matter how easy we make it.

If we do believe that, and are OK with that, then we should not extend
EOL's, and we should make sure users understand that when they choose
the source of their OpenStack software.


Except for the fact that you are now forcing deployers that may or may 
not be ok with paying for paid support to now pay for it... What is the 
adoption rate/expected adoption rate of someone transitioning there 
current cloud (which they did not pay support for) to a paid support model?


Does that require them to redeploy/convert there whole cloud using 
vendors provided packages/deployment model... If so, jeez, that sounds 
iffy...


And if a large majority of deployers aren't able to do that conversion 
(or aren't willing to pay for support) and those same deployers are 
willing to provide developers/others to ensure the old branches continue 
to work and they know the issues of CI and they are willing to stay 
on-top of that (a old-branch-dictator/leader may be needed to ensure 
this?) then meh, I think we as a community should just let those 
deployers have at it (ensuring they keep on working on the old branches 
via what 'old-branch-dictator/leader/group' says is broken/needs fixing...)


My 2 cents,

Josh



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2015-11-06 10:28:41 -0800:
> Excerpts from Clint Byrum's message of 2015-11-06 10:12:21 -0800:
> > Excerpts from Dan Smith's message of 2015-11-06 09:37:44 -0800:
> > > > Worth mentioning that OpenStack releases that come out at the same time
> > > > as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
> > > > are supported for 5 years by Canonical so are already kind of an LTS.
> > > > Support in this context means patches, updates and commercial support
> > > > (for a fee).
> > > > For paying customers 3 years of patches, updates and commercial support
> > > > for April releases, (Kilo, O, Q etc..) is also available.
> > > 
> > > Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
> > > maintaining an older release for so long is a good use of people or CI
> > > resources, especially given how hard it can be for us to keep even
> > > recent stable releases working and maintained.
> > > 
> > 
> > The argument in the original post, I think, is that we should not
> > stand in the way of the vendors continuing to collaborate on stable
> > maintenance in the upstream context after the EOL date. We already have
> > distro vendors doing work in the stable branches, but at EOL we push
> > them off to their respective distro-specific homes.
> > 
> > As much as I'd like everyone to get on the CD train, I think it might
> > make sense to enable the vendors to not diverge, but instead let them
> > show up with people and commitment and say "Hey we're going to keep
> > Juno/Mitaka/etc alive!".
> > 
> > So perhaps what would make sense is defining a process by which they can
> > make that happen.
> 
> Do we need a new process? Aren't the existing stable maintenance
> and infrastructure teams clearly defined?
> 
> We have this discussion whenever a release is about to go EOL, and
> the result is more or less the same each time. The burden of
> maintaining stable branches for longer than we do is currently
> greater than the resources being applied upstream to do that
> maintenance. Until that changes, I don't realistically see us being
> able to increase the community's commitment. That's not a lack of
> willingness, just an assessment of our current resources.

I tend to agree with you. I only bring up a new process because I wonder
if the distro vendors would even be interested in collaborating on this,
or if this is just sort of "what they do" and we should accept that
they're going to do it outside upstream no matter how easy we make it.

If we do believe that, and are OK with that, then we should not extend
EOL's, and we should make sure users understand that when they choose
the source of their OpenStack software.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Jeremy Stanley]

On 2015-11-06 10:12:21 -0800 (-0800), Clint Byrum wrote:
[...]
> Note that it's not just backporters though. It's infra resources too.

Aye, there's the rub. We don't just EOL these branches for fun or
because we hate old things or because ooh shiny squirrel. We EOL
them at a cadence where the community has demonstrated it loses its
ability to keep them healthy and testable (evaluated based on
performance over recent prior cycles because we have to warn
downstreams well in advance as to when they should expect upstream
support to cease).

Downstream maintainers regularly claim they will step up their
assistance upstream to keep stable branches alive if only we'll
extend the lifespan on them, so we tried that with Icehouse and,
based on our experience there, scaled back the lifespan of Juno
again accordingly. Keep in mind that extending support of stable
branches necessarily implies supporting a larger _number_ of stable
branches in parallel. If we switched from 12 months after release to
18 then we're maintaining at least 3 stable branches at any point in
time. If we extend it to 24 months then that's 4 stable branches.

To those who suggest solving this by claiming one is a LTS release
every couple years, you're implying a vastly different upgrade model
than we have now. If we declare Juno is a LTS and leave it supported
another 12 months, then 6 months from now when we EOL stable/kilo
we'll be telling deployers that they have to upgrade from supported
stable/juno through unsupported stable/kilo to supported
stable/icehouse before running stable/mitaka. Or else you're saying
you intend to fix the current inability of our projects to skip
intermediate releases entirely during upgrades (a great idea, and so
I'm thrilled by those of you who intend to make it a reality, we can
revisit the LTS discussion once you finish that).
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Doug Hellmann]

Excerpts from Clint Byrum's message of 2015-11-06 10:12:21 -0800:
> Excerpts from Dan Smith's message of 2015-11-06 09:37:44 -0800:
> > > Worth mentioning that OpenStack releases that come out at the same time
> > > as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
> > > are supported for 5 years by Canonical so are already kind of an LTS.
> > > Support in this context means patches, updates and commercial support
> > > (for a fee).
> > > For paying customers 3 years of patches, updates and commercial support
> > > for April releases, (Kilo, O, Q etc..) is also available.
> > 
> > Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
> > maintaining an older release for so long is a good use of people or CI
> > resources, especially given how hard it can be for us to keep even
> > recent stable releases working and maintained.
> > 
> 
> The argument in the original post, I think, is that we should not
> stand in the way of the vendors continuing to collaborate on stable
> maintenance in the upstream context after the EOL date. We already have
> distro vendors doing work in the stable branches, but at EOL we push
> them off to their respective distro-specific homes.
> 
> As much as I'd like everyone to get on the CD train, I think it might
> make sense to enable the vendors to not diverge, but instead let them
> show up with people and commitment and say "Hey we're going to keep
> Juno/Mitaka/etc alive!".
> 
> So perhaps what would make sense is defining a process by which they can
> make that happen.

Do we need a new process? Aren't the existing stable maintenance
and infrastructure teams clearly defined?

We have this discussion whenever a release is about to go EOL, and
the result is more or less the same each time. The burden of
maintaining stable branches for longer than we do is currently
greater than the resources being applied upstream to do that
maintenance. Until that changes, I don't realistically see us being
able to increase the community's commitment. That's not a lack of
willingness, just an assessment of our current resources.

Doug

> 
> Note that it's not just backporters though. It's infra resources too.
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Clint Byrum]

Excerpts from Erik McCormick's message of 2015-11-06 09:36:44 -0800:
> On Fri, Nov 6, 2015 at 12:28 PM, Mark Baker  wrote:
> > Worth mentioning that OpenStack releases that come out at the same time as
> > Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka) are
> > supported for 5 years by Canonical so are already kind of an LTS. Support in
> > this context means patches, updates and commercial support (for a fee).
> > For paying customers 3 years of patches, updates and commercial support for
> > April releases, (Kilo, O, Q etc..) is also available.
> >
> 
> Does that mean that you are actually backporting and gate testing
> patches downstream that aren't being done upstream? I somehow doubt
> it, but if so, then it would be great if you could lead some sort of
> initiative to push those patches back upstream.
> 

If Canonical and Ubuntu still work the way they worked when I was
involved, then yes and no. The initial patches still happen upstream,
in trunk. But the difference is the backporting can't happen upstream in
stable branches after EOL, because those branches are shut down. That
seems a shame, as the community at large would likely be better served
if the vendors can continue to land their stable patches for as long as
they're working on them.

That said, I think it would take a bit of a shift in participation to get
the needed resources in the right place (like infra) to make that happen.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Clint Byrum]

Excerpts from Dan Smith's message of 2015-11-06 09:37:44 -0800:
> > Worth mentioning that OpenStack releases that come out at the same time
> > as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
> > are supported for 5 years by Canonical so are already kind of an LTS.
> > Support in this context means patches, updates and commercial support
> > (for a fee).
> > For paying customers 3 years of patches, updates and commercial support
> > for April releases, (Kilo, O, Q etc..) is also available.
> 
> Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
> maintaining an older release for so long is a good use of people or CI
> resources, especially given how hard it can be for us to keep even
> recent stable releases working and maintained.
> 

The argument in the original post, I think, is that we should not
stand in the way of the vendors continuing to collaborate on stable
maintenance in the upstream context after the EOL date. We already have
distro vendors doing work in the stable branches, but at EOL we push
them off to their respective distro-specific homes.

As much as I'd like everyone to get on the CD train, I think it might
make sense to enable the vendors to not diverge, but instead let them
show up with people and commitment and say "Hey we're going to keep
Juno/Mitaka/etc alive!".

So perhaps what would make sense is defining a process by which they can
make that happen.

Note that it's not just backporters though. It's infra resources too.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Erik McCormick]

On Fri, Nov 6, 2015 at 12:28 PM, Mark Baker  wrote:
> Worth mentioning that OpenStack releases that come out at the same time as
> Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka) are
> supported for 5 years by Canonical so are already kind of an LTS. Support in
> this context means patches, updates and commercial support (for a fee).
> For paying customers 3 years of patches, updates and commercial support for
> April releases, (Kilo, O, Q etc..) is also available.
>

Does that mean that you are actually backporting and gate testing
patches downstream that aren't being done upstream? I somehow doubt
it, but if so, then it would be great if you could lead some sort of
initiative to push those patches back upstream.


-Erik

>
>
> Best Regards
>
>
> Mark Baker
>
> On Fri, Nov 6, 2015 at 5:03 PM, James King  wrote:
>>
>> +1 for some sort of LTS release system.
>>
>> Telcos and risk-averse organizations working with sensitive data might not
>> be able to upgrade nearly as fast as the releases keep coming out. From the
>> summit in Japan it sounds like companies running some fairly critical public
>> infrastructure on Openstack aren’t going to be upgrading to Kilo any time
>> soon.
>>
>> Public clouds might even benefit from this. I know we (Dreamcompute) are
>> working towards tracking the upstream releases closer… but it’s not feasible
>> for everyone.
>>
>> I’m not sure whether the resources exist to do this but it’d be a nice to
>> have, imho.
>>
>> > On Nov 6, 2015, at 11:47 AM, Donald Talton 
>> > wrote:
>> >
>> > I like the idea of LTS releases.
>> >
>> > Speaking to my own deployments, there are many new features we are not
>> > interested in, and wouldn't be, until we can get organizational (cultural)
>> > change in place, or see stability and scalability.
>> >
>> > We can't rely on, or expect, that orgs will move to the CI/CD model for
>> > infra, when they aren't even ready to do that for their own apps. It's 
>> > still
>> > a new "paradigm" for many of us. CI/CD requires a considerable engineering
>> > effort, and given that the decision to "switch" to OpenStack is often 
>> > driven
>> > by cost-savings over enterprise virtualization, adding those costs back in
>> > via engineering salaries doesn't make fiscal sense.
>> >
>> > My big argument is that if Icehouse/Juno works and is stable, and I
>> > don't need newer features from subsequent releases, why would I expend the
>> > effort until such a time that I do want those features? Thankfully there 
>> > are
>> > vendors that understand this. Keeping up with the release cycle just for 
>> > the
>> > sake of keeping up with the release cycle is exhausting.
>> >
>> > -Original Message-
>> > From: Tony Breeds [mailto:t...@bakeyournoodle.com]
>> > Sent: Thursday, November 05, 2015 11:15 PM
>> > To: OpenStack Development Mailing List
>> > Cc: openstack-operat...@lists.openstack.org
>> > Subject: [Openstack-operators] [stable][all] Keeping Juno "alive" for
>> > longer.
>> >
>> > Hello all,
>> >
>> > I'll start by acknowledging that this is a big and complex issue and I
>> > do not claim to be across all the view points, nor do I claim to be
>> > particularly persuasive ;P
>> >
>> > Having stated that, I'd like to seek constructive feedback on the idea
>> > of keeping Juno around for a little longer.  During the summit I spoke to a
>> > number of operators, vendors and developers on this topic.  There was some
>> > support and some "That's crazy pants!" responses.  I clearly didn't make it
>> > around to everyone, hence this email.
>> >
>> > Acknowledging my affiliation/bias:  I work for Rackspace in the private
>> > cloud team.  We support a number of customers currently running Juno that
>> > are, for a variety of reasons, challenged by the Kilo upgrade.
>> >
>> > Here is a summary of the main points that have come up in my
>> > conversations, both for and against.
>> >
>> > Keep Juno:
>> > * According to the current user survey[1] Icehouse still has the
>> >   biggest install base in production clouds.  Juno is second, which
>> > makes
>> >   sense. If we EOL Juno this month that means ~75% of production clouds
>> >   will be running an EOL'd release.  Clearly many of these operators
>> > have
>> >   support contracts from their vendor, so those operators won't be left
>> >   completely adrift, but I believe it's the vendors that benefit from
>> > keeping
>> >   Juno around. By working together *in the community* we'll see the best
>> >   results.
>> >
>> > * We only recently EOL'd Icehouse[2].  Sure it was well communicated,
>> > but we
>> >   still have a huge Icehouse/Juno install base.
>> >
>> > For me this is pretty compelling but for balance 
>> >
>> > Keep the current plan and EOL Juno Real Soon Now:
>> > * There is also no ignoring the elephant in the room that with HP
>> > stepping
>> >   back from public cloud there are questions about our CI capacity, and
>> >   keeping Juno will have an impact on that crit

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Dan Smith]

> Worth mentioning that OpenStack releases that come out at the same time
> as Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka)
> are supported for 5 years by Canonical so are already kind of an LTS.
> Support in this context means patches, updates and commercial support
> (for a fee).
> For paying customers 3 years of patches, updates and commercial support
> for April releases, (Kilo, O, Q etc..) is also available.

Yeah. IMHO, this is what you pay your vendor for. I don't think upstream
maintaining an older release for so long is a good use of people or CI
resources, especially given how hard it can be for us to keep even
recent stable releases working and maintained.

--Dan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Mark Baker]

Worth mentioning that OpenStack releases that come out at the same time as
Ubuntu LTS releases (12.04 + Essex, 14.04 + Icehouse, 16.04 + Mitaka) are
supported for 5 years by Canonical so are already kind of an LTS. Support
in this context means patches, updates and commercial support (for a fee).
For paying customers 3 years of patches, updates and commercial support for
April releases, (Kilo, O, Q etc..) is also available.



Best Regards


Mark Baker

On Fri, Nov 6, 2015 at 5:03 PM, James King  wrote:

> +1 for some sort of LTS release system.
>
> Telcos and risk-averse organizations working with sensitive data might not
> be able to upgrade nearly as fast as the releases keep coming out. From the
> summit in Japan it sounds like companies running some fairly critical
> public infrastructure on Openstack aren’t going to be upgrading to Kilo any
> time soon.
>
> Public clouds might even benefit from this. I know we (Dreamcompute) are
> working towards tracking the upstream releases closer… but it’s not
> feasible for everyone.
>
> I’m not sure whether the resources exist to do this but it’d be a nice to
> have, imho.
>
> > On Nov 6, 2015, at 11:47 AM, Donald Talton 
> wrote:
> >
> > I like the idea of LTS releases.
> >
> > Speaking to my own deployments, there are many new features we are not
> interested in, and wouldn't be, until we can get organizational (cultural)
> change in place, or see stability and scalability.
> >
> > We can't rely on, or expect, that orgs will move to the CI/CD model for
> infra, when they aren't even ready to do that for their own apps. It's
> still a new "paradigm" for many of us. CI/CD requires a considerable
> engineering effort, and given that the decision to "switch" to OpenStack is
> often driven by cost-savings over enterprise virtualization, adding those
> costs back in via engineering salaries doesn't make fiscal sense.
> >
> > My big argument is that if Icehouse/Juno works and is stable, and I
> don't need newer features from subsequent releases, why would I expend the
> effort until such a time that I do want those features? Thankfully there
> are vendors that understand this. Keeping up with the release cycle just
> for the sake of keeping up with the release cycle is exhausting.
> >
> > -Original Message-
> > From: Tony Breeds [mailto:t...@bakeyournoodle.com]
> > Sent: Thursday, November 05, 2015 11:15 PM
> > To: OpenStack Development Mailing List
> > Cc: openstack-operat...@lists.openstack.org
> > Subject: [Openstack-operators] [stable][all] Keeping Juno "alive" for
> longer.
> >
> > Hello all,
> >
> > I'll start by acknowledging that this is a big and complex issue and I
> do not claim to be across all the view points, nor do I claim to be
> particularly persuasive ;P
> >
> > Having stated that, I'd like to seek constructive feedback on the idea
> of keeping Juno around for a little longer.  During the summit I spoke to a
> number of operators, vendors and developers on this topic.  There was some
> support and some "That's crazy pants!" responses.  I clearly didn't make it
> around to everyone, hence this email.
> >
> > Acknowledging my affiliation/bias:  I work for Rackspace in the private
> cloud team.  We support a number of customers currently running Juno that
> are, for a variety of reasons, challenged by the Kilo upgrade.
> >
> > Here is a summary of the main points that have come up in my
> conversations, both for and against.
> >
> > Keep Juno:
> > * According to the current user survey[1] Icehouse still has the
> >   biggest install base in production clouds.  Juno is second, which makes
> >   sense. If we EOL Juno this month that means ~75% of production clouds
> >   will be running an EOL'd release.  Clearly many of these operators have
> >   support contracts from their vendor, so those operators won't be left
> >   completely adrift, but I believe it's the vendors that benefit from
> keeping
> >   Juno around. By working together *in the community* we'll see the best
> >   results.
> >
> > * We only recently EOL'd Icehouse[2].  Sure it was well communicated,
> but we
> >   still have a huge Icehouse/Juno install base.
> >
> > For me this is pretty compelling but for balance 
> >
> > Keep the current plan and EOL Juno Real Soon Now:
> > * There is also no ignoring the elephant in the room that with HP
> stepping
> >   back from public cloud there are questions about our CI capacity, and
> >   keeping Juno will have an impact on that critical resource.
> >
> > * Juno (and other stable/*) resources have a non-zero impact on *every*
> >   project, esp. @infra and release management.  We need to ensure this
> >   isn't too much of a burden.  This mostly means we need enough
> trustworthy
> >   volunteers.
> >
> > * Juno is also tied up with Python 2.6 support. When
> >   Juno goes, so will Python 2.6 which is a happy feeling for a number of
> >   people, and more importantly reduces complexity in our project
> >   infrastructure.
> >
> > * 

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[James King]

+1 for some sort of LTS release system.

Telcos and risk-averse organizations working with sensitive data might not be 
able to upgrade nearly as fast as the releases keep coming out. From the summit 
in Japan it sounds like companies running some fairly critical public 
infrastructure on Openstack aren’t going to be upgrading to Kilo any time soon.

Public clouds might even benefit from this. I know we (Dreamcompute) are 
working towards tracking the upstream releases closer… but it’s not feasible 
for everyone.

I’m not sure whether the resources exist to do this but it’d be a nice to have, 
imho.

> On Nov 6, 2015, at 11:47 AM, Donald Talton  wrote:
> 
> I like the idea of LTS releases. 
> 
> Speaking to my own deployments, there are many new features we are not 
> interested in, and wouldn't be, until we can get organizational (cultural) 
> change in place, or see stability and scalability. 
> 
> We can't rely on, or expect, that orgs will move to the CI/CD model for 
> infra, when they aren't even ready to do that for their own apps. It's still 
> a new "paradigm" for many of us. CI/CD requires a considerable engineering 
> effort, and given that the decision to "switch" to OpenStack is often driven 
> by cost-savings over enterprise virtualization, adding those costs back in 
> via engineering salaries doesn't make fiscal sense.
> 
> My big argument is that if Icehouse/Juno works and is stable, and I don't 
> need newer features from subsequent releases, why would I expend the effort 
> until such a time that I do want those features? Thankfully there are vendors 
> that understand this. Keeping up with the release cycle just for the sake of 
> keeping up with the release cycle is exhausting.
> 
> -Original Message-
> From: Tony Breeds [mailto:t...@bakeyournoodle.com] 
> Sent: Thursday, November 05, 2015 11:15 PM
> To: OpenStack Development Mailing List
> Cc: openstack-operat...@lists.openstack.org
> Subject: [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.
> 
> Hello all,
> 
> I'll start by acknowledging that this is a big and complex issue and I do not 
> claim to be across all the view points, nor do I claim to be particularly 
> persuasive ;P
> 
> Having stated that, I'd like to seek constructive feedback on the idea of 
> keeping Juno around for a little longer.  During the summit I spoke to a 
> number of operators, vendors and developers on this topic.  There was some 
> support and some "That's crazy pants!" responses.  I clearly didn't make it 
> around to everyone, hence this email.
> 
> Acknowledging my affiliation/bias:  I work for Rackspace in the private cloud 
> team.  We support a number of customers currently running Juno that are, for 
> a variety of reasons, challenged by the Kilo upgrade.
> 
> Here is a summary of the main points that have come up in my conversations, 
> both for and against.
> 
> Keep Juno:
> * According to the current user survey[1] Icehouse still has the
>   biggest install base in production clouds.  Juno is second, which makes
>   sense. If we EOL Juno this month that means ~75% of production clouds
>   will be running an EOL'd release.  Clearly many of these operators have
>   support contracts from their vendor, so those operators won't be left 
>   completely adrift, but I believe it's the vendors that benefit from keeping
>   Juno around. By working together *in the community* we'll see the best
>   results.
> 
> * We only recently EOL'd Icehouse[2].  Sure it was well communicated, but we
>   still have a huge Icehouse/Juno install base.
> 
> For me this is pretty compelling but for balance  
> 
> Keep the current plan and EOL Juno Real Soon Now:
> * There is also no ignoring the elephant in the room that with HP stepping
>   back from public cloud there are questions about our CI capacity, and
>   keeping Juno will have an impact on that critical resource.
> 
> * Juno (and other stable/*) resources have a non-zero impact on *every*
>   project, esp. @infra and release management.  We need to ensure this
>   isn't too much of a burden.  This mostly means we need enough trustworthy
>   volunteers.
> 
> * Juno is also tied up with Python 2.6 support. When
>   Juno goes, so will Python 2.6 which is a happy feeling for a number of
>   people, and more importantly reduces complexity in our project
>   infrastructure.
> 
> * Even if we keep Juno for 6 months or 1 year, that doesn't help vendors
>   that are "on the hook" for multiple years of support, so for that case
>   we're really only delaying the inevitable.
> 
> * Some number of the production clouds may never migrate from $version, in
>   which case longer support for Juno isn't going to help them.
> 
> 
> I'm sure these question were well discussed at the VYR summit where we set 
> the EOL date for Juno, but I was new then :) What I'm asking is:
> 
> 1) Is it even possible to keep Juno alive (is the impact on the project as
>   a whole acceptable)?
> 
> Assuming a positi

Re: [openstack-dev] [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

[Donald Talton]

I like the idea of LTS releases. 

Speaking to my own deployments, there are many new features we are not 
interested in, and wouldn't be, until we can get organizational (cultural) 
change in place, or see stability and scalability. 

We can't rely on, or expect, that orgs will move to the CI/CD model for infra, 
when they aren't even ready to do that for their own apps. It's still a new 
"paradigm" for many of us. CI/CD requires a considerable engineering effort, 
and given that the decision to "switch" to OpenStack is often driven by 
cost-savings over enterprise virtualization, adding those costs back in via 
engineering salaries doesn't make fiscal sense.

My big argument is that if Icehouse/Juno works and is stable, and I don't need 
newer features from subsequent releases, why would I expend the effort until 
such a time that I do want those features? Thankfully there are vendors that 
understand this. Keeping up with the release cycle just for the sake of keeping 
up with the release cycle is exhausting.

-Original Message-
From: Tony Breeds [mailto:t...@bakeyournoodle.com] 
Sent: Thursday, November 05, 2015 11:15 PM
To: OpenStack Development Mailing List
Cc: openstack-operat...@lists.openstack.org
Subject: [Openstack-operators] [stable][all] Keeping Juno "alive" for longer.

Hello all,

I'll start by acknowledging that this is a big and complex issue and I do not 
claim to be across all the view points, nor do I claim to be particularly 
persuasive ;P

Having stated that, I'd like to seek constructive feedback on the idea of 
keeping Juno around for a little longer.  During the summit I spoke to a number 
of operators, vendors and developers on this topic.  There was some support and 
some "That's crazy pants!" responses.  I clearly didn't make it around to 
everyone, hence this email.

Acknowledging my affiliation/bias:  I work for Rackspace in the private cloud 
team.  We support a number of customers currently running Juno that are, for a 
variety of reasons, challenged by the Kilo upgrade.

Here is a summary of the main points that have come up in my conversations, 
both for and against.

Keep Juno:
 * According to the current user survey[1] Icehouse still has the
   biggest install base in production clouds.  Juno is second, which makes
   sense. If we EOL Juno this month that means ~75% of production clouds
   will be running an EOL'd release.  Clearly many of these operators have
   support contracts from their vendor, so those operators won't be left 
   completely adrift, but I believe it's the vendors that benefit from keeping
   Juno around. By working together *in the community* we'll see the best
   results.

 * We only recently EOL'd Icehouse[2].  Sure it was well communicated, but we
   still have a huge Icehouse/Juno install base.

For me this is pretty compelling but for balance  

Keep the current plan and EOL Juno Real Soon Now:
 * There is also no ignoring the elephant in the room that with HP stepping
   back from public cloud there are questions about our CI capacity, and
   keeping Juno will have an impact on that critical resource.

 * Juno (and other stable/*) resources have a non-zero impact on *every*
   project, esp. @infra and release management.  We need to ensure this
   isn't too much of a burden.  This mostly means we need enough trustworthy
   volunteers.

 * Juno is also tied up with Python 2.6 support. When
   Juno goes, so will Python 2.6 which is a happy feeling for a number of
   people, and more importantly reduces complexity in our project
   infrastructure.

 * Even if we keep Juno for 6 months or 1 year, that doesn't help vendors
   that are "on the hook" for multiple years of support, so for that case
   we're really only delaying the inevitable.

 * Some number of the production clouds may never migrate from $version, in
   which case longer support for Juno isn't going to help them.


I'm sure these question were well discussed at the VYR summit where we set the 
EOL date for Juno, but I was new then :) What I'm asking is:

1) Is it even possible to keep Juno alive (is the impact on the project as
   a whole acceptable)?

Assuming a positive answer:

2) Who's going to do the work?
- Me, who else?
3) What do we do if people don't actually do the work but we as a community
   have made a commitment?
4) If we keep Juno alive for $some_time, does that imply we also bump the
   life cycle on Kilo and liberty and Mitaka etc?

Yours Tony.

[1] http://www.openstack.org/assets/survey/Public-User-Survey-Report.pdf
(page 20)
[2] http://git.openstack.org/cgit/openstack/nova/tag/?h=icehouse-eol


This email and any files transmitted with it are confidential, proprietary and 
intended solely for the individual or entity to whom they are addressed. If you 
have received this email in error please delete it immediately.
__
OpenStack Development Mailing List (not for usage quest