Re: [openstack-dev] [ironic] upgrade support between which versions of ironic?

[Devananda van der Veen]

On Wed, Apr 20, 2016 at 5:38 AM, Mathieu Mitchell 
wrote:

>
>
> On 2016-04-19 11:29 PM, Tan, Lin wrote:
>
>> I agree this is reasonable to support all these cases in “cold upgrades”
>> but in supports-rolling-upgrade (live upgrade in another word) case it is
>> different and complicated and not necessary,
>>
>> During rolling upgrade, we will have old/new services co-existed, and we
>> need to make services compatible which need some extra code work and this
>> is the main purpose of spec [1]. And as far as I can see, we are  not
>> allowed to skip over releases when rolling upgrading.  So my point is
>> support name release is enough.
>>
>> 1. Because even if we want to support major number release, admins have
>> to upgrade from 5.0 -> 6.0 then 6.0 -> 7.0 in Ruby’s case of 5.0.0, 5.1.0
>> == Mitaka, 5.2.0, 6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton. And we might
>> have a higher release frequency in the future. So it’s too much work for
>> upgrade a service every six months.
>>
>> 2. As we usually rolling upgrade the whole cloud, not for ironic only.
>> For example, other projects will upgrade from Mitaka to Netwon, there is
>> not much sense to upgrade Ironic from 5.0 -> 6.0 only.
>>
>>
> As an operator, I disagree with that statement. We follow different
> upgrade paths for Ironic and Glance for example. My area of concern around
> Ironic is compatibility with Nova and Neutron. If we can prove via CI that
> Nova on an older version still works with Ironic on master and vice versa,
> we will successfully avoid having to do them in a lockstep.


I agree that we need to test this scenario and assert via CI that it is
possible to upgrade Nova and Ironic separately, and as we're adding Neutron
integration, we will need to assert the same thing there. Let's call this a
"cloud rolling upgrade". We'll want to run this test on changes in Nova as
well.

We can test that with the grenade partial job. I do not think we need to
test the upgrade sequence in both directions, though -- as we integrate
with more services, that would explode exponentially. Instead, we should
proscribe an order to the upgrades (and document it clearly) and then test
that ordering. For starters, I would upgrade Ironic first (our API needs to
remain backwards compatible) and then upgrade Nova. That said, I'm not sure
how adding Neutron, or eventually Cinder, will affect the upgrade sequence.

Just to be clear, this doesn't replace the need to assert a "service
rolling upgrade", eg. where different releases of the Ironic API and
Conductor services are run at the same time, but we could do that in a test
environment that is only running Ironic, and wouldn't need to trigger that
test on changes in Nova, for example.

--deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] upgrade support between which versions of ironic?

[Mathieu Mitchell]

On 2016-04-19 11:29 PM, Tan, Lin wrote:

I agree this is reasonable to support all these cases in “cold upgrades” but in 
supports-rolling-upgrade (live upgrade in another word) case it is different 
and complicated and not necessary,

During rolling upgrade, we will have old/new services co-existed, and we need 
to make services compatible which need some extra code work and this is the 
main purpose of spec [1]. And as far as I can see, we are  not allowed to skip 
over releases when rolling upgrading.  So my point is support name release is 
enough.

1. Because even if we want to support major number release, admins have to upgrade 
from 5.0 -> 6.0 then 6.0 -> 7.0 in Ruby’s case of 5.0.0, 5.1.0 == Mitaka, 
5.2.0, 6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton. And we might have a higher 
release frequency in the future. So it’s too much work for upgrade a service every 
six months.

2. As we usually rolling upgrade the whole cloud, not for ironic only. For 
example, other projects will upgrade from Mitaka to Netwon, there is not much 
sense to upgrade Ironic from 5.0 -> 6.0 only.



As an operator, I disagree with that statement. We follow different 
upgrade paths for Ironic and Glance for example. My area of concern 
around Ironic is compatibility with Nova and Neutron. If we can prove 
via CI that Nova on an older version still works with Ironic on master 
and vice versa, we will successfully avoid having to do them in a lockstep.




At last, we should add a multi-node grenade CI to test the rolling upgrade 
mechanism was not broken. Here is my suggestion, we will have two nodes, Node A 
and Node B. Node A will run both of last named release of ironic-api and 
ironic-conductor and node B will run last named release ironic-conductor only. 
Multi-node grenade CI will only upgrade Node A and then we can test the 
interaction between new SHA of ironic-api and new SHA/last named release of 
ironic-conductor still works.  This should also apply to stable branch.


That's a test case for "live upgrade", +1 :)

Mathieu




B.R

Tan

[1] https://review.openstack.org/299245

From: Devananda van der Veen [mailto:devananda@gmail.com]
Sent: Wednesday, April 20, 2016 5:12 AM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [ironic] upgrade support between which versions of 
ironic?

Thanks for starting the thread, Ruby.


We need to first establish a grenade job to test "cold upgrades" and assert the 
supports-upgrade tag. I believe Ironic meets all the criteria for that tag except:
- having a job that tests it (so, you know, it might be broken and I might be 
wrong)
- having operator documentation describing the process (it should be here: 
http://docs.openstack.org/developer/ironic/deploy/upgrade-guide.html ) but all 
we have are release-specific upgrade notes.

I think all of the scenario you outline are valid upgrade paths for an operator, and we 
should try to allow all of them to work. However, some of them can be covered by one test 
case, and you also missed some things I think need to be covered. Also, I'm interpreting 
the word "master" in your scenarios to indicate the proposed change to our 
master branch, since we do pre-merge testing

So, here are the test cases I think we need to cover:

=== run on proposed changes to master ===

F. current master to new SHA

We need to ensure that we can upgrade master to the code being proposed. You 
listed this last, but I think it's actually the most important one.

D. last named release to new SHA
E. last numbered release to new SHA

Because we cut new releases from master, this is the basis of testing the 
upgrade between sequential (named or numbered) releases before we cut a new 
(named or numbered) release, and is our most important test to ensure that we 
don't break most operators. Based on the user survey, most operators are using 
named releases, so if we are resource constrained, I would prefer to cover (D) 
before (E)

=== run on proposed changes to a stable branch ===

A. stable/N-1 -> new SHA -> [ stable/N+1 or current master]

We don't need to test upgrades between two named releases (eg. Liberty -> 
Mitaka) every time we land a new patch on the master branch, but we do need to 
test any time we land a change on a stable branch. Changes to the most recent 
stable branch should be upgrade-tested to current master, whereas changes to any 
stable branch prior to that should get tested to the subsequent sequential release.

Eg, a backport to stable/liberty should trigger an upgrade test for both (stable/kilo -> 
newpatch) and (newpatch  -> stable/mitaka), whereas a backport to stable/mitaka should 
trigger a test for (stable/liberty -> newpatch) and (newpatch -> master)


Once we've done that, then yes, I agree we should also work towards asserting 
supports-rolling-upgrade. That will requi

Re: [openstack-dev] [ironic] upgrade support between which versions of ironic?

[Tan, Lin]

I agree this is reasonable to support all these cases in “cold upgrades” but in 
supports-rolling-upgrade (live upgrade in another word) case it is different 
and complicated and not necessary,

During rolling upgrade, we will have old/new services co-existed, and we need 
to make services compatible which need some extra code work and this is the 
main purpose of spec [1]. And as far as I can see, we are  not allowed to skip 
over releases when rolling upgrading.  So my point is support name release is 
enough.

1. Because even if we want to support major number release, admins have to 
upgrade from 5.0 -> 6.0 then 6.0 -> 7.0 in Ruby’s case of 5.0.0, 5.1.0 == 
Mitaka, 5.2.0, 6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton. And we might have a 
higher release frequency in the future. So it’s too much work for upgrade a 
service every six months.

2. As we usually rolling upgrade the whole cloud, not for ironic only. For 
example, other projects will upgrade from Mitaka to Netwon, there is not much 
sense to upgrade Ironic from 5.0 -> 6.0 only.

At last, we should add a multi-node grenade CI to test the rolling upgrade 
mechanism was not broken. Here is my suggestion, we will have two nodes, Node A 
and Node B. Node A will run both of last named release of ironic-api and 
ironic-conductor and node B will run last named release ironic-conductor only. 
Multi-node grenade CI will only upgrade Node A and then we can test the 
interaction between new SHA of ironic-api and new SHA/last named release of 
ironic-conductor still works.  This should also apply to stable branch.


B.R

Tan

[1] https://review.openstack.org/299245

From: Devananda van der Veen [mailto:devananda@gmail.com]
Sent: Wednesday, April 20, 2016 5:12 AM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [ironic] upgrade support between which versions of 
ironic?

Thanks for starting the thread, Ruby.


We need to first establish a grenade job to test "cold upgrades" and assert the 
supports-upgrade tag. I believe Ironic meets all the criteria for that tag 
except:
- having a job that tests it (so, you know, it might be broken and I might be 
wrong)
- having operator documentation describing the process (it should be here: 
http://docs.openstack.org/developer/ironic/deploy/upgrade-guide.html ) but all 
we have are release-specific upgrade notes.

I think all of the scenario you outline are valid upgrade paths for an 
operator, and we should try to allow all of them to work. However, some of them 
can be covered by one test case, and you also missed some things I think need 
to be covered. Also, I'm interpreting the word "master" in your scenarios to 
indicate the proposed change to our master branch, since we do pre-merge 
testing

So, here are the test cases I think we need to cover:

=== run on proposed changes to master ===

F. current master to new SHA

We need to ensure that we can upgrade master to the code being proposed. You 
listed this last, but I think it's actually the most important one.

D. last named release to new SHA
E. last numbered release to new SHA

Because we cut new releases from master, this is the basis of testing the 
upgrade between sequential (named or numbered) releases before we cut a new 
(named or numbered) release, and is our most important test to ensure that we 
don't break most operators. Based on the user survey, most operators are using 
named releases, so if we are resource constrained, I would prefer to cover (D) 
before (E)

=== run on proposed changes to a stable branch ===

A. stable/N-1 -> new SHA -> [ stable/N+1 or current master]

We don't need to test upgrades between two named releases (eg. Liberty -> 
Mitaka) every time we land a new patch on the master branch, but we do need to 
test any time we land a change on a stable branch. Changes to the most recent 
stable branch should be upgrade-tested to current master, whereas changes to 
any stable branch prior to that should get tested to the subsequent sequential 
release.

Eg, a backport to stable/liberty should trigger an upgrade test for both 
(stable/kilo -> newpatch) and (newpatch  -> stable/mitaka), whereas a backport 
to stable/mitaka should trigger a test for (stable/liberty -> newpatch) and 
(newpatch -> master)


Once we've done that, then yes, I agree we should also work towards asserting 
supports-rolling-upgrade. That will require a partial upgrade job, eg. where we 
run >1 instance of the API and Conductor services and upgrade some of them.

Regards,
Devananda



On Tue, Apr 19, 2016 at 12:52 PM, Ruby Loo 
<opensr...@gmail.com<mailto:opensr...@gmail.com>> wrote:
Hi,
Currently, ironic doesn't support ("live", "online", "rolling", or 
minimal-downtime) upgrades between named versions of ironic. (Where "named 
version" is the final release or

Re: [openstack-dev] [ironic] upgrade support between which versions of ironic?

[Devananda van der Veen]

Thanks for starting the thread, Ruby.


We need to first establish a grenade job to test "cold upgrades" and assert
the supports-upgrade tag. I believe Ironic meets all the criteria for that
tag except:
- having a job that tests it (so, you know, it might be broken and I might
be wrong)
- having operator documentation describing the process (it should be here:
http://docs.openstack.org/developer/ironic/deploy/upgrade-guide.html ) but
all we have are release-specific upgrade notes.

I think all of the scenario you outline are valid upgrade paths for an
operator, and we should try to allow all of them to work. However, some of
them can be covered by one test case, and you also missed some things I
think need to be covered. Also, I'm interpreting the word "master" in your
scenarios to indicate the proposed change to our master branch, since we do
pre-merge testing

So, here are the test cases I think we need to cover:

=== run on proposed changes to master ===

F. current master to new SHA

We need to ensure that we can upgrade master to the code being proposed.
You listed this last, but I think it's actually the most important one.

D. last named release to new SHA
E. last numbered release to new SHA

Because we cut new releases from master, this is the basis of testing the
upgrade between sequential (named or numbered) releases before we cut a new
(named or numbered) release, and is our most important test to ensure that
we don't break most operators. Based on the user survey, most operators are
using named releases, so if we are resource constrained, I would prefer to
cover (D) before (E)

=== run on proposed changes to a stable branch ===

A. stable/N-1 -> new SHA -> [ stable/N+1 or current master]

We don't need to test upgrades between two named releases (eg. Liberty ->
Mitaka) every time we land a new patch on the master branch, but we do need
to test any time we land a change on a stable branch. Changes to the most
recent stable branch should be upgrade-tested to current master, whereas
changes to any stable branch prior to that should get tested to the
subsequent sequential release.

Eg, a backport to stable/liberty should trigger an upgrade test for both
(stable/kilo -> newpatch) and (newpatch  -> stable/mitaka), whereas a
backport to stable/mitaka should trigger a test for (stable/liberty ->
newpatch) and (newpatch -> master)


Once we've done that, then yes, I agree we should also work towards
asserting supports-rolling-upgrade. That will require a partial upgrade
job, eg. where we run >1 instance of the API and Conductor services and
upgrade some of them.

Regards,
Devananda



On Tue, Apr 19, 2016 at 12:52 PM, Ruby Loo  wrote:

> Hi,
>
> Currently, ironic doesn't support ("live", "online", "rolling", or
> minimal-downtime) upgrades between named versions of ironic. (Where "named
> version" is the final release or stable release that is associated with a
> development cycle). So for example, Liberty -> Mitaka release.
>
> We've been working towards that, and have a spec [1] and a design session
> [2] at the Austin Summit. Upon reading the spec, I started to wonder --
> what do we mean/want, when we talk about supporting upgrades. Do we want to
> support:
> A. sequential named releases, eg. Mitaka -> Newton
> B. sequential major releases, eg 4.x -> 5.0; 5.0 -> 6.1
> C. sequential minor releases, eg 5.1 -> 5.2
> D. last named release to master
> E. last release (major or minor) to master
> F. some-SHA-more-recent-than-last-named to master. This could be some
> numbered (major or minor) release.
>
> Keep in mind that ironic may release any number of numbered releases
> between two named releases, so eg. 4.3.0, 5.0.0, 5.1.0 == Mitaka, 5.2.0,
> 6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton.
>
> Note that there are two governance tags: supports-upgrade[3] and
> supports-rolling-upgrade[4], and I believe our goal is for
> supports-rolling-upgrade.
>
> I think and hope that we can all agree that A. is a must :)
>
> What constitutes a major release versus a minor release may have
> implications in this, but that should be in a separate discussion.
>
> What do people think?
>
> --ruby
>
> [1] https://review.openstack.org/299245
> [2]
> https://www.openstack.org/summit/austin-2016/summit-schedule/events/9267
> [3]
> https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-upgrade.rst
> [4]
> https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-rolling-upgrade.rst
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

[openstack-dev] [ironic] upgrade support between which versions of ironic?

[Ruby Loo]

Hi,

Currently, ironic doesn't support ("live", "online", "rolling", or
minimal-downtime) upgrades between named versions of ironic. (Where "named
version" is the final release or stable release that is associated with a
development cycle). So for example, Liberty -> Mitaka release.

We've been working towards that, and have a spec [1] and a design session
[2] at the Austin Summit. Upon reading the spec, I started to wonder --
what do we mean/want, when we talk about supporting upgrades. Do we want to
support:
A. sequential named releases, eg. Mitaka -> Newton
B. sequential major releases, eg 4.x -> 5.0; 5.0 -> 6.1
C. sequential minor releases, eg 5.1 -> 5.2
D. last named release to master
E. last release (major or minor) to master
F. some-SHA-more-recent-than-last-named to master. This could be some
numbered (major or minor) release.

Keep in mind that ironic may release any number of numbered releases
between two named releases, so eg. 4.3.0, 5.0.0, 5.1.0 == Mitaka, 5.2.0,
6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton.

Note that there are two governance tags: supports-upgrade[3] and
supports-rolling-upgrade[4], and I believe our goal is for
supports-rolling-upgrade.

I think and hope that we can all agree that A. is a must :)

What constitutes a major release versus a minor release may have
implications in this, but that should be in a separate discussion.

What do people think?

--ruby

[1] https://review.openstack.org/299245
[2] https://www.openstack.org/summit/austin-2016/summit-schedule/events/9267
[3]
https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-upgrade.rst
[4]
https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-rolling-upgrade.rst
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev