On 2016-04-19 11:29 PM, Tan, Lin wrote:
I agree this is reasonable to support all these cases in “cold upgrades” but in supports-rolling-upgrade (live upgrade in another word) case it is different and complicated and not necessary, During rolling upgrade, we will have old/new services co-existed, and we need to make services compatible which need some extra code work and this is the main purpose of spec [1]. And as far as I can see, we are not allowed to skip over releases when rolling upgrading. So my point is support name release is enough. 1. Because even if we want to support major number release, admins have to upgrade from 5.0 -> 6.0 then 6.0 -> 7.0 in Ruby’s case of 5.0.0, 5.1.0 == Mitaka, 5.2.0, 6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton. And we might have a higher release frequency in the future. So it’s too much work for upgrade a service every six months. 2. As we usually rolling upgrade the whole cloud, not for ironic only. For example, other projects will upgrade from Mitaka to Netwon, there is not much sense to upgrade Ironic from 5.0 -> 6.0 only.
As an operator, I disagree with that statement. We follow different upgrade paths for Ironic and Glance for example. My area of concern around Ironic is compatibility with Nova and Neutron. If we can prove via CI that Nova on an older version still works with Ironic on master and vice versa, we will successfully avoid having to do them in a lockstep.
At last, we should add a multi-node grenade CI to test the rolling upgrade mechanism was not broken. Here is my suggestion, we will have two nodes, Node A and Node B. Node A will run both of last named release of ironic-api and ironic-conductor and node B will run last named release ironic-conductor only. Multi-node grenade CI will only upgrade Node A and then we can test the interaction between new SHA of ironic-api and new SHA/last named release of ironic-conductor still works. This should also apply to stable branch.
That's a test case for "live upgrade", +1 :) Mathieu
B.R Tan [1] https://review.openstack.org/299245 From: Devananda van der Veen [mailto:devananda....@gmail.com] Sent: Wednesday, April 20, 2016 5:12 AM To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [ironic] upgrade support between which versions of ironic? Thanks for starting the thread, Ruby. We need to first establish a grenade job to test "cold upgrades" and assert the supports-upgrade tag. I believe Ironic meets all the criteria for that tag except: - having a job that tests it (so, you know, it might be broken and I might be wrong) - having operator documentation describing the process (it should be here: http://docs.openstack.org/developer/ironic/deploy/upgrade-guide.html ) but all we have are release-specific upgrade notes. I think all of the scenario you outline are valid upgrade paths for an operator, and we should try to allow all of them to work. However, some of them can be covered by one test case, and you also missed some things I think need to be covered. Also, I'm interpreting the word "master" in your scenarios to indicate the proposed change to our master branch, since we do pre-merge testing.... So, here are the test cases I think we need to cover: === run on proposed changes to master === F. current master to new SHA We need to ensure that we can upgrade master to the code being proposed. You listed this last, but I think it's actually the most important one. D. last named release to new SHA E. last numbered release to new SHA Because we cut new releases from master, this is the basis of testing the upgrade between sequential (named or numbered) releases before we cut a new (named or numbered) release, and is our most important test to ensure that we don't break most operators. Based on the user survey, most operators are using named releases, so if we are resource constrained, I would prefer to cover (D) before (E) === run on proposed changes to a stable branch === A. stable/N-1 -> new SHA -> [ stable/N+1 or current master] We don't need to test upgrades between two named releases (eg. Liberty -> Mitaka) every time we land a new patch on the master branch, but we do need to test any time we land a change on a stable branch. Changes to the most recent stable branch should be upgrade-tested to current master, whereas changes to any stable branch prior to that should get tested to the subsequent sequential release. Eg, a backport to stable/liberty should trigger an upgrade test for both (stable/kilo -> newpatch) and (newpatch -> stable/mitaka), whereas a backport to stable/mitaka should trigger a test for (stable/liberty -> newpatch) and (newpatch -> master) Once we've done that, then yes, I agree we should also work towards asserting supports-rolling-upgrade. That will require a partial upgrade job, eg. where we run >1 instance of the API and Conductor services and upgrade some of them. Regards, Devananda On Tue, Apr 19, 2016 at 12:52 PM, Ruby Loo <opensr...@gmail.com<mailto:opensr...@gmail.com>> wrote: Hi, Currently, ironic doesn't support ("live", "online", "rolling", or minimal-downtime) upgrades between named versions of ironic. (Where "named version" is the final release or stable release that is associated with a development cycle). So for example, Liberty -> Mitaka release. We've been working towards that, and have a spec [1] and a design session [2] at the Austin Summit. Upon reading the spec, I started to wonder -- what do we mean/want, when we talk about supporting upgrades. Do we want to support: A. sequential named releases, eg. Mitaka -> Newton B. sequential major releases, eg 4.x -> 5.0; 5.0 -> 6.1 C. sequential minor releases, eg 5.1 -> 5.2 D. last named release to master E. last release (major or minor) to master F. some-SHA-more-recent-than-last-named to master. This could be some numbered (major or minor) release. Keep in mind that ironic may release any number of numbered releases between two named releases, so eg. 4.3.0, 5.0.0, 5.1.0 == Mitaka, 5.2.0, 6.0.0, 6.1.0, 7.0.0, 7.1.0, 7.2.0 == Newton. Note that there are two governance tags: supports-upgrade[3] and supports-rolling-upgrade[4], and I believe our goal is for supports-rolling-upgrade. I think and hope that we can all agree that A. is a must :) What constitutes a major release versus a minor release may have implications in this, but that should be in a separate discussion. What do people think? --ruby [1] https://review.openstack.org/299245 [2] https://www.openstack.org/summit/austin-2016/summit-schedule/events/9267 [3] https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-upgrade.rst [4] https://github.com/openstack/governance/blob/master/reference/tags/assert_supports-rolling-upgrade.rst __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
