Dmitry, Aleksandra, thank you for help and support! Regards, Igor Marnat
On Fri, Mar 4, 2016 at 1:20 AM, Aleksandra Fedorova <afedor...@mirantis.com> wrote: > As we agreed, we have switched ISO builds to latest CentOS 7.2 snapshots. > > You can see now that each ISO build (see for ex. [1]) produces several > *_id.txt artifacts. > Note that centos_mirror_id.txt points to CentOS snapshot at > http://mirror.fuel-infra.org/pkgs/ > > BVT test is stable, see [2], and nightly system tests results are > basically the same as they were before. > > > [1] https://ci.fuel-infra.org/job/9.0-community.all/3868/ > [2] > https://ci.fuel-infra.org/view/ISO/job/9.0.fuel_community.ubuntu.bvt_2/ > > On Thu, Mar 3, 2016 at 3:46 AM, Dmitry Borodaenko > <dborodae...@mirantis.com> wrote: > > Thanks for the detailed explanation, very helpful! > > > > Considering that this change is atomic and easily revertable, lets > > proceed with the change, the sooner we do that the more time we'll have > > to confirm that there is no impact and revert if necessary. > > > > -- > > Dmitry Borodaenko > > > > On Thu, Mar 03, 2016 at 03:40:22AM +0300, Aleksandra Fedorova wrote: > >> Hi, > >> > >> let me add some details about the change: > >> > >> 1) There are two repositories used to build Fuel ISO: base OS > >> repository [1], and mos repository [2], where we put Fuel dependencies > >> and packages we rebuild due to certain version requirements. > >> > >> The CentOS 7.2 feature is related to the upstream repo only. Packages > >> like RabbitMQ, MCollective, Puppet, MySQL and PostgreSQL live in mos > >> repository, which has higher priority then upstream. > >> > >> I think we need to setup a separate discussion about our policy > >> regarding these packages, but for now they are fixed and won't be > >> updated by CentOS 7.2 switch. > >> > >> 2) This change doesn't affect Fuel codebase. > >> > >> The upstream mirror we use for ISO build is controlled by environment > >> variable which is set via Jenkins [3] and can be changed anytime. > >> > >> As we have daily snapshots of CentOS repository available at [4], in > >> case of regression in upstream we can pin our builds to stable > >> snapshot and work on the issue without blocking the main development > >> flow. > > > > Please make sure that the current snapshot of CentOS 7.1 is not rotated > > away so that we don't loose the point we can revert to. > > > >> 3) The "improve snapshotting" work item which is at the moment in > >> progress, will prevent any possibility to "accidentally" migrate to > >> CentOS 7.3, when it becomes available. > >> Thus the only changes which we can fetch from upstream are changes > >> which are published to updates/ component of CentOS 7.2 repo. > >> > >> > >> As latest BVT on master is green > >> https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/69/ > >> I think we should proceed with Jenkins reconfiguration [5] and switch > >> to latest snapshots by default. > >> > >> [1] currently http://vault.centos.org/7.1.1503/ > >> [2] > http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7-fuel/os/x86_64/ > >> [3] > https://github.com/fuel-infra/jenkins-jobs/blob/76b5cdf1828b7db1957f7967180d20be099b0c63/common/scripts/all.sh#L84 > >> [4] http://mirror.fuel-infra.org/pkgs/ > >> [5] https://review.fuel-infra.org/#/c/17712/ > >> > >> On Wed, Mar 2, 2016 at 9:22 PM, Mike Scherbakov > >> <mscherba...@mirantis.com> wrote: > >> > It is not just about BVT. I'd suggest to monitor situation overall, > >> > including failures of system tests [1]. If we see regressions there, > or some > >> > test cases will start flapping (what is even worse), then we'd have to > >> > revert back to CentOS 7.1. > >> > > >> > [1] https://github.com/openstack/fuel-qa > >> > > >> > On Wed, Mar 2, 2016 at 10:16 AM Dmitry Borodaenko < > dborodae...@mirantis.com> > >> > wrote: > >> >> > >> >> I agree with Mike's concerns, and propose to make these limitations > (4 > >> >> weeks before FF for OS upgrades, 2 weeks for upgrades of key > >> >> dependencies -- RabbitMQ, MCollective, Puppet, MySQL, PostgreSQL, > >> >> anything else?) official for 10.0/Newton. > >> >> > >> >> For 9.0/Mitaka, it is too late to impose them, so we just have to be > >> >> very careful and conservative with this upgrade. First of all, we > need > >> >> to have a green BVT before and after switching to the CentOS 7.2 repo > >> >> snapshot, so while I approved the spec, we can't move forward with > this > >> >> until BVT is green again, and right now it's red: > >> >> > >> >> https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/ > >> >> > >> >> If we get it back to green but it becomes red after the upgrade, you > >> >> must switch back to CentOS 7.1 *immediately*. If you are able to > stick > >> >> to this plan, there is still time to complete the transition today > >> >> without requiring an FFE. > >> >> > >> >> -- > >> >> Dmitry Borodaenko > >> >> > >> >> > >> >> On Wed, Mar 02, 2016 at 05:53:53PM +0000, Mike Scherbakov wrote: > >> >> > Formally, we can merge it today. Historically, every update of OS > caused > >> >> > us > >> >> > instability for some time: from days to a couple of month. > >> >> > Taking this into account and number of other exceptions requested, > >> >> > overall > >> >> > stability of code, my opinion would be to postpone this to 10.0. > >> >> > > >> >> > Also, I'd suggest to change the process, and have freeze date for > all OS > >> >> > updates no later than a month before official FF date. This will > give us > >> >> > time to stabilize, and ensure that base on which all new code is > being > >> >> > developed is stable when approaching FF. > >> >> > > >> >> > I'd also propose to have freeze for major upgrades of 3rd party > packages > >> >> > no > >> >> > later than 2 weeks before FF, which Fuel depends heavily upon. For > >> >> > instance, such will include RabbitMQ, MCollective, Puppet. > >> >> > > >> >> > On Wed, Mar 2, 2016 at 7:34 AM Igor Marnat <imar...@mirantis.com> > wrote: > >> >> > > >> >> > > Igor, > >> >> > > couple of points from my side. > >> >> > > > >> >> > > CentOS 7.2 will be getting updates for several more months, and > we > >> >> > > have > >> >> > > snapshots and all the mechanics in place to switch to the next > version > >> >> > > when > >> >> > > needed. > >> >> > > > >> >> > > Speaking of getting this update into 9.0, we actually don't need > FFE, > >> >> > > we > >> >> > > can merge remaining staff today. It has enough reviews, so if > you add > >> >> > > your > >> >> > > +1 today, we don't need FFE. > >> >> > > > >> >> > > https://review.openstack.org/#/c/280338/ > >> >> > > https://review.fuel-infra.org/#/c/17400/ > >> >> > > > >> >> > > > >> >> > > > >> >> > > Regards, > >> >> > > Igor Marnat > >> >> > > > >> >> > > On Wed, Mar 2, 2016 at 6:23 PM, Dmitry Teselkin > >> >> > > <dtesel...@mirantis.com> > >> >> > > wrote: > >> >> > > > >> >> > >> Igor, > >> >> > >> > >> >> > >> Your statement about updates for 7.2 isn't correct - it will > receive > >> >> > >> updates, because it's the latest release ATM. There is *no* > pinning > >> >> > >> inside > >> >> > >> ISO, and the only place where it was 8.0 were docker containers > just > >> >> > >> because we had to workaround some issues. But there are no > docker > >> >> > >> containers in 9.0, so that's not the case. > >> >> > >> The proposed solution to switch to CentOS-7.2 in fact is based > on > >> >> > >> selecting the right snapshot with packages. There is no pinning > in > >> >> > >> ISO (it > >> >> > >> was in earlier versions of the spec but was removed). > >> >> > >> > >> >> > >> On Wed, Mar 2, 2016 at 6:11 PM, Igor Kalnitsky > >> >> > >> <ikalnit...@mirantis.com> > >> >> > >> wrote: > >> >> > >> > >> >> > >>> Dmitry, Igor, > >> >> > >>> > >> >> > >>> > Very important thing is that CentOS 7.1 which master node is > based > >> >> > >>> > now > >> >> > >>> > don't get updates any longer. > >> >> > >>> > >> >> > >>> If you are using "fixed" release you must be ready that you > won't > >> >> > >>> get > >> >> > >>> any updates. So with CentOS 7.2 the problem still the same. > >> >> > >>> > >> >> > >>> However, let's wait for Fuel PTL decision. I only shared my > POV: > >> >> > >>> that's not a critical feature, and taking into account the > risks of > >> >> > >>> regression - I'd prefer to do not accept it in 9.0. > >> >> > >>> > >> >> > >>> Regards, > >> >> > >>> Igor > >> >> > >>> > >> >> > >>> On Wed, Mar 2, 2016 at 4:42 PM, Igor Marnat < > imar...@mirantis.com> > >> >> > >>> wrote: > >> >> > >>> > Igor, > >> >> > >>> > please note that this is pretty much not like update of > master > >> >> > >>> > node > >> >> > >>> which we > >> >> > >>> > had in 8.0. This is minor _update_ of CentOS from 7.1 to 7.2 > which > >> >> > >>> > team > >> >> > >>> > tested for more than 2 months already. > >> >> > >>> > > >> >> > >>> > We don't expect it to require any additional efforts from > core or > >> >> > >>> > qa > >> >> > >>> team. > >> >> > >>> > > >> >> > >>> > Very important thing is that CentOS 7.1 which master node is > based > >> >> > >>> > now > >> >> > >>> don't > >> >> > >>> > get updates any longer. Updates are only provided for CentOS > 7.2. > >> >> > >>> > > >> >> > >>> > So we'll have to switch CentOS 7.1 to CentOS 7.2 anyways. > >> >> > >>> > > >> >> > >>> > We can do it now for more or less free, later in release > cycle for > >> >> > >>> higher > >> >> > >>> > risk and QA efforts and after the release for 2x price > because of > >> >> > >>> additional > >> >> > >>> > QA cycle we'll need to pass through. > >> >> > >>> > > >> >> > >>> > > >> >> > >>> > > >> >> > >>> > Regards, > >> >> > >>> > Igor Marnat > >> >> > >>> > > >> >> > >>> > On Wed, Mar 2, 2016 at 2:57 PM, Dmitry Teselkin < > >> >> > >>> dtesel...@mirantis.com> > >> >> > >>> > wrote: > >> >> > >>> >> > >> >> > >>> >> Hi Igor, > >> >> > >>> >> > >> >> > >>> >> Postponing this till Fuel 10 means we have to elaborate a > plan to > >> >> > >>> >> do > >> >> > >>> such > >> >> > >>> >> upgrade for Fuel 9 after the release - the underlying > system will > >> >> > >>> >> not > >> >> > >>> get > >> >> > >>> >> updated on it's own, and the security issues will not close > >> >> > >>> themselves. The > >> >> > >>> >> problem here is that such upgrade of deployed master node > >> >> > >>> >> requires a > >> >> > >>> lot of > >> >> > >>> >> QA work also. > >> >> > >>> >> > >> >> > >>> >> Since we are not going to update package we build on our own > >> >> > >>> >> (they > >> >> > >>> still > >> >> > >>> >> targeted 7.1) switching master node base to CentOS-72 is > not that > >> >> > >>> dangerous > >> >> > >>> >> as it seems. > >> >> > >>> >> > >> >> > >>> >> On Wed, Mar 2, 2016 at 1:54 PM, Igor Kalnitsky < > >> >> > >>> ikalnit...@mirantis.com> > >> >> > >>> >> wrote: > >> >> > >>> >>> > >> >> > >>> >>> Hey Dmitry, > >> >> > >>> >>> > >> >> > >>> >>> No offence, but I rather against that exception. We have > too > >> >> > >>> >>> many > >> >> > >>> >>> things to do in Mitaka, and moving to CentOS 7.2 means > >> >> > >>> >>> > >> >> > >>> >>> * extra effort from core team > >> >> > >>> >>> * extra effort from qa team > >> >> > >>> >>> > >> >> > >>> >>> Moreover, it might block development by introducing > >> >> > >>> >>> unpredictable > >> >> > >>> >>> regressions. Remember 8.0? So I think it'd be better to > reduce > >> >> > >>> >>> risk > >> >> > >>> of > >> >> > >>> >>> regressions that affects so many developers by postponing > CentOS > >> >> > >>> >>> 7.2 > >> >> > >>> >>> till Fuel 10. > >> >> > >>> >>> > >> >> > >>> >>> Thanks, > >> >> > >>> >>> Igor > >> >> > >>> >>> > >> >> > >>> >>> > >> >> > >>> >>> On Mon, Feb 29, 2016 at 7:13 PM, Dmitry Teselkin < > >> >> > >>> dtesel...@mirantis.com> > >> >> > >>> >>> wrote: > >> >> > >>> >>> > I'd like to ask for a feature freeze exception for > switching > >> >> > >>> >>> > to > >> >> > >>> >>> > CentOS-7.2 > >> >> > >>> >>> > feature [0]. > >> >> > >>> >>> > > >> >> > >>> >>> > CentOS-7.2 ISO's have been tested periodically since the > >> >> > >>> >>> > beginning > >> >> > >>> of > >> >> > >>> >>> > the > >> >> > >>> >>> > year, and all major issues were addressed / fixed at the > >> >> > >>> >>> > moment. > >> >> > >>> During > >> >> > >>> >>> > the > >> >> > >>> >>> > last weekend I've made 70 BVT runs to verify that the > >> >> > >>> >>> > solution > >> >> > >>> [2] for > >> >> > >>> >>> > the > >> >> > >>> >>> > last issue - e1000 transmit unit hangs works. And it > works, 0 > >> >> > >>> tests of > >> >> > >>> >>> > 35 > >> >> > >>> >>> > failed [3]. > >> >> > >>> >>> > > >> >> > >>> >>> > Benefits of switching to CentOS-7.2 are quite obvious - > we > >> >> > >>> >>> > will > >> >> > >>> return > >> >> > >>> >>> > to > >> >> > >>> >>> > latest supported CentOS release, will fix a lot of bugs / > >> >> > >>> >>> > security > >> >> > >>> >>> > issues > >> >> > >>> >>> > [4] and will make further updates easier. > >> >> > >>> >>> > > >> >> > >>> >>> > Risk of regression still exists, but it's quite low, 35 > >> >> > >>> >>> > successful > >> >> > >>> BVTs > >> >> > >>> >>> > can't be wrong. > >> >> > >>> >>> > > >> >> > >>> >>> > To finish that feature the following should be done: > >> >> > >>> >>> > * review and merge e1000 workaround [2] > >> >> > >>> >>> > * review and merge spec [0] > >> >> > >>> >>> > * review and merge request that switches build CI to > >> >> > >>> >>> > CentOS-7.2 [5] > >> >> > >>> >>> > > >> >> > >>> >>> > I expect the last day it could be done is March, 4. > >> >> > >>> >>> > > >> >> > >>> >>> > [0] https://review.openstack.org/#/c/280338/ > >> >> > >>> >>> > [1] https://bugs.launchpad.net/fuel/+bug/1526544 > >> >> > >>> >>> > [2] https://review.openstack.org/#/c/285306/ > >> >> > >>> >>> > [3] > >> >> > >>> > https://etherpad.openstack.org/p/r.1c4cfee8185326d6922d6c9321404350 > >> >> > >>> >>> > [4] > >> >> > >>> > https://etherpad.openstack.org/p/r.a7fe0b575d891ed81206765fa5be6630 > >> >> > >>> >>> > [5] https://review.fuel-infra.org/#/c/17400/ > >> >> > >>> >>> > > >> >> > >>> >>> > > >> >> > >>> >>> > -- > >> >> > >>> >>> > Thanks, > >> >> > >>> >>> > Dmitry Teselkin > >> >> > >>> >>> > Mirantis > >> >> > >>> >>> > http://www.mirantis.com > >> >> > >>> >>> > > >> >> > >>> >>> > > >> >> > >>> >>> > > >> >> > >>> > >> >> > >>> > __________________________________________________________________________ > >> >> > >>> >>> > OpenStack Development Mailing List (not for usage > questions) > >> >> > >>> >>> > Unsubscribe: > >> >> > >>> >>> > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> >> > >>> >>> > > >> >> > >>> >>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> >> > >>> >>> > > >> >> > >>> >>> > >> >> > >>> >>> > >> >> > >>> >>> > >> >> > >>> > >> >> > >>> > __________________________________________________________________________ > >> >> > >>> >>> OpenStack Development Mailing List (not for usage > questions) > >> >> > >>> >>> Unsubscribe: > >> >> > >>> >>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> >> > >>> >>> > >> >> > >>> >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> >> > >>> >> > >> >> > >>> >> > >> >> > >>> >> > >> >> > >>> >> > >> >> > >>> >> -- > >> >> > >>> >> Thanks, > >> >> > >>> >> Dmitry Teselkin > >> >> > >>> >> Mirantis > >> >> > >>> >> http://www.mirantis.com > >> >> > >>> >> > >> >> > >>> >> > >> >> > >>> > >> >> > >>> > __________________________________________________________________________ > >> >> > >>> >> OpenStack Development Mailing List (not for usage questions) > >> >> > >>> >> Unsubscribe: > >> >> > >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> >> > >>> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> >> > >>> >> > >> >> > >>> > > >> >> > >>> > > >> >> > >>> > > >> >> > >>> > >> >> > >>> > __________________________________________________________________________ > >> >> > >>> > OpenStack Development Mailing List (not for usage questions) > >> >> > >>> > Unsubscribe: > >> >> > >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> >> > >>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> >> > >>> > > >> >> > >>> > >> >> > >>> > >> >> > >>> > >> >> > >>> > __________________________________________________________________________ > >> >> > >>> OpenStack Development Mailing List (not for usage questions) > >> >> > >>> Unsubscribe: > >> >> > >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> >> > >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> >> > >>> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> > >> -- > >> >> > >> Thanks, > >> >> > >> Dmitry Teselkin > >> >> > >> Mirantis > >> >> > >> http://www.mirantis.com > >> >> > >> > >> >> > >> > >> >> > >> > __________________________________________________________________________ > >> >> > >> OpenStack Development Mailing List (not for usage questions) > >> >> > >> Unsubscribe: > >> >> > >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> >> > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> >> > >> > >> >> > >> > >> >> > > > >> >> > > > __________________________________________________________________________ > >> >> > > OpenStack Development Mailing List (not for usage questions) > >> >> > > Unsubscribe: > >> >> > > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> >> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> >> > > > >> >> > -- > >> >> > Mike Scherbakov > >> >> > #mihgen > >> >> > >> >> > > >> >> > > __________________________________________________________________________ > >> >> > OpenStack Development Mailing List (not for usage questions) > >> >> > Unsubscribe: > >> >> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> >> > >> >> > >> >> > __________________________________________________________________________ > >> >> OpenStack Development Mailing List (not for usage questions) > >> >> Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > -- > >> > Mike Scherbakov > >> > #mihgen > >> > > >> > > __________________________________________________________________________ > >> > OpenStack Development Mailing List (not for usage questions) > >> > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > >> > >> > >> > >> -- > >> Aleksandra Fedorova > >> CI Team Lead > >> bookwar > >> > >> > __________________________________________________________________________ > >> OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > -- > Aleksandra Fedorova > CI Team Lead > bookwar > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev