What would be your suggestions then on the issue? We want to avoid breaking of our fuel-library and other code without knowing about it, so it's not an option to simply forget about compatibility with Icehouse packages.
And I'm actually +1 for introducing Kilo CI jobs as soon as we can, so to keep our CI cycle very close to the upstream. On Thu, Sep 11, 2014 at 11:22 PM, Roman Vyalov <rvya...@mirantis.com> wrote: > Mike, > 2 jobs for Icehouse and Juno equal 2 different repository with packages > for Fuel 6.0. This can be problem for current osci workflow. > For example: We need building new packages. Which repository we must put > packages? to icehouse or/and Juno ? > if new packages will break icehouse repository, but required for Juno ... > > On Wed, Sep 10, 2014 at 12:39 AM, Mike Scherbakov < > mscherba...@mirantis.com> wrote: > >> Aleksandra, >> you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but "4 >> fuel-library tests" should happen right after we create stable/5.1. Also, >> for Fuel CI for OSTF - I don't think it's actually necessary to support >> <5.0 envs. >> >> Your questions: >> >> 1. Create jobs for both Icehouse and Juno, but it doesn't make sense >> to do staging for Juno till it starts to pass deployment in HA mode. Once >> it passes deployment in HA, staging should be enabled. Then, once it >> passes >> OSTF - we extend criteria, and pass only those mirrors which also pass >> OSTF >> phase >> 2. Once Juno starts to pass BVT with OSTF check enabled, I think we >> can disable Icehouse checks. Not sure about fuel-library tests on Fuel CI >> with Icehouse - we might want to continue using them. >> >> Thanks, >> >> On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova < >> afedor...@mirantis.com> wrote: >> >>> > Our Fuel CI can do 4 builds against puppet modules: 2 voting, with >>> Icehouse packages; 2 non-voting, with Juno packages. >>> > Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) >>> actually before Juno becomes stable. We will be able to run 2 sets of BVTs >>> (against Icehouse and Juno), and it means that we will be able to see >>> almost immediately if something in nailgun/astute/puppet integration broke. >>> For Juno builds it's going to be all red initially. >>> >>> Let me rephrase: >>> >>> We keep one Fuel master branch for two OpenStack releases. And we make >>> sure that Fuel master code is compatible with both of them. And we use >>> current release (Icehouse) as a reference for test results of upcoming >>> release, till we obtain stable enough reference point in Juno itself. >>> Moreover we'd like to have OSTF code running on all previous Fuel releases. >>> >>> Changes to CI workflow look as follows: >>> >>> Nightly builds: >>> 1) We build two mirrors: one for Icehouse and one for Juno. >>> 2) From each mirror we build Fuel ISO using exactly the same fuel >>> master branch code. >>> 3) Then we run BVT tests on both (using the same fuel-main code for >>> system tests). >>> 4) If Icehouse BVT tests pass, we deploy both ISO images (even with >>> failed Juno tests) onto Fuel CI. >>> >>> On Fuel CI we should run: >>> - 4 fuel-library tests (revert master node, inject fuel-library code >>> in master node and run deployment): >>> 2 (ubuntu and centos) voting Icehouse tests and 2 non-voting >>> Juno tests >>> - 5 OSTF tests (revert deployed environment, inject OSTF code into >>> master node, run OSTF): >>> voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on >>> master/Juno >>> - other tests, which don't use prebuilt environment, work as before >>> >>> The major action point here would be OSTF tests, as we don't have yet >>> working implementation of injecting OSTF code into deployed environment. >>> And we don't run any tests on old environments. >>> >>> >>> Questions: >>> >>> 1) How should we test mirrors? >>> >>> Current master mirrors go through the 4 hours test cycle involving Fuel >>> ISO build: >>> 1. we build temporary mirror >>> 2. build custom iso from it >>> 3. run two custom bvt jobs >>> 4. if they pass we move mirror to stable and sitch to it for our >>> "primary" fuel_master_iso >>> >>> Should we test only Icehouse mirrors, or both, but ignoring again failed >>> BVT for Juno? Maybe we should enable these tests only later in release >>> cycle, say, after SCF? >>> >>> 2) It is not clear for me when and how we will switch from supporting >>> two releases back to one. >>> Should we add one more milestone to our release process? The "Switching >>> point", when we disable and remove Icehouse tasks and move to Juno >>> completely? I guess it should happen before next SCF? >>> >>> >>> >>> On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov < >>> mscherba...@mirantis.com> wrote: >>> >>>> > What we need to achieve that is have 2 build series based on Fuel >>>> master: one with Icehouse packages, and one with Juno, and, as Mike >>>> proposed, keep our manifests backwards compatible with Icehouse. >>>> Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting, >>>> with Icehouse packages; 2 non-voting, with Juno packages. >>>> >>>> Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) >>>> actually before Juno becomes stable. We will be able to run 2 sets of BVTs >>>> (against Icehouse and Juno), and it means that we will be able to see >>>> almost immediately if something in nailgun/astute/puppet integration broke. >>>> For Juno builds it's going to be all red initially. >>>> >>>> Another suggestion would be to lower green switch in BVTs for Juno: >>>> first, when it passes deployment; and then, if it finally passes OSTF. >>>> >>>> I'd like to hear QA & DevOps opinion on all the above. Immediately we >>>> would need just standard stuff which is in checklists for OSCI & DevOps >>>> teams, and ideally soon after that - ability to have Fuel CI running 4 >>>> builds, not 2, against our master, as mentioned above. >>>> >>>> On Tue, Sep 9, 2014 at 9:28 PM, Roman Vyalov <rvya...@mirantis.com> >>>> wrote: >>>> >>>>> All OSCI action items for prepare HCF check list has been done >>>>> >>>>> >>>>> On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov < >>>>> mscherba...@mirantis.com> wrote: >>>>> >>>>>> Thanks Alexandra. >>>>>> >>>>>> We land a few patches a day currently, so I think we can open stable >>>>>> branch. If we see no serious objections in next 12 hours, let's do it. We >>>>>> would need to immediately notify everyone in mailing list - that for >>>>>> every >>>>>> patch for 5.1, it should go first to master, and then to stable/5.1. >>>>>> >>>>>> Is everything ready from DevOps, OSCI (packaging) side to do this? >>>>>> Fuel CI, OBS, etc.? >>>>>> >>>>>> On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova < >>>>>> afedor...@mirantis.com> wrote: >>>>>> >>>>>>> As I understand your proposal, we need to split our HCF milestone >>>>>>> into two check points: Branching Point and HCF itself. >>>>>>> >>>>>>> Branching point should happen somewhere in between SCF and HCF. And >>>>>>> though It may coincide with HCF, it needs its own list of requirements. >>>>>>> This will give us the possibility to untie two events and make a >>>>>>> separate >>>>>>> decision on branching without enforcing all HCF criteria. >>>>>>> >>>>>>> From the DevOps point of view it changes almost nothing, it just >>>>>>> adds a bit more discussion items on the management side and slight >>>>>>> modifications to our checklists. >>>>>>> >>>>>>> >>>>>>> On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko < >>>>>>> dborodae...@mirantis.com> wrote: >>>>>>> >>>>>>>> TL;DR: Yes, our work on 6.0 features is currently blocked and it is >>>>>>>> becoming a major problem. No, I don't think we should create >>>>>>>> pre-release or feature branches. Instead, we should create >>>>>>>> stable/5.1 >>>>>>>> branches and open master for 6.0 work. >>>>>>>> >>>>>>>> We have reached a point in 5.1 release cycle where the scope of >>>>>>>> issues >>>>>>>> we are willing to address in this release is narrow enough to not >>>>>>>> require full attention of the whole team. We have engineers working >>>>>>>> on >>>>>>>> 6.0 features, and their work is essentially blocked until they have >>>>>>>> somewhere to commit their changes. >>>>>>>> >>>>>>>> Simply creating new branches is not even close to solving this >>>>>>>> problem: we have a whole CI infrastructure around every active >>>>>>>> release >>>>>>>> series (currently 5.1, 5.0, 4.1), including test jobs for gerrit >>>>>>>> commits, package repository mirrors updates, ISO image builds, >>>>>>>> smoke, >>>>>>>> build verification, and swarm tests for ISO images, documentation >>>>>>>> builds, etc. A branch without all that infrastructure isn't any >>>>>>>> better >>>>>>>> than current status quo: every developer tracking their own 6.0 work >>>>>>>> locally. >>>>>>>> >>>>>>>> Unrelated to all that, we also had a lot of very negative experience >>>>>>>> with feature branches in the past [0] [1], which is why we have >>>>>>>> decided to follow the OpenStack branching strategy: commit all >>>>>>>> feature >>>>>>>> changes directly to master and track bugfixes for stable releases in >>>>>>>> stable/* branches. >>>>>>>> >>>>>>>> [0] https://lists.launchpad.net/fuel-dev/msg00127.html >>>>>>>> [1] https://lists.launchpad.net/fuel-dev/msg00028.html >>>>>>>> >>>>>>>> I'm also against declaring a "hard code freeze with exceptions", HCF >>>>>>>> should remain tied to our ability to declare a release candidate. If >>>>>>>> we can't release with the bugs we already know about, declaring HCF >>>>>>>> before fixing these bugs would be an empty gesture. >>>>>>>> >>>>>>>> Creating stable/5.1 now instead of waiting for hard code freeze for >>>>>>>> 5.1 will cost us two things: >>>>>>>> >>>>>>>> 1) DevOps team will have to update our CI infrastructure for one >>>>>>>> more >>>>>>>> release series. It's something we have to do for 6.0 sooner or >>>>>>>> later, >>>>>>>> so this may be a disruption, but not an additional effort. >>>>>>>> >>>>>>>> 2) All commits targeted for 5.1 will have to be proposed for two >>>>>>>> branches (master and stable/5.1) instead of just one (master). This >>>>>>>> will require additional effort, but I think that it is significantly >>>>>>>> smaller than the cost of spinning our wheels on 6.0 efforts. >>>>>>>> >>>>>>>> -DmitryB >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov >>>>>>>> <dmescherya...@mirantis.com> wrote: >>>>>>>> > Hello Fuelers, >>>>>>>> > >>>>>>>> > Right now we have the following policy in place: the branches for >>>>>>>> a >>>>>>>> > release are opened only after its 'parent' release have reached >>>>>>>> hard >>>>>>>> > code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 >>>>>>>> and >>>>>>>> > 6.0. >>>>>>>> > >>>>>>>> > And that is the problem: if parent release is delayed, we can't >>>>>>>> > properly start development of a child release because we don't >>>>>>>> have >>>>>>>> > branches to commit. That is current issue with 6.0: we already >>>>>>>> started >>>>>>>> > to work on pushing Juno in to 6.0, but if we are to make changes >>>>>>>> to >>>>>>>> > our deployment code we have nowhere to store them. >>>>>>>> > >>>>>>>> > IMHO the issue could easily be resolved by creation of pre-release >>>>>>>> > branches, which are merged together with parent branches once the >>>>>>>> > parent reaches HCF. Say, we use branch 'pre-6.0' for initial >>>>>>>> > development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into >>>>>>>> master >>>>>>>> > and continue development here. After that pre-6.0 is abandoned. >>>>>>>> > >>>>>>>> > What do you think? >>>>>>>> > >>>>>>>> > Thanks, >>>>>>>> > >>>>>>>> > Dmitry >>>>>>>> > >>>>>>>> > _______________________________________________ >>>>>>>> > OpenStack-dev mailing list >>>>>>>> > OpenStack-dev@lists.openstack.org >>>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Dmitry Borodaenko >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-dev mailing list >>>>>>>> OpenStack-dev@lists.openstack.org >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Aleksandra Fedorova >>>>>>> bookwar >>>>>>> >>>>>>> _______________________________________________ >>>>>>> OpenStack-dev mailing list >>>>>>> OpenStack-dev@lists.openstack.org >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Mike Scherbakov >>>>>> #mihgen >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> OpenStack-dev mailing list >>>>>> OpenStack-dev@lists.openstack.org >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> OpenStack-dev mailing list >>>>> OpenStack-dev@lists.openstack.org >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Mike Scherbakov >>>> #mihgen >>>> >>>> >>>> _______________________________________________ >>>> OpenStack-dev mailing list >>>> OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>> >>> >>> -- >>> Aleksandra Fedorova >>> bookwar >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> -- >> Mike Scherbakov >> #mihgen >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Mike Scherbakov #mihgen
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
