Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
Mike, 2 jobs for Icehouse and Juno equal 2 different repository with packages for Fuel 6.0. This can be problem for current osci workflow. For example: We need building new packages. Which repository we must put packages? to icehouse or/and Juno ? if new packages will break icehouse repository, but required for Juno ... On Wed, Sep 10, 2014 at 12:39 AM, Mike Scherbakov mscherba...@mirantis.com wrote: Aleksandra, you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but 4 fuel-library tests should happen right after we create stable/5.1. Also, for Fuel CI for OSTF - I don't think it's actually necessary to support 5.0 envs. Your questions: 1. Create jobs for both Icehouse and Juno, but it doesn't make sense to do staging for Juno till it starts to pass deployment in HA mode. Once it passes deployment in HA, staging should be enabled. Then, once it passes OSTF - we extend criteria, and pass only those mirrors which also pass OSTF phase 2. Once Juno starts to pass BVT with OSTF check enabled, I think we can disable Icehouse checks. Not sure about fuel-library tests on Fuel CI with Icehouse - we might want to continue using them. Thanks, On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova afedor...@mirantis.com wrote: Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Let me rephrase: We keep one Fuel master branch for two OpenStack releases. And we make sure that Fuel master code is compatible with both of them. And we use current release (Icehouse) as a reference for test results of upcoming release, till we obtain stable enough reference point in Juno itself. Moreover we'd like to have OSTF code running on all previous Fuel releases. Changes to CI workflow look as follows: Nightly builds: 1) We build two mirrors: one for Icehouse and one for Juno. 2) From each mirror we build Fuel ISO using exactly the same fuel master branch code. 3) Then we run BVT tests on both (using the same fuel-main code for system tests). 4) If Icehouse BVT tests pass, we deploy both ISO images (even with failed Juno tests) onto Fuel CI. On Fuel CI we should run: - 4 fuel-library tests (revert master node, inject fuel-library code in master node and run deployment): 2 (ubuntu and centos) voting Icehouse tests and 2 non-voting Juno tests - 5 OSTF tests (revert deployed environment, inject OSTF code into master node, run OSTF): voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on master/Juno - other tests, which don't use prebuilt environment, work as before The major action point here would be OSTF tests, as we don't have yet working implementation of injecting OSTF code into deployed environment. And we don't run any tests on old environments. Questions: 1) How should we test mirrors? Current master mirrors go through the 4 hours test cycle involving Fuel ISO build: 1. we build temporary mirror 2. build custom iso from it 3. run two custom bvt jobs 4. if they pass we move mirror to stable and sitch to it for our primary fuel_master_iso Should we test only Icehouse mirrors, or both, but ignoring again failed BVT for Juno? Maybe we should enable these tests only later in release cycle, say, after SCF? 2) It is not clear for me when and how we will switch from supporting two releases back to one. Should we add one more milestone to our release process? The Switching point, when we disable and remove Icehouse tasks and move to Juno completely? I guess it should happen before next SCF? On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov mscherba...@mirantis.com wrote: What we need to achieve that is have 2 build series based on Fuel master: one with Icehouse packages, and one with Juno, and, as Mike proposed, keep our manifests backwards compatible with Icehouse. Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Another suggestion would be to lower green switch in BVTs for Juno: first, when it passes deployment; and then, if it finally passes OSTF. I'd like to hear QA DevOps opinion on all the above. Immediately we would need
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
What would be your suggestions then on the issue? We want to avoid breaking of our fuel-library and other code without knowing about it, so it's not an option to simply forget about compatibility with Icehouse packages. And I'm actually +1 for introducing Kilo CI jobs as soon as we can, so to keep our CI cycle very close to the upstream. On Thu, Sep 11, 2014 at 11:22 PM, Roman Vyalov rvya...@mirantis.com wrote: Mike, 2 jobs for Icehouse and Juno equal 2 different repository with packages for Fuel 6.0. This can be problem for current osci workflow. For example: We need building new packages. Which repository we must put packages? to icehouse or/and Juno ? if new packages will break icehouse repository, but required for Juno ... On Wed, Sep 10, 2014 at 12:39 AM, Mike Scherbakov mscherba...@mirantis.com wrote: Aleksandra, you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but 4 fuel-library tests should happen right after we create stable/5.1. Also, for Fuel CI for OSTF - I don't think it's actually necessary to support 5.0 envs. Your questions: 1. Create jobs for both Icehouse and Juno, but it doesn't make sense to do staging for Juno till it starts to pass deployment in HA mode. Once it passes deployment in HA, staging should be enabled. Then, once it passes OSTF - we extend criteria, and pass only those mirrors which also pass OSTF phase 2. Once Juno starts to pass BVT with OSTF check enabled, I think we can disable Icehouse checks. Not sure about fuel-library tests on Fuel CI with Icehouse - we might want to continue using them. Thanks, On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova afedor...@mirantis.com wrote: Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Let me rephrase: We keep one Fuel master branch for two OpenStack releases. And we make sure that Fuel master code is compatible with both of them. And we use current release (Icehouse) as a reference for test results of upcoming release, till we obtain stable enough reference point in Juno itself. Moreover we'd like to have OSTF code running on all previous Fuel releases. Changes to CI workflow look as follows: Nightly builds: 1) We build two mirrors: one for Icehouse and one for Juno. 2) From each mirror we build Fuel ISO using exactly the same fuel master branch code. 3) Then we run BVT tests on both (using the same fuel-main code for system tests). 4) If Icehouse BVT tests pass, we deploy both ISO images (even with failed Juno tests) onto Fuel CI. On Fuel CI we should run: - 4 fuel-library tests (revert master node, inject fuel-library code in master node and run deployment): 2 (ubuntu and centos) voting Icehouse tests and 2 non-voting Juno tests - 5 OSTF tests (revert deployed environment, inject OSTF code into master node, run OSTF): voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on master/Juno - other tests, which don't use prebuilt environment, work as before The major action point here would be OSTF tests, as we don't have yet working implementation of injecting OSTF code into deployed environment. And we don't run any tests on old environments. Questions: 1) How should we test mirrors? Current master mirrors go through the 4 hours test cycle involving Fuel ISO build: 1. we build temporary mirror 2. build custom iso from it 3. run two custom bvt jobs 4. if they pass we move mirror to stable and sitch to it for our primary fuel_master_iso Should we test only Icehouse mirrors, or both, but ignoring again failed BVT for Juno? Maybe we should enable these tests only later in release cycle, say, after SCF? 2) It is not clear for me when and how we will switch from supporting two releases back to one. Should we add one more milestone to our release process? The Switching point, when we disable and remove Icehouse tasks and move to Juno completely? I guess it should happen before next SCF? On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov mscherba...@mirantis.com wrote: What we need to achieve that is have 2 build series based on Fuel master: one with Icehouse packages, and one with Juno, and, as Mike proposed, keep our manifests backwards compatible with Icehouse. Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
Roman, In the workflow with 2 package repositories per 1 fuel branch set (e.g. icehouse juno for 6.0, or juno kilo later in 6.0 release cycle), we should aim to keep packages targeted for both repositories as close as possible, so default mode of operation should be to put new packages in both repositories. Use cases: 1. We need new packages for Juno, we don't expect them to break Icehouse. We should put packages to both repos: if there's impact on Icehouse we want to know sooner rather than later. 2. We need to add a patch to a package. If it's not a backport of bugfix from Juno, most likely the bug is present in both Icehouse and Juno, and so should be the fix. 3. We need to make a packaging change (init script change, add a missing file, modify dependencies, etc.). Unless a change is explicitly Juno specific (e.g. config option that doesn't exist in Icehouse, or versioned dependency incompatible with Icehouse), it should be applied to both releases. The flow should be pretty much the same as master-stable rule we have in git: apply the change to Juno, backport to Icehouse. -DmitryB On Thu, Sep 11, 2014 at 12:22 PM, Roman Vyalov rvya...@mirantis.com wrote: Mike, 2 jobs for Icehouse and Juno equal 2 different repository with packages for Fuel 6.0. This can be problem for current osci workflow. For example: We need building new packages. Which repository we must put packages? to icehouse or/and Juno ? if new packages will break icehouse repository, but required for Juno ... On Wed, Sep 10, 2014 at 12:39 AM, Mike Scherbakov mscherba...@mirantis.com wrote: Aleksandra, you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but 4 fuel-library tests should happen right after we create stable/5.1. Also, for Fuel CI for OSTF - I don't think it's actually necessary to support 5.0 envs. Your questions: Create jobs for both Icehouse and Juno, but it doesn't make sense to do staging for Juno till it starts to pass deployment in HA mode. Once it passes deployment in HA, staging should be enabled. Then, once it passes OSTF - we extend criteria, and pass only those mirrors which also pass OSTF phase Once Juno starts to pass BVT with OSTF check enabled, I think we can disable Icehouse checks. Not sure about fuel-library tests on Fuel CI with Icehouse - we might want to continue using them. Thanks, On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova afedor...@mirantis.com wrote: Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Let me rephrase: We keep one Fuel master branch for two OpenStack releases. And we make sure that Fuel master code is compatible with both of them. And we use current release (Icehouse) as a reference for test results of upcoming release, till we obtain stable enough reference point in Juno itself. Moreover we'd like to have OSTF code running on all previous Fuel releases. Changes to CI workflow look as follows: Nightly builds: 1) We build two mirrors: one for Icehouse and one for Juno. 2) From each mirror we build Fuel ISO using exactly the same fuel master branch code. 3) Then we run BVT tests on both (using the same fuel-main code for system tests). 4) If Icehouse BVT tests pass, we deploy both ISO images (even with failed Juno tests) onto Fuel CI. On Fuel CI we should run: - 4 fuel-library tests (revert master node, inject fuel-library code in master node and run deployment): 2 (ubuntu and centos) voting Icehouse tests and 2 non-voting Juno tests - 5 OSTF tests (revert deployed environment, inject OSTF code into master node, run OSTF): voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on master/Juno - other tests, which don't use prebuilt environment, work as before The major action point here would be OSTF tests, as we don't have yet working implementation of injecting OSTF code into deployed environment. And we don't run any tests on old environments. Questions: 1) How should we test mirrors? Current master mirrors go through the 4 hours test cycle involving Fuel ISO build: 1. we build temporary mirror 2. build custom iso from it 3. run two custom bvt jobs 4. if they pass we move mirror to stable and sitch to it for our primary fuel_master_iso Should we test only Icehouse mirrors, or both, but ignoring again failed BVT for Juno? Maybe we should enable these tests only later in release cycle, say, after SCF? 2) It is not clear for me when and how we will switch from supporting two releases back to one. Should
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
+1 to DmitryB, I think in this particular time and case we should open stable/5.1. But not to call it HCF [1]. Though I think we should retrospect our approaches here. 1. Sometimes we can squash 30 bugs a day, and formally reach HCF. Though the day after we will get 30 New bugs from QA. We might want to reconsider the whole approach on criteria, and come up with flow instead. Like if we have =5 High bugs at the moment, and over last 3 days we were seeing 10 confirmed High/Critical bugs, then we can call for HCF (if 60 bugs in 3 last days, then no way for HCF) 2. Consumption of new OpenStack release is hard and will be as such unless we will be using Fuel in gating process for every patch being pushed to OpenStack upstream. We want to deploy Juno now for 6.0, and the only way to do it now - is to build all packages, try to run it, observe issues, fix, run again, observe other issues... - and this process continues for many iterations before we get stable ISO which passes BVTs. It is obvious, that if we drop the Juno packages in, then our master is going to be broken. If we do any other feature development, then we don't know whether it's because Juno or that another feature. What should we do then? My suggestion on #2 is that we could keep backward compatibility with Icehouse code (on puppet side), and can continue to use BVT's, other testing against master branch using both Icehouse packages and Juno. Thus we can keep using gating process for fuel library, relying on stable Icehouse version. As for immediate action, again, I'm in favor of creating stable/5.1 in order to unblock feature development in master, while we are fixing last issues with OpenStack patching. [1] https://wiki.openstack.org/wiki/Fuel/Hard_Code_Freeze On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko dborodae...@mirantis.com wrote: TL;DR: Yes, our work on 6.0 features is currently blocked and it is becoming a major problem. No, I don't think we should create pre-release or feature branches. Instead, we should create stable/5.1 branches and open master for 6.0 work. We have reached a point in 5.1 release cycle where the scope of issues we are willing to address in this release is narrow enough to not require full attention of the whole team. We have engineers working on 6.0 features, and their work is essentially blocked until they have somewhere to commit their changes. Simply creating new branches is not even close to solving this problem: we have a whole CI infrastructure around every active release series (currently 5.1, 5.0, 4.1), including test jobs for gerrit commits, package repository mirrors updates, ISO image builds, smoke, build verification, and swarm tests for ISO images, documentation builds, etc. A branch without all that infrastructure isn't any better than current status quo: every developer tracking their own 6.0 work locally. Unrelated to all that, we also had a lot of very negative experience with feature branches in the past [0] [1], which is why we have decided to follow the OpenStack branching strategy: commit all feature changes directly to master and track bugfixes for stable releases in stable/* branches. [0] https://lists.launchpad.net/fuel-dev/msg00127.html [1] https://lists.launchpad.net/fuel-dev/msg00028.html I'm also against declaring a hard code freeze with exceptions, HCF should remain tied to our ability to declare a release candidate. If we can't release with the bugs we already know about, declaring HCF before fixing these bugs would be an empty gesture. Creating stable/5.1 now instead of waiting for hard code freeze for 5.1 will cost us two things: 1) DevOps team will have to update our CI infrastructure for one more release series. It's something we have to do for 6.0 sooner or later, so this may be a disruption, but not an additional effort. 2) All commits targeted for 5.1 will have to be proposed for two branches (master and stable/5.1) instead of just one (master). This will require additional effort, but I think that it is significantly smaller than the cost of spinning our wheels on 6.0 efforts. -DmitryB On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov dmescherya...@mirantis.com wrote: Hello Fuelers, Right now we have the following policy in place: the branches for a release are opened only after its 'parent' release have reached hard code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 and 6.0. And that is the problem: if parent release is delayed, we can't properly start development of a child release because we don't have branches to commit. That is current issue with 6.0: we already started to work on pushing Juno in to 6.0, but if we are to make changes to our deployment code we have nowhere to store them. IMHO the issue could easily be resolved by creation of pre-release branches, which are merged together with parent branches once the
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
Mike, just to clarify - do we want consider this as an exception, which is not going to be repeated next release? If not, we might want to consider updating the statement It is the time when master opens for next release changes, including features. in [1]. If I got you correct, we are going to open master for development of new features now. [1] https://wiki.openstack.org/wiki/Fuel/Hard_Code_Freeze Regards, Igor Marnat On Tue, Sep 9, 2014 at 12:07 PM, Mike Scherbakov mscherba...@mirantis.com wrote: +1 to DmitryB, I think in this particular time and case we should open stable/5.1. But not to call it HCF [1]. Though I think we should retrospect our approaches here. Sometimes we can squash 30 bugs a day, and formally reach HCF. Though the day after we will get 30 New bugs from QA. We might want to reconsider the whole approach on criteria, and come up with flow instead. Like if we have =5 High bugs at the moment, and over last 3 days we were seeing 10 confirmed High/Critical bugs, then we can call for HCF (if 60 bugs in 3 last days, then no way for HCF) Consumption of new OpenStack release is hard and will be as such unless we will be using Fuel in gating process for every patch being pushed to OpenStack upstream. We want to deploy Juno now for 6.0, and the only way to do it now - is to build all packages, try to run it, observe issues, fix, run again, observe other issues... - and this process continues for many iterations before we get stable ISO which passes BVTs. It is obvious, that if we drop the Juno packages in, then our master is going to be broken. If we do any other feature development, then we don't know whether it's because Juno or that another feature. What should we do then? My suggestion on #2 is that we could keep backward compatibility with Icehouse code (on puppet side), and can continue to use BVT's, other testing against master branch using both Icehouse packages and Juno. Thus we can keep using gating process for fuel library, relying on stable Icehouse version. As for immediate action, again, I'm in favor of creating stable/5.1 in order to unblock feature development in master, while we are fixing last issues with OpenStack patching. [1] https://wiki.openstack.org/wiki/Fuel/Hard_Code_Freeze On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko dborodae...@mirantis.com wrote: TL;DR: Yes, our work on 6.0 features is currently blocked and it is becoming a major problem. No, I don't think we should create pre-release or feature branches. Instead, we should create stable/5.1 branches and open master for 6.0 work. We have reached a point in 5.1 release cycle where the scope of issues we are willing to address in this release is narrow enough to not require full attention of the whole team. We have engineers working on 6.0 features, and their work is essentially blocked until they have somewhere to commit their changes. Simply creating new branches is not even close to solving this problem: we have a whole CI infrastructure around every active release series (currently 5.1, 5.0, 4.1), including test jobs for gerrit commits, package repository mirrors updates, ISO image builds, smoke, build verification, and swarm tests for ISO images, documentation builds, etc. A branch without all that infrastructure isn't any better than current status quo: every developer tracking their own 6.0 work locally. Unrelated to all that, we also had a lot of very negative experience with feature branches in the past [0] [1], which is why we have decided to follow the OpenStack branching strategy: commit all feature changes directly to master and track bugfixes for stable releases in stable/* branches. [0] https://lists.launchpad.net/fuel-dev/msg00127.html [1] https://lists.launchpad.net/fuel-dev/msg00028.html I'm also against declaring a hard code freeze with exceptions, HCF should remain tied to our ability to declare a release candidate. If we can't release with the bugs we already know about, declaring HCF before fixing these bugs would be an empty gesture. Creating stable/5.1 now instead of waiting for hard code freeze for 5.1 will cost us two things: 1) DevOps team will have to update our CI infrastructure for one more release series. It's something we have to do for 6.0 sooner or later, so this may be a disruption, but not an additional effort. 2) All commits targeted for 5.1 will have to be proposed for two branches (master and stable/5.1) instead of just one (master). This will require additional effort, but I think that it is significantly smaller than the cost of spinning our wheels on 6.0 efforts. -DmitryB On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov dmescherya...@mirantis.com wrote: Hello Fuelers, Right now we have the following policy in place: the branches for a release are opened only after its 'parent' release have reached hard code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
Currently I think we should take it as an exception, and discuss two points which I brought up. Obviously, if we open stable/5.1, then we are opening master for new features. We will modify HCF definition once we settle on final decision. Thanks, On Tue, Sep 9, 2014 at 1:02 PM, Igor Marnat imar...@mirantis.com wrote: Mike, just to clarify - do we want consider this as an exception, which is not going to be repeated next release? If not, we might want to consider updating the statement It is the time when master opens for next release changes, including features. in [1]. If I got you correct, we are going to open master for development of new features now. [1] https://wiki.openstack.org/wiki/Fuel/Hard_Code_Freeze Regards, Igor Marnat On Tue, Sep 9, 2014 at 12:07 PM, Mike Scherbakov mscherba...@mirantis.com wrote: +1 to DmitryB, I think in this particular time and case we should open stable/5.1. But not to call it HCF [1]. Though I think we should retrospect our approaches here. Sometimes we can squash 30 bugs a day, and formally reach HCF. Though the day after we will get 30 New bugs from QA. We might want to reconsider the whole approach on criteria, and come up with flow instead. Like if we have =5 High bugs at the moment, and over last 3 days we were seeing 10 confirmed High/Critical bugs, then we can call for HCF (if 60 bugs in 3 last days, then no way for HCF) Consumption of new OpenStack release is hard and will be as such unless we will be using Fuel in gating process for every patch being pushed to OpenStack upstream. We want to deploy Juno now for 6.0, and the only way to do it now - is to build all packages, try to run it, observe issues, fix, run again, observe other issues... - and this process continues for many iterations before we get stable ISO which passes BVTs. It is obvious, that if we drop the Juno packages in, then our master is going to be broken. If we do any other feature development, then we don't know whether it's because Juno or that another feature. What should we do then? My suggestion on #2 is that we could keep backward compatibility with Icehouse code (on puppet side), and can continue to use BVT's, other testing against master branch using both Icehouse packages and Juno. Thus we can keep using gating process for fuel library, relying on stable Icehouse version. As for immediate action, again, I'm in favor of creating stable/5.1 in order to unblock feature development in master, while we are fixing last issues with OpenStack patching. [1] https://wiki.openstack.org/wiki/Fuel/Hard_Code_Freeze On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko dborodae...@mirantis.com wrote: TL;DR: Yes, our work on 6.0 features is currently blocked and it is becoming a major problem. No, I don't think we should create pre-release or feature branches. Instead, we should create stable/5.1 branches and open master for 6.0 work. We have reached a point in 5.1 release cycle where the scope of issues we are willing to address in this release is narrow enough to not require full attention of the whole team. We have engineers working on 6.0 features, and their work is essentially blocked until they have somewhere to commit their changes. Simply creating new branches is not even close to solving this problem: we have a whole CI infrastructure around every active release series (currently 5.1, 5.0, 4.1), including test jobs for gerrit commits, package repository mirrors updates, ISO image builds, smoke, build verification, and swarm tests for ISO images, documentation builds, etc. A branch without all that infrastructure isn't any better than current status quo: every developer tracking their own 6.0 work locally. Unrelated to all that, we also had a lot of very negative experience with feature branches in the past [0] [1], which is why we have decided to follow the OpenStack branching strategy: commit all feature changes directly to master and track bugfixes for stable releases in stable/* branches. [0] https://lists.launchpad.net/fuel-dev/msg00127.html [1] https://lists.launchpad.net/fuel-dev/msg00028.html I'm also against declaring a hard code freeze with exceptions, HCF should remain tied to our ability to declare a release candidate. If we can't release with the bugs we already know about, declaring HCF before fixing these bugs would be an empty gesture. Creating stable/5.1 now instead of waiting for hard code freeze for 5.1 will cost us two things: 1) DevOps team will have to update our CI infrastructure for one more release series. It's something we have to do for 6.0 sooner or later, so this may be a disruption, but not an additional effort. 2) All commits targeted for 5.1 will have to be proposed for two branches (master and stable/5.1) instead of just one (master). This will require additional effort, but
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
As I understand your proposal, we need to split our HCF milestone into two check points: Branching Point and HCF itself. Branching point should happen somewhere in between SCF and HCF. And though It may coincide with HCF, it needs its own list of requirements. This will give us the possibility to untie two events and make a separate decision on branching without enforcing all HCF criteria. From the DevOps point of view it changes almost nothing, it just adds a bit more discussion items on the management side and slight modifications to our checklists. On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko dborodae...@mirantis.com wrote: TL;DR: Yes, our work on 6.0 features is currently blocked and it is becoming a major problem. No, I don't think we should create pre-release or feature branches. Instead, we should create stable/5.1 branches and open master for 6.0 work. We have reached a point in 5.1 release cycle where the scope of issues we are willing to address in this release is narrow enough to not require full attention of the whole team. We have engineers working on 6.0 features, and their work is essentially blocked until they have somewhere to commit their changes. Simply creating new branches is not even close to solving this problem: we have a whole CI infrastructure around every active release series (currently 5.1, 5.0, 4.1), including test jobs for gerrit commits, package repository mirrors updates, ISO image builds, smoke, build verification, and swarm tests for ISO images, documentation builds, etc. A branch without all that infrastructure isn't any better than current status quo: every developer tracking their own 6.0 work locally. Unrelated to all that, we also had a lot of very negative experience with feature branches in the past [0] [1], which is why we have decided to follow the OpenStack branching strategy: commit all feature changes directly to master and track bugfixes for stable releases in stable/* branches. [0] https://lists.launchpad.net/fuel-dev/msg00127.html [1] https://lists.launchpad.net/fuel-dev/msg00028.html I'm also against declaring a hard code freeze with exceptions, HCF should remain tied to our ability to declare a release candidate. If we can't release with the bugs we already know about, declaring HCF before fixing these bugs would be an empty gesture. Creating stable/5.1 now instead of waiting for hard code freeze for 5.1 will cost us two things: 1) DevOps team will have to update our CI infrastructure for one more release series. It's something we have to do for 6.0 sooner or later, so this may be a disruption, but not an additional effort. 2) All commits targeted for 5.1 will have to be proposed for two branches (master and stable/5.1) instead of just one (master). This will require additional effort, but I think that it is significantly smaller than the cost of spinning our wheels on 6.0 efforts. -DmitryB On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov dmescherya...@mirantis.com wrote: Hello Fuelers, Right now we have the following policy in place: the branches for a release are opened only after its 'parent' release have reached hard code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 and 6.0. And that is the problem: if parent release is delayed, we can't properly start development of a child release because we don't have branches to commit. That is current issue with 6.0: we already started to work on pushing Juno in to 6.0, but if we are to make changes to our deployment code we have nowhere to store them. IMHO the issue could easily be resolved by creation of pre-release branches, which are merged together with parent branches once the parent reaches HCF. Say, we use branch 'pre-6.0' for initial development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into master and continue development here. After that pre-6.0 is abandoned. What do you think? Thanks, Dmitry ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Dmitry Borodaenko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Aleksandra Fedorova bookwar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
Thanks Alexandra. We land a few patches a day currently, so I think we can open stable branch. If we see no serious objections in next 12 hours, let's do it. We would need to immediately notify everyone in mailing list - that for every patch for 5.1, it should go first to master, and then to stable/5.1. Is everything ready from DevOps, OSCI (packaging) side to do this? Fuel CI, OBS, etc.? On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova afedor...@mirantis.com wrote: As I understand your proposal, we need to split our HCF milestone into two check points: Branching Point and HCF itself. Branching point should happen somewhere in between SCF and HCF. And though It may coincide with HCF, it needs its own list of requirements. This will give us the possibility to untie two events and make a separate decision on branching without enforcing all HCF criteria. From the DevOps point of view it changes almost nothing, it just adds a bit more discussion items on the management side and slight modifications to our checklists. On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko dborodae...@mirantis.com wrote: TL;DR: Yes, our work on 6.0 features is currently blocked and it is becoming a major problem. No, I don't think we should create pre-release or feature branches. Instead, we should create stable/5.1 branches and open master for 6.0 work. We have reached a point in 5.1 release cycle where the scope of issues we are willing to address in this release is narrow enough to not require full attention of the whole team. We have engineers working on 6.0 features, and their work is essentially blocked until they have somewhere to commit their changes. Simply creating new branches is not even close to solving this problem: we have a whole CI infrastructure around every active release series (currently 5.1, 5.0, 4.1), including test jobs for gerrit commits, package repository mirrors updates, ISO image builds, smoke, build verification, and swarm tests for ISO images, documentation builds, etc. A branch without all that infrastructure isn't any better than current status quo: every developer tracking their own 6.0 work locally. Unrelated to all that, we also had a lot of very negative experience with feature branches in the past [0] [1], which is why we have decided to follow the OpenStack branching strategy: commit all feature changes directly to master and track bugfixes for stable releases in stable/* branches. [0] https://lists.launchpad.net/fuel-dev/msg00127.html [1] https://lists.launchpad.net/fuel-dev/msg00028.html I'm also against declaring a hard code freeze with exceptions, HCF should remain tied to our ability to declare a release candidate. If we can't release with the bugs we already know about, declaring HCF before fixing these bugs would be an empty gesture. Creating stable/5.1 now instead of waiting for hard code freeze for 5.1 will cost us two things: 1) DevOps team will have to update our CI infrastructure for one more release series. It's something we have to do for 6.0 sooner or later, so this may be a disruption, but not an additional effort. 2) All commits targeted for 5.1 will have to be proposed for two branches (master and stable/5.1) instead of just one (master). This will require additional effort, but I think that it is significantly smaller than the cost of spinning our wheels on 6.0 efforts. -DmitryB On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov dmescherya...@mirantis.com wrote: Hello Fuelers, Right now we have the following policy in place: the branches for a release are opened only after its 'parent' release have reached hard code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 and 6.0. And that is the problem: if parent release is delayed, we can't properly start development of a child release because we don't have branches to commit. That is current issue with 6.0: we already started to work on pushing Juno in to 6.0, but if we are to make changes to our deployment code we have nowhere to store them. IMHO the issue could easily be resolved by creation of pre-release branches, which are merged together with parent branches once the parent reaches HCF. Say, we use branch 'pre-6.0' for initial development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into master and continue development here. After that pre-6.0 is abandoned. What do you think? Thanks, Dmitry ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Dmitry Borodaenko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Aleksandra Fedorova bookwar ___ OpenStack-dev mailing list
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
+1 on adding flow based criteria for HCF and Branching Point. Tracking down how many bugs was reported in LP on a given day is a bit tricky, so I think in both cases it would be easier to rely on flow of commits (which after Soft Code Freeze becomes a direct indicator of how many bugs are fixed per day). For HCF, not having merged any code changes for 24 hours (on top of meeting the bug count criteria) would be a solid proof of code stabilization. For Branching Point, the threshold has to be more relaxed, for example, less than 10 commits across all our code repositories within last 24 hours. I agree that landing Juno packages right now is going be disruptive. I continue to think that the only way to address this long term is to have a parallel OpenStack master branch based Fuel/MOS build series. What we need to achieve that is have 2 build series based on Fuel master: one with Icehouse packages, and one with Juno, and, as Mike proposed, keep our manifests backwards compatible with Icehouse. That way, we can land changes unrelated to Juno into Fuel master and debug them using the Icehouse based build series, and land changes needed for Juno integration into the same Fuel master branch, and debug them in Juno based build series. In order to minimize impact of non Juno related changes on Juno integration, we should minimize breakage of Icehouse based builds. Two ways to address that problem would be to serialize landing of major changes (so that only one feature at a time can be blamed for breakage), and agressively revert changes that cause BVT failure if they are not fixed within 24 hours. As soon as Juno builds reach the same level of stability of Icehouse, we can drop the Icehouse based builds and introduce Kilo based builds instead. Thoughts? -DmitryB On Tue, Sep 9, 2014 at 7:27 AM, Mike Scherbakov mscherba...@mirantis.com wrote: Thanks Alexandra. We land a few patches a day currently, so I think we can open stable branch. If we see no serious objections in next 12 hours, let's do it. We would need to immediately notify everyone in mailing list - that for every patch for 5.1, it should go first to master, and then to stable/5.1. Is everything ready from DevOps, OSCI (packaging) side to do this? Fuel CI, OBS, etc.? On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova afedor...@mirantis.com wrote: As I understand your proposal, we need to split our HCF milestone into two check points: Branching Point and HCF itself. Branching point should happen somewhere in between SCF and HCF. And though It may coincide with HCF, it needs its own list of requirements. This will give us the possibility to untie two events and make a separate decision on branching without enforcing all HCF criteria. From the DevOps point of view it changes almost nothing, it just adds a bit more discussion items on the management side and slight modifications to our checklists. On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko dborodae...@mirantis.com wrote: TL;DR: Yes, our work on 6.0 features is currently blocked and it is becoming a major problem. No, I don't think we should create pre-release or feature branches. Instead, we should create stable/5.1 branches and open master for 6.0 work. We have reached a point in 5.1 release cycle where the scope of issues we are willing to address in this release is narrow enough to not require full attention of the whole team. We have engineers working on 6.0 features, and their work is essentially blocked until they have somewhere to commit their changes. Simply creating new branches is not even close to solving this problem: we have a whole CI infrastructure around every active release series (currently 5.1, 5.0, 4.1), including test jobs for gerrit commits, package repository mirrors updates, ISO image builds, smoke, build verification, and swarm tests for ISO images, documentation builds, etc. A branch without all that infrastructure isn't any better than current status quo: every developer tracking their own 6.0 work locally. Unrelated to all that, we also had a lot of very negative experience with feature branches in the past [0] [1], which is why we have decided to follow the OpenStack branching strategy: commit all feature changes directly to master and track bugfixes for stable releases in stable/* branches. [0] https://lists.launchpad.net/fuel-dev/msg00127.html [1] https://lists.launchpad.net/fuel-dev/msg00028.html I'm also against declaring a hard code freeze with exceptions, HCF should remain tied to our ability to declare a release candidate. If we can't release with the bugs we already know about, declaring HCF before fixing these bugs would be an empty gesture. Creating stable/5.1 now instead of waiting for hard code freeze for 5.1 will cost us two things: 1) DevOps team will have to update our CI infrastructure for one more release series. It's something we have to do for 6.0 sooner or later, so this may be
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
All OSCI action items for prepare HCF check list has been done On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov mscherba...@mirantis.com wrote: Thanks Alexandra. We land a few patches a day currently, so I think we can open stable branch. If we see no serious objections in next 12 hours, let's do it. We would need to immediately notify everyone in mailing list - that for every patch for 5.1, it should go first to master, and then to stable/5.1. Is everything ready from DevOps, OSCI (packaging) side to do this? Fuel CI, OBS, etc.? On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova afedor...@mirantis.com wrote: As I understand your proposal, we need to split our HCF milestone into two check points: Branching Point and HCF itself. Branching point should happen somewhere in between SCF and HCF. And though It may coincide with HCF, it needs its own list of requirements. This will give us the possibility to untie two events and make a separate decision on branching without enforcing all HCF criteria. From the DevOps point of view it changes almost nothing, it just adds a bit more discussion items on the management side and slight modifications to our checklists. On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko dborodae...@mirantis.com wrote: TL;DR: Yes, our work on 6.0 features is currently blocked and it is becoming a major problem. No, I don't think we should create pre-release or feature branches. Instead, we should create stable/5.1 branches and open master for 6.0 work. We have reached a point in 5.1 release cycle where the scope of issues we are willing to address in this release is narrow enough to not require full attention of the whole team. We have engineers working on 6.0 features, and their work is essentially blocked until they have somewhere to commit their changes. Simply creating new branches is not even close to solving this problem: we have a whole CI infrastructure around every active release series (currently 5.1, 5.0, 4.1), including test jobs for gerrit commits, package repository mirrors updates, ISO image builds, smoke, build verification, and swarm tests for ISO images, documentation builds, etc. A branch without all that infrastructure isn't any better than current status quo: every developer tracking their own 6.0 work locally. Unrelated to all that, we also had a lot of very negative experience with feature branches in the past [0] [1], which is why we have decided to follow the OpenStack branching strategy: commit all feature changes directly to master and track bugfixes for stable releases in stable/* branches. [0] https://lists.launchpad.net/fuel-dev/msg00127.html [1] https://lists.launchpad.net/fuel-dev/msg00028.html I'm also against declaring a hard code freeze with exceptions, HCF should remain tied to our ability to declare a release candidate. If we can't release with the bugs we already know about, declaring HCF before fixing these bugs would be an empty gesture. Creating stable/5.1 now instead of waiting for hard code freeze for 5.1 will cost us two things: 1) DevOps team will have to update our CI infrastructure for one more release series. It's something we have to do for 6.0 sooner or later, so this may be a disruption, but not an additional effort. 2) All commits targeted for 5.1 will have to be proposed for two branches (master and stable/5.1) instead of just one (master). This will require additional effort, but I think that it is significantly smaller than the cost of spinning our wheels on 6.0 efforts. -DmitryB On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov dmescherya...@mirantis.com wrote: Hello Fuelers, Right now we have the following policy in place: the branches for a release are opened only after its 'parent' release have reached hard code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 and 6.0. And that is the problem: if parent release is delayed, we can't properly start development of a child release because we don't have branches to commit. That is current issue with 6.0: we already started to work on pushing Juno in to 6.0, but if we are to make changes to our deployment code we have nowhere to store them. IMHO the issue could easily be resolved by creation of pre-release branches, which are merged together with parent branches once the parent reaches HCF. Say, we use branch 'pre-6.0' for initial development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into master and continue development here. After that pre-6.0 is abandoned. What do you think? Thanks, Dmitry ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Dmitry Borodaenko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
What we need to achieve that is have 2 build series based on Fuel master: one with Icehouse packages, and one with Juno, and, as Mike proposed, keep our manifests backwards compatible with Icehouse. Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Another suggestion would be to lower green switch in BVTs for Juno: first, when it passes deployment; and then, if it finally passes OSTF. I'd like to hear QA DevOps opinion on all the above. Immediately we would need just standard stuff which is in checklists for OSCI DevOps teams, and ideally soon after that - ability to have Fuel CI running 4 builds, not 2, against our master, as mentioned above. On Tue, Sep 9, 2014 at 9:28 PM, Roman Vyalov rvya...@mirantis.com wrote: All OSCI action items for prepare HCF check list has been done On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov mscherba...@mirantis.com wrote: Thanks Alexandra. We land a few patches a day currently, so I think we can open stable branch. If we see no serious objections in next 12 hours, let's do it. We would need to immediately notify everyone in mailing list - that for every patch for 5.1, it should go first to master, and then to stable/5.1. Is everything ready from DevOps, OSCI (packaging) side to do this? Fuel CI, OBS, etc.? On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova afedor...@mirantis.com wrote: As I understand your proposal, we need to split our HCF milestone into two check points: Branching Point and HCF itself. Branching point should happen somewhere in between SCF and HCF. And though It may coincide with HCF, it needs its own list of requirements. This will give us the possibility to untie two events and make a separate decision on branching without enforcing all HCF criteria. From the DevOps point of view it changes almost nothing, it just adds a bit more discussion items on the management side and slight modifications to our checklists. On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko dborodae...@mirantis.com wrote: TL;DR: Yes, our work on 6.0 features is currently blocked and it is becoming a major problem. No, I don't think we should create pre-release or feature branches. Instead, we should create stable/5.1 branches and open master for 6.0 work. We have reached a point in 5.1 release cycle where the scope of issues we are willing to address in this release is narrow enough to not require full attention of the whole team. We have engineers working on 6.0 features, and their work is essentially blocked until they have somewhere to commit their changes. Simply creating new branches is not even close to solving this problem: we have a whole CI infrastructure around every active release series (currently 5.1, 5.0, 4.1), including test jobs for gerrit commits, package repository mirrors updates, ISO image builds, smoke, build verification, and swarm tests for ISO images, documentation builds, etc. A branch without all that infrastructure isn't any better than current status quo: every developer tracking their own 6.0 work locally. Unrelated to all that, we also had a lot of very negative experience with feature branches in the past [0] [1], which is why we have decided to follow the OpenStack branching strategy: commit all feature changes directly to master and track bugfixes for stable releases in stable/* branches. [0] https://lists.launchpad.net/fuel-dev/msg00127.html [1] https://lists.launchpad.net/fuel-dev/msg00028.html I'm also against declaring a hard code freeze with exceptions, HCF should remain tied to our ability to declare a release candidate. If we can't release with the bugs we already know about, declaring HCF before fixing these bugs would be an empty gesture. Creating stable/5.1 now instead of waiting for hard code freeze for 5.1 will cost us two things: 1) DevOps team will have to update our CI infrastructure for one more release series. It's something we have to do for 6.0 sooner or later, so this may be a disruption, but not an additional effort. 2) All commits targeted for 5.1 will have to be proposed for two branches (master and stable/5.1) instead of just one (master). This will require additional effort, but I think that it is significantly smaller than the cost of spinning our wheels on 6.0 efforts. -DmitryB On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov dmescherya...@mirantis.com wrote: Hello Fuelers, Right now we have the following policy in place: the branches for a release are opened only after its 'parent' release have reached
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Let me rephrase: We keep one Fuel master branch for two OpenStack releases. And we make sure that Fuel master code is compatible with both of them. And we use current release (Icehouse) as a reference for test results of upcoming release, till we obtain stable enough reference point in Juno itself. Moreover we'd like to have OSTF code running on all previous Fuel releases. Changes to CI workflow look as follows: Nightly builds: 1) We build two mirrors: one for Icehouse and one for Juno. 2) From each mirror we build Fuel ISO using exactly the same fuel master branch code. 3) Then we run BVT tests on both (using the same fuel-main code for system tests). 4) If Icehouse BVT tests pass, we deploy both ISO images (even with failed Juno tests) onto Fuel CI. On Fuel CI we should run: - 4 fuel-library tests (revert master node, inject fuel-library code in master node and run deployment): 2 (ubuntu and centos) voting Icehouse tests and 2 non-voting Juno tests - 5 OSTF tests (revert deployed environment, inject OSTF code into master node, run OSTF): voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on master/Juno - other tests, which don't use prebuilt environment, work as before The major action point here would be OSTF tests, as we don't have yet working implementation of injecting OSTF code into deployed environment. And we don't run any tests on old environments. Questions: 1) How should we test mirrors? Current master mirrors go through the 4 hours test cycle involving Fuel ISO build: 1. we build temporary mirror 2. build custom iso from it 3. run two custom bvt jobs 4. if they pass we move mirror to stable and sitch to it for our primary fuel_master_iso Should we test only Icehouse mirrors, or both, but ignoring again failed BVT for Juno? Maybe we should enable these tests only later in release cycle, say, after SCF? 2) It is not clear for me when and how we will switch from supporting two releases back to one. Should we add one more milestone to our release process? The Switching point, when we disable and remove Icehouse tasks and move to Juno completely? I guess it should happen before next SCF? On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov mscherba...@mirantis.com wrote: What we need to achieve that is have 2 build series based on Fuel master: one with Icehouse packages, and one with Juno, and, as Mike proposed, keep our manifests backwards compatible with Icehouse. Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Another suggestion would be to lower green switch in BVTs for Juno: first, when it passes deployment; and then, if it finally passes OSTF. I'd like to hear QA DevOps opinion on all the above. Immediately we would need just standard stuff which is in checklists for OSCI DevOps teams, and ideally soon after that - ability to have Fuel CI running 4 builds, not 2, against our master, as mentioned above. On Tue, Sep 9, 2014 at 9:28 PM, Roman Vyalov rvya...@mirantis.com wrote: All OSCI action items for prepare HCF check list has been done On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov mscherba...@mirantis.com wrote: Thanks Alexandra. We land a few patches a day currently, so I think we can open stable branch. If we see no serious objections in next 12 hours, let's do it. We would need to immediately notify everyone in mailing list - that for every patch for 5.1, it should go first to master, and then to stable/5.1. Is everything ready from DevOps, OSCI (packaging) side to do this? Fuel CI, OBS, etc.? On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova afedor...@mirantis.com wrote: As I understand your proposal, we need to split our HCF milestone into two check points: Branching Point and HCF itself. Branching point should happen somewhere in between SCF and HCF. And though It may coincide with HCF, it needs its own list of requirements. This will give us the possibility to untie two events and make a separate decision on branching without enforcing all HCF criteria. From the DevOps point of view it changes almost
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
Aleksandra, you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but 4 fuel-library tests should happen right after we create stable/5.1. Also, for Fuel CI for OSTF - I don't think it's actually necessary to support 5.0 envs. Your questions: 1. Create jobs for both Icehouse and Juno, but it doesn't make sense to do staging for Juno till it starts to pass deployment in HA mode. Once it passes deployment in HA, staging should be enabled. Then, once it passes OSTF - we extend criteria, and pass only those mirrors which also pass OSTF phase 2. Once Juno starts to pass BVT with OSTF check enabled, I think we can disable Icehouse checks. Not sure about fuel-library tests on Fuel CI with Icehouse - we might want to continue using them. Thanks, On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova afedor...@mirantis.com wrote: Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Let me rephrase: We keep one Fuel master branch for two OpenStack releases. And we make sure that Fuel master code is compatible with both of them. And we use current release (Icehouse) as a reference for test results of upcoming release, till we obtain stable enough reference point in Juno itself. Moreover we'd like to have OSTF code running on all previous Fuel releases. Changes to CI workflow look as follows: Nightly builds: 1) We build two mirrors: one for Icehouse and one for Juno. 2) From each mirror we build Fuel ISO using exactly the same fuel master branch code. 3) Then we run BVT tests on both (using the same fuel-main code for system tests). 4) If Icehouse BVT tests pass, we deploy both ISO images (even with failed Juno tests) onto Fuel CI. On Fuel CI we should run: - 4 fuel-library tests (revert master node, inject fuel-library code in master node and run deployment): 2 (ubuntu and centos) voting Icehouse tests and 2 non-voting Juno tests - 5 OSTF tests (revert deployed environment, inject OSTF code into master node, run OSTF): voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on master/Juno - other tests, which don't use prebuilt environment, work as before The major action point here would be OSTF tests, as we don't have yet working implementation of injecting OSTF code into deployed environment. And we don't run any tests on old environments. Questions: 1) How should we test mirrors? Current master mirrors go through the 4 hours test cycle involving Fuel ISO build: 1. we build temporary mirror 2. build custom iso from it 3. run two custom bvt jobs 4. if they pass we move mirror to stable and sitch to it for our primary fuel_master_iso Should we test only Icehouse mirrors, or both, but ignoring again failed BVT for Juno? Maybe we should enable these tests only later in release cycle, say, after SCF? 2) It is not clear for me when and how we will switch from supporting two releases back to one. Should we add one more milestone to our release process? The Switching point, when we disable and remove Icehouse tasks and move to Juno completely? I guess it should happen before next SCF? On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov mscherba...@mirantis.com wrote: What we need to achieve that is have 2 build series based on Fuel master: one with Icehouse packages, and one with Juno, and, as Mike proposed, keep our manifests backwards compatible with Icehouse. Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Another suggestion would be to lower green switch in BVTs for Juno: first, when it passes deployment; and then, if it finally passes OSTF. I'd like to hear QA DevOps opinion on all the above. Immediately we would need just standard stuff which is in checklists for OSCI DevOps teams, and ideally soon after that - ability to have Fuel CI running 4 builds, not 2, against our master, as mentioned above. On Tue, Sep 9, 2014 at 9:28 PM, Roman Vyalov rvya...@mirantis.com wrote: All OSCI action items for prepare HCF check list has been done On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov mscherba...@mirantis.com wrote: Thanks
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
A clarification on 2: we are going to keep fuel_5.1_* jobs around for the benefict 5.1.x maintenance releases, that should take care of Icehouse testing for us, so I don't think we should keep Icehouse jobs in 6.0/master after Juno is stabilized. What we should do instead is drop Icehouse jobs and introduce Kilo jobs tracking OpenStack master, and try to keep manifests in Fuel master forward-compatible with Kilo through the 6.x release series. On Tue, Sep 9, 2014 at 1:39 PM, Mike Scherbakov mscherba...@mirantis.com wrote: Aleksandra, you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but 4 fuel-library tests should happen right after we create stable/5.1. Also, for Fuel CI for OSTF - I don't think it's actually necessary to support 5.0 envs. Your questions: Create jobs for both Icehouse and Juno, but it doesn't make sense to do staging for Juno till it starts to pass deployment in HA mode. Once it passes deployment in HA, staging should be enabled. Then, once it passes OSTF - we extend criteria, and pass only those mirrors which also pass OSTF phase Once Juno starts to pass BVT with OSTF check enabled, I think we can disable Icehouse checks. Not sure about fuel-library tests on Fuel CI with Icehouse - we might want to continue using them. Thanks, On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova afedor...@mirantis.com wrote: Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Let me rephrase: We keep one Fuel master branch for two OpenStack releases. And we make sure that Fuel master code is compatible with both of them. And we use current release (Icehouse) as a reference for test results of upcoming release, till we obtain stable enough reference point in Juno itself. Moreover we'd like to have OSTF code running on all previous Fuel releases. Changes to CI workflow look as follows: Nightly builds: 1) We build two mirrors: one for Icehouse and one for Juno. 2) From each mirror we build Fuel ISO using exactly the same fuel master branch code. 3) Then we run BVT tests on both (using the same fuel-main code for system tests). 4) If Icehouse BVT tests pass, we deploy both ISO images (even with failed Juno tests) onto Fuel CI. On Fuel CI we should run: - 4 fuel-library tests (revert master node, inject fuel-library code in master node and run deployment): 2 (ubuntu and centos) voting Icehouse tests and 2 non-voting Juno tests - 5 OSTF tests (revert deployed environment, inject OSTF code into master node, run OSTF): voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on master/Juno - other tests, which don't use prebuilt environment, work as before The major action point here would be OSTF tests, as we don't have yet working implementation of injecting OSTF code into deployed environment. And we don't run any tests on old environments. Questions: 1) How should we test mirrors? Current master mirrors go through the 4 hours test cycle involving Fuel ISO build: 1. we build temporary mirror 2. build custom iso from it 3. run two custom bvt jobs 4. if they pass we move mirror to stable and sitch to it for our primary fuel_master_iso Should we test only Icehouse mirrors, or both, but ignoring again failed BVT for Juno? Maybe we should enable these tests only later in release cycle, say, after SCF? 2) It is not clear for me when and how we will switch from supporting two releases back to one. Should we add one more milestone to our release process? The Switching point, when we disable and remove Icehouse tasks and move to Juno completely? I guess it should happen before next SCF? On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov mscherba...@mirantis.com wrote: What we need to achieve that is have 2 build series based on Fuel master: one with Icehouse packages, and one with Juno, and, as Mike proposed, keep our manifests backwards compatible with Icehouse. Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting, with Icehouse packages; 2 non-voting, with Juno packages. Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually before Juno becomes stable. We will be able to run 2 sets of BVTs (against Icehouse and Juno), and it means that we will be able to see almost immediately if something in nailgun/astute/puppet integration broke. For Juno builds it's going to be all red initially. Another suggestion would be to lower green switch in BVTs for Juno: first, when it passes deployment; and then, if it
[openstack-dev] [Fuel] Working on 6.0 and new releases in general
Hello Fuelers, Right now we have the following policy in place: the branches for a release are opened only after its 'parent' release have reached hard code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 and 6.0. And that is the problem: if parent release is delayed, we can't properly start development of a child release because we don't have branches to commit. That is current issue with 6.0: we already started to work on pushing Juno in to 6.0, but if we are to make changes to our deployment code we have nowhere to store them. IMHO the issue could easily be resolved by creation of pre-release branches, which are merged together with parent branches once the parent reaches HCF. Say, we use branch 'pre-6.0' for initial development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into master and continue development here. After that pre-6.0 is abandoned. What do you think? Thanks, Dmitry ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Working on 6.0 and new releases in general
TL;DR: Yes, our work on 6.0 features is currently blocked and it is becoming a major problem. No, I don't think we should create pre-release or feature branches. Instead, we should create stable/5.1 branches and open master for 6.0 work. We have reached a point in 5.1 release cycle where the scope of issues we are willing to address in this release is narrow enough to not require full attention of the whole team. We have engineers working on 6.0 features, and their work is essentially blocked until they have somewhere to commit their changes. Simply creating new branches is not even close to solving this problem: we have a whole CI infrastructure around every active release series (currently 5.1, 5.0, 4.1), including test jobs for gerrit commits, package repository mirrors updates, ISO image builds, smoke, build verification, and swarm tests for ISO images, documentation builds, etc. A branch without all that infrastructure isn't any better than current status quo: every developer tracking their own 6.0 work locally. Unrelated to all that, we also had a lot of very negative experience with feature branches in the past [0] [1], which is why we have decided to follow the OpenStack branching strategy: commit all feature changes directly to master and track bugfixes for stable releases in stable/* branches. [0] https://lists.launchpad.net/fuel-dev/msg00127.html [1] https://lists.launchpad.net/fuel-dev/msg00028.html I'm also against declaring a hard code freeze with exceptions, HCF should remain tied to our ability to declare a release candidate. If we can't release with the bugs we already know about, declaring HCF before fixing these bugs would be an empty gesture. Creating stable/5.1 now instead of waiting for hard code freeze for 5.1 will cost us two things: 1) DevOps team will have to update our CI infrastructure for one more release series. It's something we have to do for 6.0 sooner or later, so this may be a disruption, but not an additional effort. 2) All commits targeted for 5.1 will have to be proposed for two branches (master and stable/5.1) instead of just one (master). This will require additional effort, but I think that it is significantly smaller than the cost of spinning our wheels on 6.0 efforts. -DmitryB On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov dmescherya...@mirantis.com wrote: Hello Fuelers, Right now we have the following policy in place: the branches for a release are opened only after its 'parent' release have reached hard code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 and 6.0. And that is the problem: if parent release is delayed, we can't properly start development of a child release because we don't have branches to commit. That is current issue with 6.0: we already started to work on pushing Juno in to 6.0, but if we are to make changes to our deployment code we have nowhere to store them. IMHO the issue could easily be resolved by creation of pre-release branches, which are merged together with parent branches once the parent reaches HCF. Say, we use branch 'pre-6.0' for initial development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into master and continue development here. After that pre-6.0 is abandoned. What do you think? Thanks, Dmitry ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Dmitry Borodaenko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
