Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Dmitry Teselkin]

Hello,

Status update for Dec, 5-6 - we're on track.

CentOS7 based ISO looks good, we are building it on product CI using
the same jobs as for CentOS6.

ISO #258 passed nightly swarm with a slightly reduced coverage (69%).
We've got one known issue [0] that affects some jobs when run on a
loaded slave (there is a fix [1]), and issues on QA side.

Smoke and BVT tests also green [2], however, we've got interference
from Ubuntu upstream here, and had to disable trusty-proposed channel
to get tests passed. More details in a bug [3].

According to our plan [4] we've got over decision point #4 with a
decision to go with CentOS7.

Please note that merge freeze still taken place, it will be explicitly
notified when it's lifted.

[0] https://bugs.launchpad.net/mos/+bug/1523117
[1] https://review.openstack.org/#/c/253843/
[2] https://product-ci.infra.mirantis.net/job/8.0.test_all/245/
[3] https://bugs.launchpad.net/fuel/+bug/1523092
[4]
http://lists.openstack.org/pipermail/openstack-dev/2015-December/081026.html

On Tue, 1 Dec 2015 13:48:00 -0800
Dmitry Borodaenko  wrote:

> With bit more details, I hope this covers all the risks and decision
> points now.
> 
> First of all, current list of outstanding commits:
> https://etherpad.openstack.org/p/fuel_on_centos7
> 
> The above list has two sections: backwards compatible changes that can
> be merged one at a time even if the rest of CentOS7 support isn't
> merged, and backwards incompatible changes that break support for
> CentOS6 and must be merged (and, if needed, reverted) all at once.
> 
> Decision point 1: FFE for CentOS7
> 
> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can
> it be allowed a Feature Freeze Exception? So far, the disruption of
> the Fuel development process implied by the proposed merge plan is
> acceptable, if anything goes wrong and we become unable to have a
> stable ISO with merged CentOS7 support on Monday, December 7, the FFE
> will be revoked.
> 
> Wed, Dec 2: Merge party
> 
> Merge party before 8.0 FF, we should do our best to merge all
> remaining feature commits before end of day (including backwards
> compatible CentOS7 support commits), without breaking the build too
> much.
> 
> At the end of the day we'll start a swarm test over the result of the
> merge party, and we expect QA to analyze and summarize the results by
> 17:00 MSK (6:00 PST) on Thu Dec 3.
> 
> Risk 1: Merge party breaks the build
> 
> If there is a large regression in swarm pass percentage, we won't be
> able to afford a merge freeze which is necessary to merge CentOS7
> support, we'll have to be merging bugfixes until swarm test pass rate
> is back around 70%.
> 
> Risk 2: More features get FFE
> 
> If some essential 8.0 features are not completely merged by end of day
> Wed Dec 2 and are granted FFE, merging the remaining commits can
> interfere with merging CentOS7 support, not just from merge conflicts
> perspective, but also invalidating swarm results and making it
> practically impossible to bisect and attribute potential regressions.
> 
> Thu, Dec 3: Start merge freeze for CentOS7
> 
> Decision point 2: Other FFEs
> 
> In the morning MSK time, we will assess Risk 2 and decide what to do
> with the other FFEs. The options are: integrate remaining commits into
> CentOS7 merge plan, block remaining commits until Monday, revoke
> CentOS7 FFE.
> 
> If the decision is to go ahead with CentOS7 merge, we announce merge
> freeze for all git repositories that go into Fuel ISO, and spend the
> rest of the day rebasing and cleaning up the rest of the CentOS7
> commits to make sure they're all in mergeable state by the end of the
> day. The outcome of this work must be a custom ISO image with all
> remaining commits, with additional requirement that it must not use
> Jenkins job parameters (only patches to fuel-main that change default
> repository paths) to specify all required package repositories. This
> will validate the proposed fuel-main patches and ensure that no
> unmerged package changes are used to produce the ISO.
> 
> Decision point 3: Swarm pass rate
> 
> After swarm results from Wed are available, we will assess the Risk 1.
> If the pass rate regression is significant, CentOS7 FFE is revoked and
> merge freeze is lifted. If regression is acceptable, we proceed with
> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
> 
> Fri, Dec 4: Merge and test CentOS7
> 
> The team will have until 17:00 MSK to produce a non-custom ISO that
> passes BVT and can be run through swarm.
> 
> Sat, Dec 5: Assess CentOS7 swarm and bugfix
> 
> First of all, someone from CI and QA teams should commit to monitoring
> the CentOS7 swarm run and report the results as soon as possible.
> Based on the results (which once again must be available by 17:00
> MSK), we can decide on the final step of the plan.
> 
> Decision point 4: Keep or revert
> 
> If CentOS7 based swarm shows significant regression, we have to spend
> the 

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Dmitry Teselkin]

Hello,

According to the CentOS7 merging plan we've merged all CentOS7 CRs and
are going to get our first builds in production CI.

Please note that failures are possible for short period of time, please
don't panic, we will keep an eye on every build to investigate and fix
ASAP. We will send another notification letter when it'll be stable
again. We are going to work on fixing issues (if any) until Sunday
(Dec, 6) since it's the next decision point when it'll be clear should
we go with CentOS7 further or revert.

I'd like to remind that merge freeze still in place, no merges allowed
until it will be explicitly notified.


On Tue, 1 Dec 2015 13:48:00 -0800
Dmitry Borodaenko  wrote:

> With bit more details, I hope this covers all the risks and decision
> points now.
> 
> First of all, current list of outstanding commits:
> https://etherpad.openstack.org/p/fuel_on_centos7
> 
> The above list has two sections: backwards compatible changes that can
> be merged one at a time even if the rest of CentOS7 support isn't
> merged, and backwards incompatible changes that break support for
> CentOS6 and must be merged (and, if needed, reverted) all at once.
> 
> Decision point 1: FFE for CentOS7
> 
> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can
> it be allowed a Feature Freeze Exception? So far, the disruption of
> the Fuel development process implied by the proposed merge plan is
> acceptable, if anything goes wrong and we become unable to have a
> stable ISO with merged CentOS7 support on Monday, December 7, the FFE
> will be revoked.
> 
> Wed, Dec 2: Merge party
> 
> Merge party before 8.0 FF, we should do our best to merge all
> remaining feature commits before end of day (including backwards
> compatible CentOS7 support commits), without breaking the build too
> much.
> 
> At the end of the day we'll start a swarm test over the result of the
> merge party, and we expect QA to analyze and summarize the results by
> 17:00 MSK (6:00 PST) on Thu Dec 3.
> 
> Risk 1: Merge party breaks the build
> 
> If there is a large regression in swarm pass percentage, we won't be
> able to afford a merge freeze which is necessary to merge CentOS7
> support, we'll have to be merging bugfixes until swarm test pass rate
> is back around 70%.
> 
> Risk 2: More features get FFE
> 
> If some essential 8.0 features are not completely merged by end of day
> Wed Dec 2 and are granted FFE, merging the remaining commits can
> interfere with merging CentOS7 support, not just from merge conflicts
> perspective, but also invalidating swarm results and making it
> practically impossible to bisect and attribute potential regressions.
> 
> Thu, Dec 3: Start merge freeze for CentOS7
> 
> Decision point 2: Other FFEs
> 
> In the morning MSK time, we will assess Risk 2 and decide what to do
> with the other FFEs. The options are: integrate remaining commits into
> CentOS7 merge plan, block remaining commits until Monday, revoke
> CentOS7 FFE.
> 
> If the decision is to go ahead with CentOS7 merge, we announce merge
> freeze for all git repositories that go into Fuel ISO, and spend the
> rest of the day rebasing and cleaning up the rest of the CentOS7
> commits to make sure they're all in mergeable state by the end of the
> day. The outcome of this work must be a custom ISO image with all
> remaining commits, with additional requirement that it must not use
> Jenkins job parameters (only patches to fuel-main that change default
> repository paths) to specify all required package repositories. This
> will validate the proposed fuel-main patches and ensure that no
> unmerged package changes are used to produce the ISO.
> 
> Decision point 3: Swarm pass rate
> 
> After swarm results from Wed are available, we will assess the Risk 1.
> If the pass rate regression is significant, CentOS7 FFE is revoked and
> merge freeze is lifted. If regression is acceptable, we proceed with
> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
> 
> Fri, Dec 4: Merge and test CentOS7
> 
> The team will have until 17:00 MSK to produce a non-custom ISO that
> passes BVT and can be run through swarm.
> 
> Sat, Dec 5: Assess CentOS7 swarm and bugfix
> 
> First of all, someone from CI and QA teams should commit to monitoring
> the CentOS7 swarm run and report the results as soon as possible.
> Based on the results (which once again must be available by 17:00
> MSK), we can decide on the final step of the plan.
> 
> Decision point 4: Keep or revert
> 
> If CentOS7 based swarm shows significant regression, we have to spend
> the rest of the weekend including Sunday reverting all CentOS7 commits
> that were merged during merge freeze. Once revert is completed, we
> will lift the merge freeze.
> 
> If the regression is acceptable, we lift the merge freeze straight
> away and proceed with bugfixing as usual. At this point CI team will
> need to update the Fuel ISO used for deployment tests in our CI to
> this same 

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Dmitry Teselkin]

Hello,

Status update for Dec, 4 - we're on track.

As there were no significant issues found during BVT / swarm test runs
on custom CentOS7 ISO it was decided to go with merge party and merge
all CentOS7 related CRs. Our CI was partially updated then and we
started building ISO using production jobs. We've got several green
ISOs which passed BVT. At the moment full set of swarm tests is being
run, we will analyze them tomorrow.

There are sporadic failures of OSTF tests tied with cinder, we're going
to analyze them tomorrow too.

On Tue, 1 Dec 2015 13:48:00 -0800
Dmitry Borodaenko  wrote:

> With bit more details, I hope this covers all the risks and decision
> points now.
> 
> First of all, current list of outstanding commits:
> https://etherpad.openstack.org/p/fuel_on_centos7
> 
> The above list has two sections: backwards compatible changes that can
> be merged one at a time even if the rest of CentOS7 support isn't
> merged, and backwards incompatible changes that break support for
> CentOS6 and must be merged (and, if needed, reverted) all at once.
> 
> Decision point 1: FFE for CentOS7
> 
> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can
> it be allowed a Feature Freeze Exception? So far, the disruption of
> the Fuel development process implied by the proposed merge plan is
> acceptable, if anything goes wrong and we become unable to have a
> stable ISO with merged CentOS7 support on Monday, December 7, the FFE
> will be revoked.
> 
> Wed, Dec 2: Merge party
> 
> Merge party before 8.0 FF, we should do our best to merge all
> remaining feature commits before end of day (including backwards
> compatible CentOS7 support commits), without breaking the build too
> much.
> 
> At the end of the day we'll start a swarm test over the result of the
> merge party, and we expect QA to analyze and summarize the results by
> 17:00 MSK (6:00 PST) on Thu Dec 3.
> 
> Risk 1: Merge party breaks the build
> 
> If there is a large regression in swarm pass percentage, we won't be
> able to afford a merge freeze which is necessary to merge CentOS7
> support, we'll have to be merging bugfixes until swarm test pass rate
> is back around 70%.
> 
> Risk 2: More features get FFE
> 
> If some essential 8.0 features are not completely merged by end of day
> Wed Dec 2 and are granted FFE, merging the remaining commits can
> interfere with merging CentOS7 support, not just from merge conflicts
> perspective, but also invalidating swarm results and making it
> practically impossible to bisect and attribute potential regressions.
> 
> Thu, Dec 3: Start merge freeze for CentOS7
> 
> Decision point 2: Other FFEs
> 
> In the morning MSK time, we will assess Risk 2 and decide what to do
> with the other FFEs. The options are: integrate remaining commits into
> CentOS7 merge plan, block remaining commits until Monday, revoke
> CentOS7 FFE.
> 
> If the decision is to go ahead with CentOS7 merge, we announce merge
> freeze for all git repositories that go into Fuel ISO, and spend the
> rest of the day rebasing and cleaning up the rest of the CentOS7
> commits to make sure they're all in mergeable state by the end of the
> day. The outcome of this work must be a custom ISO image with all
> remaining commits, with additional requirement that it must not use
> Jenkins job parameters (only patches to fuel-main that change default
> repository paths) to specify all required package repositories. This
> will validate the proposed fuel-main patches and ensure that no
> unmerged package changes are used to produce the ISO.
> 
> Decision point 3: Swarm pass rate
> 
> After swarm results from Wed are available, we will assess the Risk 1.
> If the pass rate regression is significant, CentOS7 FFE is revoked and
> merge freeze is lifted. If regression is acceptable, we proceed with
> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
> 
> Fri, Dec 4: Merge and test CentOS7
> 
> The team will have until 17:00 MSK to produce a non-custom ISO that
> passes BVT and can be run through swarm.
> 
> Sat, Dec 5: Assess CentOS7 swarm and bugfix
> 
> First of all, someone from CI and QA teams should commit to monitoring
> the CentOS7 swarm run and report the results as soon as possible.
> Based on the results (which once again must be available by 17:00
> MSK), we can decide on the final step of the plan.
> 
> Decision point 4: Keep or revert
> 
> If CentOS7 based swarm shows significant regression, we have to spend
> the rest of the weekend including Sunday reverting all CentOS7 commits
> that were merged during merge freeze. Once revert is completed, we
> will lift the merge freeze.
> 
> If the regression is acceptable, we lift the merge freeze straight
> away and proceed with bugfixing as usual. At this point CI team will
> need to update the Fuel ISO used for deployment tests in our CI to
> this same ISO.
> 
> One way or the other, we will be able to resume bugfixing on Monday
> morning MSK 

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Mike Scherbakov]

Great progress, thank you for keeping everyone informed.

On Fri, Dec 4, 2015 at 4:43 PM Dmitry Teselkin 
wrote:

> Hello,
>
> Status update for Dec, 4 - we're on track.
>
> As there were no significant issues found during BVT / swarm test runs
> on custom CentOS7 ISO it was decided to go with merge party and merge
> all CentOS7 related CRs. Our CI was partially updated then and we
> started building ISO using production jobs. We've got several green
> ISOs which passed BVT. At the moment full set of swarm tests is being
> run, we will analyze them tomorrow.
>
> There are sporadic failures of OSTF tests tied with cinder, we're going
> to analyze them tomorrow too.
>
> On Tue, 1 Dec 2015 13:48:00 -0800
> Dmitry Borodaenko  wrote:
>
> > With bit more details, I hope this covers all the risks and decision
> > points now.
> >
> > First of all, current list of outstanding commits:
> > https://etherpad.openstack.org/p/fuel_on_centos7
> >
> > The above list has two sections: backwards compatible changes that can
> > be merged one at a time even if the rest of CentOS7 support isn't
> > merged, and backwards incompatible changes that break support for
> > CentOS6 and must be merged (and, if needed, reverted) all at once.
> >
> > Decision point 1: FFE for CentOS7
> >
> > CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can
> > it be allowed a Feature Freeze Exception? So far, the disruption of
> > the Fuel development process implied by the proposed merge plan is
> > acceptable, if anything goes wrong and we become unable to have a
> > stable ISO with merged CentOS7 support on Monday, December 7, the FFE
> > will be revoked.
> >
> > Wed, Dec 2: Merge party
> >
> > Merge party before 8.0 FF, we should do our best to merge all
> > remaining feature commits before end of day (including backwards
> > compatible CentOS7 support commits), without breaking the build too
> > much.
> >
> > At the end of the day we'll start a swarm test over the result of the
> > merge party, and we expect QA to analyze and summarize the results by
> > 17:00 MSK (6:00 PST) on Thu Dec 3.
> >
> > Risk 1: Merge party breaks the build
> >
> > If there is a large regression in swarm pass percentage, we won't be
> > able to afford a merge freeze which is necessary to merge CentOS7
> > support, we'll have to be merging bugfixes until swarm test pass rate
> > is back around 70%.
> >
> > Risk 2: More features get FFE
> >
> > If some essential 8.0 features are not completely merged by end of day
> > Wed Dec 2 and are granted FFE, merging the remaining commits can
> > interfere with merging CentOS7 support, not just from merge conflicts
> > perspective, but also invalidating swarm results and making it
> > practically impossible to bisect and attribute potential regressions.
> >
> > Thu, Dec 3: Start merge freeze for CentOS7
> >
> > Decision point 2: Other FFEs
> >
> > In the morning MSK time, we will assess Risk 2 and decide what to do
> > with the other FFEs. The options are: integrate remaining commits into
> > CentOS7 merge plan, block remaining commits until Monday, revoke
> > CentOS7 FFE.
> >
> > If the decision is to go ahead with CentOS7 merge, we announce merge
> > freeze for all git repositories that go into Fuel ISO, and spend the
> > rest of the day rebasing and cleaning up the rest of the CentOS7
> > commits to make sure they're all in mergeable state by the end of the
> > day. The outcome of this work must be a custom ISO image with all
> > remaining commits, with additional requirement that it must not use
> > Jenkins job parameters (only patches to fuel-main that change default
> > repository paths) to specify all required package repositories. This
> > will validate the proposed fuel-main patches and ensure that no
> > unmerged package changes are used to produce the ISO.
> >
> > Decision point 3: Swarm pass rate
> >
> > After swarm results from Wed are available, we will assess the Risk 1.
> > If the pass rate regression is significant, CentOS7 FFE is revoked and
> > merge freeze is lifted. If regression is acceptable, we proceed with
> > merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
> >
> > Fri, Dec 4: Merge and test CentOS7
> >
> > The team will have until 17:00 MSK to produce a non-custom ISO that
> > passes BVT and can be run through swarm.
> >
> > Sat, Dec 5: Assess CentOS7 swarm and bugfix
> >
> > First of all, someone from CI and QA teams should commit to monitoring
> > the CentOS7 swarm run and report the results as soon as possible.
> > Based on the results (which once again must be available by 17:00
> > MSK), we can decide on the final step of the plan.
> >
> > Decision point 4: Keep or revert
> >
> > If CentOS7 based swarm shows significant regression, we have to spend
> > the rest of the weekend including Sunday reverting all CentOS7 commits
> > that were merged during merge freeze. Once revert is completed, we
> > will lift the merge 

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Dmitry Teselkin]

Hello,

Status update for Dec, 3 - we're on track.

There were issues with passing swarm test in the beginning of the day
caused by various reasons, not all of them related to CentOS7 itself.
We've fixed all issues we've found, at EOD we've got several
successful ISOs built from master (with our patches applied). Two
latest ISOs are being tested (BVT / swarm), the results are expected in
3-5 hours. We don't expect them to fail though.

There are 19 open CRs tagged with 'centos7-master-node' [0], and 2 were
merged [1].

[0]
https://review.openstack.org/#/q/status:open+topic:centos7-master-node,n,z
[1]
https://review.openstack.org/#/q/status:merged+topic:centos7-master-node,n,z

On Tue, 1 Dec 2015 13:48:00 -0800
Dmitry Borodaenko  wrote:

> With bit more details, I hope this covers all the risks and decision
> points now.
> 
> First of all, current list of outstanding commits:
> https://etherpad.openstack.org/p/fuel_on_centos7
> 
> The above list has two sections: backwards compatible changes that can
> be merged one at a time even if the rest of CentOS7 support isn't
> merged, and backwards incompatible changes that break support for
> CentOS6 and must be merged (and, if needed, reverted) all at once.
> 
> Decision point 1: FFE for CentOS7
> 
> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can
> it be allowed a Feature Freeze Exception? So far, the disruption of
> the Fuel development process implied by the proposed merge plan is
> acceptable, if anything goes wrong and we become unable to have a
> stable ISO with merged CentOS7 support on Monday, December 7, the FFE
> will be revoked.
> 
> Wed, Dec 2: Merge party
> 
> Merge party before 8.0 FF, we should do our best to merge all
> remaining feature commits before end of day (including backwards
> compatible CentOS7 support commits), without breaking the build too
> much.
> 
> At the end of the day we'll start a swarm test over the result of the
> merge party, and we expect QA to analyze and summarize the results by
> 17:00 MSK (6:00 PST) on Thu Dec 3.
> 
> Risk 1: Merge party breaks the build
> 
> If there is a large regression in swarm pass percentage, we won't be
> able to afford a merge freeze which is necessary to merge CentOS7
> support, we'll have to be merging bugfixes until swarm test pass rate
> is back around 70%.
> 
> Risk 2: More features get FFE
> 
> If some essential 8.0 features are not completely merged by end of day
> Wed Dec 2 and are granted FFE, merging the remaining commits can
> interfere with merging CentOS7 support, not just from merge conflicts
> perspective, but also invalidating swarm results and making it
> practically impossible to bisect and attribute potential regressions.
> 
> Thu, Dec 3: Start merge freeze for CentOS7
> 
> Decision point 2: Other FFEs
> 
> In the morning MSK time, we will assess Risk 2 and decide what to do
> with the other FFEs. The options are: integrate remaining commits into
> CentOS7 merge plan, block remaining commits until Monday, revoke
> CentOS7 FFE.
> 
> If the decision is to go ahead with CentOS7 merge, we announce merge
> freeze for all git repositories that go into Fuel ISO, and spend the
> rest of the day rebasing and cleaning up the rest of the CentOS7
> commits to make sure they're all in mergeable state by the end of the
> day. The outcome of this work must be a custom ISO image with all
> remaining commits, with additional requirement that it must not use
> Jenkins job parameters (only patches to fuel-main that change default
> repository paths) to specify all required package repositories. This
> will validate the proposed fuel-main patches and ensure that no
> unmerged package changes are used to produce the ISO.
> 
> Decision point 3: Swarm pass rate
> 
> After swarm results from Wed are available, we will assess the Risk 1.
> If the pass rate regression is significant, CentOS7 FFE is revoked and
> merge freeze is lifted. If regression is acceptable, we proceed with
> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
> 
> Fri, Dec 4: Merge and test CentOS7
> 
> The team will have until 17:00 MSK to produce a non-custom ISO that
> passes BVT and can be run through swarm.
> 
> Sat, Dec 5: Assess CentOS7 swarm and bugfix
> 
> First of all, someone from CI and QA teams should commit to monitoring
> the CentOS7 swarm run and report the results as soon as possible.
> Based on the results (which once again must be available by 17:00
> MSK), we can decide on the final step of the plan.
> 
> Decision point 4: Keep or revert
> 
> If CentOS7 based swarm shows significant regression, we have to spend
> the rest of the weekend including Sunday reverting all CentOS7 commits
> that were merged during merge freeze. Once revert is completed, we
> will lift the merge freeze.
> 
> If the regression is acceptable, we lift the merge freeze straight
> away and proceed with bugfixing as usual. At this point CI team will
> need to update the Fuel 

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Andrew Maksimov]

Thank you Dmitry for very detailed plan and risks assessment.
Do we want to run swarm against custom iso with centos7 on Thu evening to
measure level of regression? I remember that we were considering this
approach.

Regards,
Andrey Maximov


On Wed, Dec 2, 2015 at 12:48 AM, Dmitry Borodaenko  wrote:

> With bit more details, I hope this covers all the risks and decision
> points now.
>
> First of all, current list of outstanding commits:
> https://etherpad.openstack.org/p/fuel_on_centos7
>
> The above list has two sections: backwards compatible changes that can
> be merged one at a time even if the rest of CentOS7 support isn't
> merged, and backwards incompatible changes that break support for
> CentOS6 and must be merged (and, if needed, reverted) all at once.
>
> Decision point 1: FFE for CentOS7
>
> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can it
> be allowed a Feature Freeze Exception? So far, the disruption of the
> Fuel development process implied by the proposed merge plan is
> acceptable, if anything goes wrong and we become unable to have a stable
> ISO with merged CentOS7 support on Monday, December 7, the FFE will be
> revoked.
>
> Wed, Dec 2: Merge party
>
> Merge party before 8.0 FF, we should do our best to merge all remaining
> feature commits before end of day (including backwards compatible
> CentOS7 support commits), without breaking the build too much.
>
> At the end of the day we'll start a swarm test over the result of the
> merge party, and we expect QA to analyze and summarize the results by
> 17:00 MSK (6:00 PST) on Thu Dec 3.
>
> Risk 1: Merge party breaks the build
>
> If there is a large regression in swarm pass percentage, we won't be
> able to afford a merge freeze which is necessary to merge CentOS7
> support, we'll have to be merging bugfixes until swarm test pass rate is
> back around 70%.
>
> Risk 2: More features get FFE
>
> If some essential 8.0 features are not completely merged by end of day
> Wed Dec 2 and are granted FFE, merging the remaining commits can
> interfere with merging CentOS7 support, not just from merge conflicts
> perspective, but also invalidating swarm results and making it
> practically impossible to bisect and attribute potential regressions.
>
> Thu, Dec 3: Start merge freeze for CentOS7
>
> Decision point 2: Other FFEs
>
> In the morning MSK time, we will assess Risk 2 and decide what to do
> with the other FFEs. The options are: integrate remaining commits into
> CentOS7 merge plan, block remaining commits until Monday, revoke CentOS7
> FFE.
>
> If the decision is to go ahead with CentOS7 merge, we announce merge
> freeze for all git repositories that go into Fuel ISO, and spend the
> rest of the day rebasing and cleaning up the rest of the CentOS7 commits
> to make sure they're all in mergeable state by the end of the day. The
> outcome of this work must be a custom ISO image with all remaining
> commits, with additional requirement that it must not use Jenkins job
> parameters (only patches to fuel-main that change default repository
> paths) to specify all required package repositories. This will validate
> the proposed fuel-main patches and ensure that no unmerged package
> changes are used to produce the ISO.
>
> Decision point 3: Swarm pass rate
>
> After swarm results from Wed are available, we will assess the Risk 1.
> If the pass rate regression is significant, CentOS7 FFE is revoked and
> merge freeze is lifted. If regression is acceptable, we proceed with
> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
>
> Fri, Dec 4: Merge and test CentOS7
>
> The team will have until 17:00 MSK to produce a non-custom ISO that
> passes BVT and can be run through swarm.
>
> Sat, Dec 5: Assess CentOS7 swarm and bugfix
>
> First of all, someone from CI and QA teams should commit to monitoring
> the CentOS7 swarm run and report the results as soon as possible. Based
> on the results (which once again must be available by 17:00 MSK), we can
> decide on the final step of the plan.
>
> Decision point 4: Keep or revert
>
> If CentOS7 based swarm shows significant regression, we have to spend
> the rest of the weekend including Sunday reverting all CentOS7 commits
> that were merged during merge freeze. Once revert is completed, we will
> lift the merge freeze.
>
> If the regression is acceptable, we lift the merge freeze straight away
> and proceed with bugfixing as usual. At this point CI team will need to
> update the Fuel ISO used for deployment tests in our CI to this same
> ISO.
>
> One way or the other, we will be able to resume bugfixing on Monday
> morning MSK time, and will have lost 2 business days (Thu-Fri) during
> which we won't be able to merge bugfixes. In addition to that, someone
> from QA and everyone from CentOS7 support team has to work on Saturday,
> and someone from CI will have to work a few hours on Sunday.
>
> --
> Dmitry Borodaenko
>
>
> On Tue, Dec 01, 2015 

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Igor Marnat]

Dmitry,
thank you!

I confirm that the plan looks good for our team, we'll follow it.

Regards,
Igor Marnat

On Wed, Dec 2, 2015 at 9:39 PM, Andrew Maksimov 
wrote:

> Thank you Dmitry for very detailed plan and risks assessment.
> Do we want to run swarm against custom iso with centos7 on Thu evening to
> measure level of regression? I remember that we were considering this
> approach.
>
> Regards,
> Andrey Maximov
>
>
> On Wed, Dec 2, 2015 at 12:48 AM, Dmitry Borodaenko <
> dborodae...@mirantis.com> wrote:
>
>> With bit more details, I hope this covers all the risks and decision
>> points now.
>>
>> First of all, current list of outstanding commits:
>> https://etherpad.openstack.org/p/fuel_on_centos7
>>
>> The above list has two sections: backwards compatible changes that can
>> be merged one at a time even if the rest of CentOS7 support isn't
>> merged, and backwards incompatible changes that break support for
>> CentOS6 and must be merged (and, if needed, reverted) all at once.
>>
>> Decision point 1: FFE for CentOS7
>>
>> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can it
>> be allowed a Feature Freeze Exception? So far, the disruption of the
>> Fuel development process implied by the proposed merge plan is
>> acceptable, if anything goes wrong and we become unable to have a stable
>> ISO with merged CentOS7 support on Monday, December 7, the FFE will be
>> revoked.
>>
>> Wed, Dec 2: Merge party
>>
>> Merge party before 8.0 FF, we should do our best to merge all remaining
>> feature commits before end of day (including backwards compatible
>> CentOS7 support commits), without breaking the build too much.
>>
>> At the end of the day we'll start a swarm test over the result of the
>> merge party, and we expect QA to analyze and summarize the results by
>> 17:00 MSK (6:00 PST) on Thu Dec 3.
>>
>> Risk 1: Merge party breaks the build
>>
>> If there is a large regression in swarm pass percentage, we won't be
>> able to afford a merge freeze which is necessary to merge CentOS7
>> support, we'll have to be merging bugfixes until swarm test pass rate is
>> back around 70%.
>>
>> Risk 2: More features get FFE
>>
>> If some essential 8.0 features are not completely merged by end of day
>> Wed Dec 2 and are granted FFE, merging the remaining commits can
>> interfere with merging CentOS7 support, not just from merge conflicts
>> perspective, but also invalidating swarm results and making it
>> practically impossible to bisect and attribute potential regressions.
>>
>> Thu, Dec 3: Start merge freeze for CentOS7
>>
>> Decision point 2: Other FFEs
>>
>> In the morning MSK time, we will assess Risk 2 and decide what to do
>> with the other FFEs. The options are: integrate remaining commits into
>> CentOS7 merge plan, block remaining commits until Monday, revoke CentOS7
>> FFE.
>>
>> If the decision is to go ahead with CentOS7 merge, we announce merge
>> freeze for all git repositories that go into Fuel ISO, and spend the
>> rest of the day rebasing and cleaning up the rest of the CentOS7 commits
>> to make sure they're all in mergeable state by the end of the day. The
>> outcome of this work must be a custom ISO image with all remaining
>> commits, with additional requirement that it must not use Jenkins job
>> parameters (only patches to fuel-main that change default repository
>> paths) to specify all required package repositories. This will validate
>> the proposed fuel-main patches and ensure that no unmerged package
>> changes are used to produce the ISO.
>>
>> Decision point 3: Swarm pass rate
>>
>> After swarm results from Wed are available, we will assess the Risk 1.
>> If the pass rate regression is significant, CentOS7 FFE is revoked and
>> merge freeze is lifted. If regression is acceptable, we proceed with
>> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
>>
>> Fri, Dec 4: Merge and test CentOS7
>>
>> The team will have until 17:00 MSK to produce a non-custom ISO that
>> passes BVT and can be run through swarm.
>>
>> Sat, Dec 5: Assess CentOS7 swarm and bugfix
>>
>> First of all, someone from CI and QA teams should commit to monitoring
>> the CentOS7 swarm run and report the results as soon as possible. Based
>> on the results (which once again must be available by 17:00 MSK), we can
>> decide on the final step of the plan.
>>
>> Decision point 4: Keep or revert
>>
>> If CentOS7 based swarm shows significant regression, we have to spend
>> the rest of the weekend including Sunday reverting all CentOS7 commits
>> that were merged during merge freeze. Once revert is completed, we will
>> lift the merge freeze.
>>
>> If the regression is acceptable, we lift the merge freeze straight away
>> and proceed with bugfixing as usual. At this point CI team will need to
>> update the Fuel ISO used for deployment tests in our CI to this same
>> ISO.
>>
>> One way or the other, we will be able to resume bugfixing on Monday
>> morning MSK time, and will 

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Dmitry Teselkin]

Hello,

Here is our status (Dec, 2).

We are on track now.

This (Dec, 2) morning we've got results of 6 tests (swarm) - 3
succeeded, 3 failed. Failures were caused issues in ISO and QA scripts,
and were fixed during the day. In the end of the day (Dec, 2) we've got
another green custom ISO, which now runs BVT and the same 6 swarm tests.

There were several merges during the day, we rebased our patches
without significant issues.

We have around 17 patch sets in 'open' state:
* compatible - 5 (can be merged any time)
* not compatible - 12 (can be merged only if we merge CentOS7)

On Tue, 1 Dec 2015 13:48:00 -0800
Dmitry Borodaenko  wrote:

> With bit more details, I hope this covers all the risks and decision
> points now.
> 
> First of all, current list of outstanding commits:
> https://etherpad.openstack.org/p/fuel_on_centos7
> 
> The above list has two sections: backwards compatible changes that can
> be merged one at a time even if the rest of CentOS7 support isn't
> merged, and backwards incompatible changes that break support for
> CentOS6 and must be merged (and, if needed, reverted) all at once.
> 
> Decision point 1: FFE for CentOS7
> 
> CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can
> it be allowed a Feature Freeze Exception? So far, the disruption of
> the Fuel development process implied by the proposed merge plan is
> acceptable, if anything goes wrong and we become unable to have a
> stable ISO with merged CentOS7 support on Monday, December 7, the FFE
> will be revoked.
> 
> Wed, Dec 2: Merge party
> 
> Merge party before 8.0 FF, we should do our best to merge all
> remaining feature commits before end of day (including backwards
> compatible CentOS7 support commits), without breaking the build too
> much.
> 
> At the end of the day we'll start a swarm test over the result of the
> merge party, and we expect QA to analyze and summarize the results by
> 17:00 MSK (6:00 PST) on Thu Dec 3.
> 
> Risk 1: Merge party breaks the build
> 
> If there is a large regression in swarm pass percentage, we won't be
> able to afford a merge freeze which is necessary to merge CentOS7
> support, we'll have to be merging bugfixes until swarm test pass rate
> is back around 70%.
> 
> Risk 2: More features get FFE
> 
> If some essential 8.0 features are not completely merged by end of day
> Wed Dec 2 and are granted FFE, merging the remaining commits can
> interfere with merging CentOS7 support, not just from merge conflicts
> perspective, but also invalidating swarm results and making it
> practically impossible to bisect and attribute potential regressions.
> 
> Thu, Dec 3: Start merge freeze for CentOS7
> 
> Decision point 2: Other FFEs
> 
> In the morning MSK time, we will assess Risk 2 and decide what to do
> with the other FFEs. The options are: integrate remaining commits into
> CentOS7 merge plan, block remaining commits until Monday, revoke
> CentOS7 FFE.
> 
> If the decision is to go ahead with CentOS7 merge, we announce merge
> freeze for all git repositories that go into Fuel ISO, and spend the
> rest of the day rebasing and cleaning up the rest of the CentOS7
> commits to make sure they're all in mergeable state by the end of the
> day. The outcome of this work must be a custom ISO image with all
> remaining commits, with additional requirement that it must not use
> Jenkins job parameters (only patches to fuel-main that change default
> repository paths) to specify all required package repositories. This
> will validate the proposed fuel-main patches and ensure that no
> unmerged package changes are used to produce the ISO.
> 
> Decision point 3: Swarm pass rate
> 
> After swarm results from Wed are available, we will assess the Risk 1.
> If the pass rate regression is significant, CentOS7 FFE is revoked and
> merge freeze is lifted. If regression is acceptable, we proceed with
> merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.
> 
> Fri, Dec 4: Merge and test CentOS7
> 
> The team will have until 17:00 MSK to produce a non-custom ISO that
> passes BVT and can be run through swarm.
> 
> Sat, Dec 5: Assess CentOS7 swarm and bugfix
> 
> First of all, someone from CI and QA teams should commit to monitoring
> the CentOS7 swarm run and report the results as soon as possible.
> Based on the results (which once again must be available by 17:00
> MSK), we can decide on the final step of the plan.
> 
> Decision point 4: Keep or revert
> 
> If CentOS7 based swarm shows significant regression, we have to spend
> the rest of the weekend including Sunday reverting all CentOS7 commits
> that were merged during merge freeze. Once revert is completed, we
> will lift the merge freeze.
> 
> If the regression is acceptable, we lift the merge freeze straight
> away and proceed with bugfixing as usual. At this point CI team will
> need to update the Fuel ISO used for deployment tests in our CI to
> this same ISO.
> 
> One way or the other, we will be 

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Dmitry Borodaenko]

With bit more details, I hope this covers all the risks and decision
points now.

First of all, current list of outstanding commits:
https://etherpad.openstack.org/p/fuel_on_centos7

The above list has two sections: backwards compatible changes that can
be merged one at a time even if the rest of CentOS7 support isn't
merged, and backwards incompatible changes that break support for
CentOS6 and must be merged (and, if needed, reverted) all at once.

Decision point 1: FFE for CentOS7

CentOS7 support cannot be fully merged on Dec 2, so it misses FF. Can it
be allowed a Feature Freeze Exception? So far, the disruption of the
Fuel development process implied by the proposed merge plan is
acceptable, if anything goes wrong and we become unable to have a stable
ISO with merged CentOS7 support on Monday, December 7, the FFE will be
revoked.

Wed, Dec 2: Merge party

Merge party before 8.0 FF, we should do our best to merge all remaining
feature commits before end of day (including backwards compatible
CentOS7 support commits), without breaking the build too much.

At the end of the day we'll start a swarm test over the result of the
merge party, and we expect QA to analyze and summarize the results by
17:00 MSK (6:00 PST) on Thu Dec 3.

Risk 1: Merge party breaks the build

If there is a large regression in swarm pass percentage, we won't be
able to afford a merge freeze which is necessary to merge CentOS7
support, we'll have to be merging bugfixes until swarm test pass rate is
back around 70%.

Risk 2: More features get FFE

If some essential 8.0 features are not completely merged by end of day
Wed Dec 2 and are granted FFE, merging the remaining commits can
interfere with merging CentOS7 support, not just from merge conflicts
perspective, but also invalidating swarm results and making it
practically impossible to bisect and attribute potential regressions.

Thu, Dec 3: Start merge freeze for CentOS7

Decision point 2: Other FFEs

In the morning MSK time, we will assess Risk 2 and decide what to do
with the other FFEs. The options are: integrate remaining commits into
CentOS7 merge plan, block remaining commits until Monday, revoke CentOS7
FFE.

If the decision is to go ahead with CentOS7 merge, we announce merge
freeze for all git repositories that go into Fuel ISO, and spend the
rest of the day rebasing and cleaning up the rest of the CentOS7 commits
to make sure they're all in mergeable state by the end of the day. The
outcome of this work must be a custom ISO image with all remaining
commits, with additional requirement that it must not use Jenkins job
parameters (only patches to fuel-main that change default repository
paths) to specify all required package repositories. This will validate
the proposed fuel-main patches and ensure that no unmerged package
changes are used to produce the ISO.

Decision point 3: Swarm pass rate

After swarm results from Wed are available, we will assess the Risk 1.
If the pass rate regression is significant, CentOS7 FFE is revoked and
merge freeze is lifted. If regression is acceptable, we proceed with
merging remaining CentOS7 commmits through Thu Dec 3 and Fri Dec 4.

Fri, Dec 4: Merge and test CentOS7

The team will have until 17:00 MSK to produce a non-custom ISO that
passes BVT and can be run through swarm.

Sat, Dec 5: Assess CentOS7 swarm and bugfix

First of all, someone from CI and QA teams should commit to monitoring
the CentOS7 swarm run and report the results as soon as possible. Based
on the results (which once again must be available by 17:00 MSK), we can
decide on the final step of the plan.

Decision point 4: Keep or revert

If CentOS7 based swarm shows significant regression, we have to spend
the rest of the weekend including Sunday reverting all CentOS7 commits
that were merged during merge freeze. Once revert is completed, we will
lift the merge freeze.

If the regression is acceptable, we lift the merge freeze straight away
and proceed with bugfixing as usual. At this point CI team will need to
update the Fuel ISO used for deployment tests in our CI to this same
ISO.

One way or the other, we will be able to resume bugfixing on Monday
morning MSK time, and will have lost 2 business days (Thu-Fri) during
which we won't be able to merge bugfixes. In addition to that, someone
from QA and everyone from CentOS7 support team has to work on Saturday,
and someone from CI will have to work a few hours on Sunday.

-- 
Dmitry Borodaenko


On Tue, Dec 01, 2015 at 05:58:42PM +0300, Dmitry Teselkin wrote:
> Hello,
> 
> We're almost got green BVT on custom CentOS7 ISO and it seems that it's
> the time to discuss the plan how this feature could be merged.
> 
> This is not the only one feature that is in a queue. Unfortunately,
> almost any other feature will be broken if merged after CentOS7, so it
> was decided to merge our changes last.
> 
> This is not an official announcement, rather a notification letter to
> start a discussion and find any objections.
> 
> So the 

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Aleksandr Didenko]

Hi,

in your plan you need to have an item about updating ISO on fuel-library CI
gates.

Regards,
Alex

On Tue, Dec 1, 2015 at 4:11 PM, Sergii Golovatiuk 
wrote:

> Hi,
>
> On Tue, Dec 1, 2015 at 3:58 PM, Dmitry Teselkin 
> wrote:
>
>> Hello,
>>
>> We're almost got green BVT on custom CentOS7 ISO and it seems that it's
>> the time to discuss the plan how this feature could be merged.
>>
>> This is not the only one feature that is in a queue. Unfortunately,
>> almost any other feature will be broken if merged after CentOS7, so it
>> was decided to merge our changes last.
>>
>> This is not an official announcement, rather a notification letter to
>> start a discussion and find any objections.
>>
>> So the plan is:
>>
>> * merge all features that are going to be merged before Thusday, Dec 3
>>
>
> As far as I know some features won't be ready. They will require FFE or
> should be moved to 9.0
>
>
>> * call for merge freeze starting at Dec 3, due Dec 7
>> * rebase all CentOS7-related pathsets and resolve any conflicts with
>>   merged code (Dec 3)
>> * build custom ISO, pass BVT (and other tests) (Dec 3)
>> * merge all CentOS7-related patchsets at once (Dec 4)
>>
>
> I guess CI environment should be rebuilt
>
> * build an ISO and pass BVT again (Dec 4)
>> * run additional test during weekend (Dec 5, 6) to be sure that ISO
>>   good enough
>
>
>> According to this plan on Monday, Dec 7 we should either get CentOS7
>> based ISO, or revert all incompatible changes.
>>
>> --
>> Thanks,
>> Dmitry Teselkin
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] CentOS7 Merging Plan

[Sergii Golovatiuk]

Hi,

On Tue, Dec 1, 2015 at 3:58 PM, Dmitry Teselkin 
wrote:

> Hello,
>
> We're almost got green BVT on custom CentOS7 ISO and it seems that it's
> the time to discuss the plan how this feature could be merged.
>
> This is not the only one feature that is in a queue. Unfortunately,
> almost any other feature will be broken if merged after CentOS7, so it
> was decided to merge our changes last.
>
> This is not an official announcement, rather a notification letter to
> start a discussion and find any objections.
>
> So the plan is:
>
> * merge all features that are going to be merged before Thusday, Dec 3
>

As far as I know some features won't be ready. They will require FFE or
should be moved to 9.0


> * call for merge freeze starting at Dec 3, due Dec 7
> * rebase all CentOS7-related pathsets and resolve any conflicts with
>   merged code (Dec 3)
> * build custom ISO, pass BVT (and other tests) (Dec 3)
> * merge all CentOS7-related patchsets at once (Dec 4)
>

I guess CI environment should be rebuilt

* build an ISO and pass BVT again (Dec 4)
> * run additional test during weekend (Dec 5, 6) to be sure that ISO
>   good enough


> According to this plan on Monday, Dec 7 we should either get CentOS7
> based ISO, or revert all incompatible changes.
>
> --
> Thanks,
> Dmitry Teselkin
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] CentOS7 Merging Plan

[Dmitry Teselkin]

Hello,

We're almost got green BVT on custom CentOS7 ISO and it seems that it's
the time to discuss the plan how this feature could be merged.

This is not the only one feature that is in a queue. Unfortunately,
almost any other feature will be broken if merged after CentOS7, so it
was decided to merge our changes last.

This is not an official announcement, rather a notification letter to
start a discussion and find any objections.

So the plan is:

* merge all features that are going to be merged before Thusday, Dec 3
* call for merge freeze starting at Dec 3, due Dec 7
* rebase all CentOS7-related pathsets and resolve any conflicts with
  merged code (Dec 3)
* build custom ISO, pass BVT (and other tests) (Dec 3)
* merge all CentOS7-related patchsets at once (Dec 4)
* build an ISO and pass BVT again (Dec 4)
* run additional test during weekend (Dec 5, 6) to be sure that ISO
  good enough

According to this plan on Monday, Dec 7 we should either get CentOS7
based ISO, or revert all incompatible changes.

-- 
Thanks,
Dmitry Teselkin


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev