Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering

[Emilien Macchi]

On Mon, Mar 13, 2017 at 12:29 PM, Alan Pevec  wrote:
> 2017-03-09 14:58 GMT+01:00 Jeremy Stanley :
>> In the past we addressed this by automatically merging the release
>> tag back into master, but we stopped doing that a cycle ago because
>> it complicated release note generation.
>
> Also this was including RC >= 2 and final tags so as soon as the first
> stable maintenance version was released, master was again lower
> version.

topic sounds staled.
Alan,  do we have an ETA on the RDO workaround?

Thanks,

> Cheers,
> Alan
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering

[Alan Pevec]

2017-03-13 15:49 GMT+01:00 Doug Hellmann :
...
> We test this upgrade scenario in the upstream CI, too. The difference
> is that grenade can tell pip "install exactly the version I am
> pointing to in this directory on disk," rather than relying on
> version numbers to notice that an upgrade is needed (or should be
> avoided, as the case may be).
>
> Is it possible to do that with system packages in some way, too, without
> pinning package versions in puppet or tripleo?

That would not test the regular update flow, we'd have to use
workaround for the period before milestone 1
and run explicit yum downgrade with the explicit package list to force
lower version packages to install.
That means real upgrades would not be tested until this kludge is
removed after first milestone is released.
Instead, I'll look at adding automatic bump in RDO Trunk builder
before milestone1 i.e. when it detects master version is lower than
stable branches.

Cheers,
Alan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering

[Alan Pevec]

2017-03-09 14:58 GMT+01:00 Jeremy Stanley :
> In the past we addressed this by automatically merging the release
> tag back into master, but we stopped doing that a cycle ago because
> it complicated release note generation.

Also this was including RC >= 2 and final tags so as soon as the first
stable maintenance version was released, master was again lower
version.

Cheers,
Alan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering

[Doug Hellmann]

Excerpts from Thierry Carrez's message of 2017-03-13 15:13:18 +0100:
> Jeremy Stanley wrote:
> > On 2017-03-09 12:12:01 +0100 (+0100), Alan Pevec wrote:
> >> 2017-03-09 10:41 GMT+01:00 Alfredo Moralejo Alonso :
> >>> On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy  wrote:
>  https://bugs.launchpad.net/tripleo/+bug/1669462
> 
>  I'm not clear on the best path forward at this point, but the simplest 
>  one
>  suggested so far is to simply tag a new pre-milestone/alpha release for 
>  all
>  master branches, which will enable testing upgrades to master.
> >>
> >> That would be indeed simplest and cleanest solution but it has been
> >> rejected in the past, IIUC with reasoning that projects are not sure
> >> what will be their next version. IMHO that should not be the case if

There is the problem that some deliverables don't know their next
version in advance. There is also the problem that the vast majority
of our deliverables do not use pre-release versions (like alphas)
at all.

> >> the project has decided to have a stable branch and follows semver and
> >> stable branch policy: in that case project will keep X.Y frozen on
> >> stable so master should be bumped to at least X.Y+1 immediately after
> >> branching stable.
> > [...]
> > 
> > In the past we addressed this by automatically merging the release
> > tag back into master, but we stopped doing that a cycle ago because
> > it complicated release note generation.
> 
> Right, there is a bit of history here.
> 
> One option is to tag the first commit on master after the stable branch
> point, to reduce (but not eliminate) the window where stable > master,
> but that relied on a lot of discipline and checks (since it's hard to
> automate).

Yes, I think the level of coordination involved in making this work for
all projects would be challenging to accomplish with the small release
team. We have the release task list down to a fairly small minimum right
now, and I don't relish the idea of adding more complexity if we can
avoid it.

> We preferred the option to merge tags back on master, which was not
> perfect (creates overlap in versions automatically generated) but at
> least the dev version was always > last released version.
> 
> However, the merged tags history prevented reno from determining
> correctly where it was and generate appropriate release notes. So we

The problem was with a tag appearing on multiple branches, and basically
being inserted into the master branch at a random spot, it wasn't
possible to actually tell which series a version belonged to and which
notes should go with which versions.

> dropped it... Robert Collins argued that users should consume from a
> single channel anyway: either follow master or follow a stable branch,
> and not combine both. Here the problem is that you test upgrades from
> stable/$foo to master, which is basically something the model does not
> support.

We test this upgrade scenario in the upstream CI, too. The difference
is that grenade can tell pip "install exactly the version I am
pointing to in this directory on disk," rather than relying on
version numbers to notice that an upgrade is needed (or should be
avoided, as the case may be).

Is it possible to do that with system packages in some way, too, without
pinning package versions in puppet or tripleo?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering

[Thierry Carrez]

Jeremy Stanley wrote:
> On 2017-03-09 12:12:01 +0100 (+0100), Alan Pevec wrote:
>> 2017-03-09 10:41 GMT+01:00 Alfredo Moralejo Alonso :
>>> On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy  wrote:
 https://bugs.launchpad.net/tripleo/+bug/1669462

 I'm not clear on the best path forward at this point, but the simplest one
 suggested so far is to simply tag a new pre-milestone/alpha release for all
 master branches, which will enable testing upgrades to master.
>>
>> That would be indeed simplest and cleanest solution but it has been
>> rejected in the past, IIUC with reasoning that projects are not sure
>> what will be their next version. IMHO that should not be the case if
>> the project has decided to have a stable branch and follows semver and
>> stable branch policy: in that case project will keep X.Y frozen on
>> stable so master should be bumped to at least X.Y+1 immediately after
>> branching stable.
> [...]
> 
> In the past we addressed this by automatically merging the release
> tag back into master, but we stopped doing that a cycle ago because
> it complicated release note generation.

Right, there is a bit of history here.

One option is to tag the first commit on master after the stable branch
point, to reduce (but not eliminate) the window where stable > master,
but that relied on a lot of discipline and checks (since it's hard to
automate).

We preferred the option to merge tags back on master, which was not
perfect (creates overlap in versions automatically generated) but at
least the dev version was always > last released version.

However, the merged tags history prevented reno from determining
correctly where it was and generate appropriate release notes. So we
dropped it... Robert Collins argued that users should consume from a
single channel anyway: either follow master or follow a stable branch,
and not combine both. Here the problem is that you test upgrades from
stable/$foo to master, which is basically something the model does not
support.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering

[Jeremy Stanley]

On 2017-03-09 12:12:01 +0100 (+0100), Alan Pevec wrote:
> 2017-03-09 10:41 GMT+01:00 Alfredo Moralejo Alonso :
> > On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy  wrote:
> >> https://bugs.launchpad.net/tripleo/+bug/1669462
> >>
> >> I'm not clear on the best path forward at this point, but the simplest one
> >> suggested so far is to simply tag a new pre-milestone/alpha release for all
> >> master branches, which will enable testing upgrades to master.
> 
> That would be indeed simplest and cleanest solution but it has been
> rejected in the past, IIUC with reasoning that projects are not sure
> what will be their next version. IMHO that should not be the case if
> the project has decided to have a stable branch and follows semver and
> stable branch policy: in that case project will keep X.Y frozen on
> stable so master should be bumped to at least X.Y+1 immediately after
> branching stable.
[...]

In the past we addressed this by automatically merging the release
tag back into master, but we stopped doing that a cycle ago because
it complicated release note generation.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering

[Alan Pevec]

2017-03-09 10:41 GMT+01:00 Alfredo Moralejo Alonso :
> On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy  wrote:
>> https://bugs.launchpad.net/tripleo/+bug/1669462
>>
>> I'm not clear on the best path forward at this point, but the simplest one
>> suggested so far is to simply tag a new pre-milestone/alpha release for all
>> master branches, which will enable testing upgrades to master.

That would be indeed simplest and cleanest solution but it has been
rejected in the past, IIUC with reasoning that projects are not sure
what will be their next version. IMHO that should not be the case if
the project has decided to have a stable branch and follows semver and
stable branch policy: in that case project will keep X.Y frozen on
stable so master should be bumped to at least X.Y+1 immediately after
branching stable.
For a project using PBR postversioning (almost all by now) this can be
achieved by either pushing the appropriate alpha git tag or by using
PBR Sem-Ver: feature https://docs.openstack.org/developer/pbr/#version
Now, purists would complain that artificial commit right after
branching is not really adding a feature but you could look at it as a
credit because you surely will implement new features in the new
version!
If project is without stable:follows-policy tag and it will keep
adding features on the stable branch, increasing Y part of semver or
if they already decided to bump X major version, Sem-Ver: api-break
can be used.
Example for TripleO projects for opening Ocata:
https://review.openstack.org/#/q/topic:open-ocata+Sem-Ver:

> As reference, puppet team proposed a solution for this issue in
> openstack puppet modules in
> http://lists.openstack.org/pipermail/openstack-dev/2017-March/113494.html
>
> However, as mentioned before this is affecting all projects, not only
> puppet ones.

Solution for OpenStack Puppet modules is to explicitly set pre-version
in metadata.json.
We had used pre-versioning before in regular OpenStack projects by
setting version in setup.cfg but this has been abandoned and instead
we rely on PBR to compute the next version.

>> I know we don't expect to fully support upgrades to pre-milestone releases,
>> but the requirement here is to simply enable testing them.
>>
>> A side-benefit of this regular testing e.g via CI is we'll find upgrade 
>> issues
>> much faster than waiting for one or more milestone releases to happen then
>> doing an upgrade-debug firedrill later in the cycle (which has been bad for
>> project and deployment teams IMO, so it'd be good to figure out this first
>> step to enable earlier testing of upgrades to the development release).
>>
>> Any thoughts on how we can resolve this would be much appreciated, thanks!

I was suggested that upgrade jobs could force install i.e. basically
doing downgrades before milestone1 but that's not testing upgrades!
Versioning is very fundamental for packaging and reflects upstream
version, we cannot fake versions in packaging.
In the spirit of Continuous Everything, versioning should also be
continuous and ensured that master version is always > previous
releases.
Projects doing stable branches could take action individually using
Sem-Ver keyword but better would be to define this as a rule for
having stable branches and enforce it cross projects by the release
procedure.

Cheers,
Alan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering

[Alfredo Moralejo Alonso]

On Wed, Mar 8, 2017 at 12:44 PM, Steven Hardy  wrote:
> Hi all,
>
> I wanted to raise visibility of a problem we're experiencing in TripleO CI,
> which I think will potentially affect other projects that consume trunk
> builds from the RDO repositories (and potentially other distros too I
> guess)
>
> The problem is that we tag our final ocata releases after branching
> stable/ocata, but there is then a period prior to cutting pike-1 milestone
> releases (or some other intermediate release for projects not following the
> milestone model) where trunk package builds n-v-r is older than the stable
> branches.
>
> This presents a big problem if you want to test package-derived updates
> from stable/$foo to master, as the pacakges don't get updated where the
> installed stable one is newer than the one built from master.
>
> I raised this bug initially thinking it was specific to the puppet-*
> projects, but it seems from subsequent discussion that it's a broader issue
> that may impact many OpenStack projects.
>
> https://bugs.launchpad.net/tripleo/+bug/1669462
>
> I'm not clear on the best path forward at this point, but the simplest one
> suggested so far is to simply tag a new pre-milestone/alpha release for all
> master branches, which will enable testing upgrades to master.
>

As reference, puppet team proposed a solution for this issue in
openstack puppet modules in
http://lists.openstack.org/pipermail/openstack-dev/2017-March/113494.html

However, as mentioned before this is affecting all projects, not only
puppet ones.

> I know we don't expect to fully support upgrades to pre-milestone releases,
> but the requirement here is to simply enable testing them.
>
> A side-benefit of this regular testing e.g via CI is we'll find upgrade issues
> much faster than waiting for one or more milestone releases to happen then
> doing an upgrade-debug firedrill later in the cycle (which has been bad for
> project and deployment teams IMO, so it'd be good to figure out this first
> step to enable earlier testing of upgrades to the development release).
>
> Any thoughts on how we can resolve this would be much appreciated, thanks!
>
> Steve
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO][release][deployment] Packaging problems due to branch/release ordering

[Steven Hardy]

Hi all,

I wanted to raise visibility of a problem we're experiencing in TripleO CI,
which I think will potentially affect other projects that consume trunk
builds from the RDO repositories (and potentially other distros too I
guess)

The problem is that we tag our final ocata releases after branching
stable/ocata, but there is then a period prior to cutting pike-1 milestone
releases (or some other intermediate release for projects not following the
milestone model) where trunk package builds n-v-r is older than the stable
branches.

This presents a big problem if you want to test package-derived updates
from stable/$foo to master, as the pacakges don't get updated where the
installed stable one is newer than the one built from master.

I raised this bug initially thinking it was specific to the puppet-*
projects, but it seems from subsequent discussion that it's a broader issue
that may impact many OpenStack projects.

https://bugs.launchpad.net/tripleo/+bug/1669462

I'm not clear on the best path forward at this point, but the simplest one
suggested so far is to simply tag a new pre-milestone/alpha release for all
master branches, which will enable testing upgrades to master.

I know we don't expect to fully support upgrades to pre-milestone releases,
but the requirement here is to simply enable testing them.

A side-benefit of this regular testing e.g via CI is we'll find upgrade issues
much faster than waiting for one or more milestone releases to happen then
doing an upgrade-debug firedrill later in the cycle (which has been bad for
project and deployment teams IMO, so it'd be good to figure out this first
step to enable earlier testing of upgrades to the development release).

Any thoughts on how we can resolve this would be much appreciated, thanks!

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev