Than you Corey this sounds like a good path forward to me. As you allude
to we have seen a significant increase in the number of SRUs required
for some releases of Openstack, particularly those that correspond with
LTS releases of Ubuntu e.g. Queens and this is in part due to the
significant delta between the last upstream point release and current
stable branch HEAD for extended maintenance releases of Openstack. This
has itself led to challenges with some SRUs particularly where a fix
constitutes more than one backported patch which themselves potentially
have other dependencies. This is most acute for releases like Nova and
Neutron where the delta can be in the 100s. As you point out, providing
release snapshots is not risk-free but if we provide a sufficient level
of testing and focus resources on testing snapshots rather than
continuous SRUs for the same release I'm sure we will see a net gain
both in terms of reducing load on the SRU team as well getting fixes out
to more users faster. As a starting point I would really like to see us
start building snapshots e.g. every week or two and making them
available for testing. Once we have enough data to prove their stability
we can then consider their suitability for release. And again as you
point out, the inital snapshot will be the largest delta and therefore
require significant testing but subsequent releases will be small so
lower risk (we would apply the same level of testing though).
Thanks,
Ed
On 24/03/2021 13:38, Corey Bryant wrote:
On Tue, Mar 9, 2021 at 2:55 PM Brian Murray <[email protected]
<mailto:[email protected]>> wrote:
On Mon, Mar 08, 2021 at 11:58:18AM -0500, Corey Bryant wrote:
> On Mon, Feb 22, 2021 at 5:16 PM Corey Bryant
<[email protected] <mailto:[email protected]>>
> wrote:
>
> > Hello SRU Team,
> >
> > I'm writing to discuss delivery of patches from extended
maintenance
> > stable branches as snapshots.
> >
> > This would occur during the extended maintenance phase of an
upstream
> > stable branch, which falls between the maintained phase and
EOL phase.
> > I would see this process being similar to the current process
that is used
> > to deliver stable point releases, described at:
> > https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates.
> >
> > Snapshots refers to the process by which we currently deliver
package
> > updates during a development release and prior to the final
release. For
> > example, nova 3:22.0.1+git2021012713.d92c0740c6-0ubuntu1 in
hirsute
> > contains everything in the upstream master git branch up to
commit hash
> > d92c0740c6.
> >
> > == What is Extended Maintenance? ==
> > First, it's worth understanding what the upstream "Maintained"
phase is.
> > During this phase, upstream regularly releases stable point
releases that
> > are based on a specific commit (e.g. the latest HEAD commit
for the
> > stable/queens branch at a point in time), and we then package
those point
> > releases up in distro.
> >
> > Following the "Maintained" phases is the "Extended
Maintenance" phase.
> > During this phase, upstream no longer cuts any stable point
releases for
> > that stable branch, however patches continue to be backported
to the
> > branch. This is the phase where we would like to have the
ability to
> > deliver snapshots of stable branches. This would allow us to
pick up all
> > stable branch patches since, say, the final point release for
stable/queens
> > (bionic). For example, the current version of nova in bionic is
> > 2:17.0.13-0ubuntu1, and that is based on the final point
release for
> > upstream stable/queens. If we were to do a snapshot today for
nova in
> > bionic, it would be versioned along the lines of
> > 2:17.0.13+git2021022200.944443a7b0-0ubuntu1.
> >
> > More details on upstream maintenance phases can be reviewed at:
> >
https://docs.openstack.org/project-team-guide/stable-branches.html#maintenance-phases
> >
> > == Advantages of Stable Snapshots ==
> > * Less overhead for the OpenStack engineering and SRU teams.
Currently we
> > have various cherry-picked patches that are dealt with via
separate SRU
> > bugs requiring individual testing.
> > * As the cherry-pick process is reactive, many of these SRUs
tend to be
> > high impact or critical for users, requiring immediate action
from these
> > teams.
> > * We have a lot of users that are now on, and moving to,
bionic (queens)
> > and we would like to be more proactive in fixing bugs for them.
> > * This would naturally result in a regular cadence along the
lines of the
> > monthly stable point release cadence that we currently follow.
> > * Cherry-picking individual patches can increase regression
potential for
> > frequently modified code when they don't apply cleanly.
> > * All patches that have landed in an extended maintenance
branch have
> > already landed in all ensuing stable branches and master,
therefore
> > negating much of the regression potential mentioned in
"Disadvantages"
> > below.
> >
> > == Disadvantages ==
> > * Upstream stable branches that are in extended maintenance do not
> > guarantee the same upstream test coverage that supported
branches do. The
> > maintenance-phases documentation states "There is no statement
about the
> > level of testing and upgrades from Extended Maintenance are
not supported
> > within the Community."
> >
> > == What Ubuntu Releases would this affect? ==
> > Stable snapshots would only be needed for Ubuntu LTS releases
(Upstream
> > "Maintained" period is 18 months for stable branches).
Currently this would
> > only be applicable to Ubuntu 18.04 as bionic is the only LTS
with a
> > corresponding upstream stable branch that's in extended
maintenance.
> >
> > == How many patches are we talking about? ==
> > For a sampling of stable/queens, new snapshots would pick up
this many
> > patches:
> >
> > nova: 337
> > neutron: 106
> > heat: 47
> > cinder: 21
> > glance: 11
> > designate: 11
> > swift: 10
> > horizon: 8
> > keystone: 7
> >
> > A few of those look daunting, yes, but perhaps that depends on
how you
> > look at it. Also please keep in mind, queens has been in extended
> > maintenance since October of 2019 so this would be quite the
one-time
> > waterfall for bionic. In the future we'd pick up fewer patches
at a time at
> > a regular cadence.
> >
> > Thanks for taking the time to read this. Please let me know if
you have
> > any opinions or questions about this approach.
> >
> > Corey
> >
>
> Hello SRU Team,
>
> I wanted to see if I could revive this topic and discuss the
possibility of
> moving forward with stable snapshots as described in this thread.
>
> Would it make sense to add this to a meeting agenda or is the
mailing list
> a good place to discuss?
The Ubuntu SRU team does have a monthly meeting (conducted via Google
Meet or whatever it is called now) and the next one is scheduled for
Thursday, March 18th. Unfortunately I won't be able to attend that
meeting but perhaps the other SRU team members can invite you so
it can
be discussed.
Cheers,
--
Brian Murray
Last week I attended the SRU team meeting and also met with Edward
Hope-Morley from Canonical Sustaining Engineering to discuss stable
snapshots. Ed's team is one of the primary drivers/stakeholders for
Ubuntu OpenStack SRU work. Thank you to all who provided input at the
meetings.
The main result of the discussion with the SRU team is that we would
need to ensure we are closing the gap, specifically with extended
maintenance branches, where upstream may have a reduced level of
testing (including upgrade testing). Our QA process at
https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates would need to
be enhanced with testing efforts that would mitigate the regression
potential introduced by reduced upstream gate testing.
In discussion with Ed, we came to the conclusion that a proof of
concept makes sense before moving forward. For example we can snapshot
the upstream stable/queens branches (see section labeled "How many
patches are we talking about?" above in thread) and both Ed's team and
ours can test them in a PPAs. This will be an opportunity to assess
the code quality and enhance our QA tests before moving forward with
anything official. So that's the plan for now, and I will plan on
reporting back on this thread once we are ready (or not) to move on
from the POC.
Thanks,
Corey
--
Ubuntu-release mailing list
[email protected]
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-release
