On 04/28/2017 11:17 AM, Jeff Nelson wrote: > On Fri, 28 Apr 2017 15:58:11 +0200 > Amador Segundo <asegu...@redhat.com> wrote: > >> On Thu, Apr 27, 2017 at 5:15 PM, Cleber Rosa <cr...@redhat.com> wrote: >>> >>> >>> On 04/27/2017 04:04 AM, Amador Segundo wrote: >>>> On Thu, Apr 27, 2017 at 1:07 AM, Cleber Rosa <cr...@redhat.com> wrote: >>>>> ========================== >>>>> RFC: Long Term Stability >>>>> ========================== >>>>> >>>>> This RFC contains proposals and clarifications regarding the >>>>> maintenance and release processes of Avocado. >>>>> >>>>> We understand there are multiple teams currently depending on the >>>>> stability of Avocado and we don't want their work to be disrupted by >>>>> incompatibilities nor instabilities in new releases. >>>>> >>>>> This version is a minor update to previous versions of the same RFC >>>>> (see `Changelog`_) which drove the release of Avocado 36.0 LTS. The >>>>> Avocado team has plans for a new LTS release in the near future, so >>>>> please consider reading and providing feedback on the proposals here. >>>>> >>>>> TL;DR >>>>> ===== >>>>> >>>>> We plan to keep the current approach of sprint releases every 3-4 >>>>> weeks, but we're introducing "Long Term Stability" releases which >>>>> should be adopted in production environments where users can't keep up >>>>> with frequent upgrades. >>>>> >>>>> Changelog >>>>> ========= >>>>> >>>>> Changes from `Version 3`_: >>>>> >>>>> * Converted formatting to REStructuredText >>>>> * Replaced "me" mentions on version 1 changelog with proper name >>>>> (Ademar Reis) >>>>> * Renamed section "Misc Details" to `Deployment Details`_ >>>>> * Renamed "avocado-vt" to "Avocado-VT" >>>>> * Start the timeline example with version 36.0 >>>>> * Be explicit on timeline example that a minor bug did not generate >>>>> an immediate release >>>>> >>>>> Changes from `Version 2`_: >>>>> >>>>> * Wording changes on second paragraph ("... nor instabilities...") >>>>> * Clarified on "Introduction" that change of behavior is introduced >>>>> between regular releases >>>>> * Updated distro versions for which official packages are built >>>>> * Add more clear explanation on official packages on the various >>>>> hardware platforms >>>>> * Used more recent version numbers as examples, and the planned >>>>> new LTS version too >>>>> * Explain how users can get the LTS version when using tools such as >>>>> pip >>>>> * Simplified the timeline example, with examples that will possibly >>>>> match the future versions and releases >>>>> * Documented current status of Avocado-VT releases and packages >>>>> >>>>> Changes from `Version 1`_: >>>>> >>>>> * Changed "Support" to "Stability" and "supported" to "maintained" >>>>> [Jeff Nelson] >>>>> * Misc improvements and clarifications in the >>>>> supportability/stability statements [Jeff Nelson, Ademar Reis] >>>>> * Fixed a few typos [Jeff Nelson, Ademar Reis] >>>>> >>>>> Introduction >>>>> ============ >>>>> >>>>> We make new releases of Avocado every 3-4 weeks on average. In theory >>>>> at least, we're very careful with backwards compatibility. We test >>>>> Avocado for regressions and we try to document any issues, so >>>>> upgrading to a new version should be (again, in theory) safe. >>>>> >>>>> But in practice both intended and unintended changes are introduced >>>>> during development, and both can be frustrating for conservative >>>>> users. We also understand it's not feasible for users to upgrade >>>>> Avocado very frequently in a production environment. >>>>> >>>>> The objective of this RFC is to clarify our maintenance practices and >>>>> introduce Long Term Stability (LTS) releases, which are intended to >>>>> solve, or at least mitigate, these problems. >>>>> >>>>> >>>>> Our definition of maintained, or stable >>>>> ======================================= >>>>> >>>>> First of all, Avocado and its sub-projects are provided 'AS IS' and >>>>> WITHOUT ANY WARRANTY, as described in the LICENSE file. >>>>> >>>>> The process described here doesn't imply any commitments or >>>>> promises. It's just a set of best practices and recommendations. >>>>> >>>>> When something is identified as "stable" or "maintained", it means the >>>>> development community makes a conscious effort to keep it working and >>>>> consider reports of bugs and issues as high priorities. Fixes >>>>> submitted for these issues will also be considered high priorities, >>>>> although they will be accepted only if they pass the general >>>>> acceptance criteria for new contributions (design, quality, >>>>> documentation, testing, etc), at the development team discretion. >>>>> >>>>> >>>>> Maintained projects and platforms >>>>> ================================= >>>>> >>>>> The only maintained project as of today is the Avocado Test Runner, >>>>> including its APIs and core plugins (the contents of the main avocado >>>>> git repository). >>>>> >>>>> Other projects kept under the "Avocado Umbrella" in github may be >>>>> maintained by different teams (e.g.: Avocado-VT) or be considered >>>>> experimental (e.g.: avocado-server and avocado-virt). >>>>> >>>>> More about Avocado-VT in its own section further down. >>>>> >>>>> As a general rule, fixes and bug reports for Avocado when running in >>>>> any modern Linux distribution are welcome. >>>>> >>>>> But given the limited capacity of the development team, packaged >>>>> versions of Avocado will be tested and maintained only for the >>>>> following Linux distributions: >>>>> >>>>> * RHEL 7.x (latest) >>>>> * Fedora (stable releases from the Fedora projects) >>>>> >>>>> Currently all packages produced by the Avocado projects are "noarch". >>>>> That means that they could be installable on any hardware platform. >>>>> Still, the development team will currently attempt to provide versions >>>>> that are stable for the following platforms: >>>>> >>>>> * x86 >>>>> * ppc64 >>>>> * ppc64le >>>>> >>>>> Contributions from the community to maintain other platforms and >>>>> operating systems are very welcome. >>>>> >>>>> The lists above may change without prior notice. >>>>> >>>>> Avocado Releases >>>>> ================ >>>>> >>>>> The proposal is to have two different types of Avocado releases: >>>>> >>>>> Sprint Releases >>>>> --------------- >>>>> >>>>> (This is the model we currently adopt in Avocado) >>>>> >>>>> They happen every 3-4 weeks (the schedule is not fixed) and >>>>> their versions are numbered serially, with decimal digits in >>>>> the format <major>.<minor>. Examples: 47.0, 48.0, 49.0. Minor >>>>> releases are rare, but necessary to correct some major issue >>>>> with the original release (47.1, 47.2, etc). >>>>> >>>>> Only the latest Sprint Release is maintained. >>>>> >>>>> In Sprint Releases we make a conscious effort to keep backwards >>>>> compatibility with the previous version (APIs and behavior) and >>>>> as a general rule and best practice, incompatible changes in >>>>> Sprint Releases should be documented in the release notes and >>>>> if possible deprecated slowly, to give users time to adapt >>>>> their environments. >>>>> >>>>> But we understand changes are inevitable as the software >>>>> evolves and therefore there's no absolute promise for API and >>>>> behavioral stability. >>>>> >>>>> Long Term Stability (LTS) Releases >>>>> ---------------------------------- >>>>> >>>>> LTS releases should happen whenever the team feels the code is >>>>> stable enough to be maintained for a longer period of time, ideally >>>>> once or twice per year (no fixed schedule). >>>>> >>>>> They should be maintained for 18 months, receiving fixes for major >>>>> bugs in the form of minor (sub-)releases. With the exception of >>>>> these fixes, no API or behavior should change in a minor LTS >>>>> release. >>>>> >>>>> They will be versioned just like Sprint Releases, so looking at the >>>>> version number alone will not reveal the differentiate release >>>>> process and stability characteristics. >>>>> >>>>> In practice each major LTS release will imply in the creation of a >>>>> git branch where only serious issues affecting users will be >>>>> fixed. The code in a LTS branch is stable, frozen for new features. >>>> >>>> I'd improve the sentence "only serious issues affecting users will be >>>> fixed". In fact, we "backport" (meaning upstream first) "any issues" >>>> (not only the serious ones) that are possible to be fixed *and* are >>>> affecting users. >>>> >>> >>> OK, good point. My suggestion is to use "important issues", because the >>> definition of a "bug" and a "serious issue" is too relative. We may >>> identify a bug in LTS that is mostly harmless, and decide to not fix it >>> because it could require major changes. Anyway, my proposal: >>> >>> "In practice each major LTS release will imply in the creation of a git >>> branch where only important issues affecting users will be fixed, >>> usually as a backport of a fix initially applied upstream. The code in >>> a LTS branch is stable, frozen for new features." >>> >>> How does that sound? >> >> Yup, looks great. >> >>> >>>> Not sure if it is also worth it to define 'backport'. Depending on the >>>> code distance between the LTS and the master branch, it can be a >>>> simple 'cherry-pick' or a completely different fix in terms of code, >>>> as long as the bug gets fixed. >>>> >>> >>> I'd expect readers to be familiar with the concept of backport. I'm >>> reluctant about introducing the concept in this document. >>> >>>>> >>>>> Notice that although within a LTS release there's a expectation >>>>> of stability because the code is frozen, different (major) LTS >>>>> releases may include changes in behavior, API incompatibilities >>>>> and new features. The development team will make a considerable >>>>> effort to minimize and properly document these changes (changes >>>>> when comparing it to the last major LTS release). >>>>> >>>>> Sprint Releases are replaced by LTS releases. I.e., in the cycle >>>>> when 52.0 (LTS) is released, that's also the version used as a >>>>> Sprint Release (there's no 52.0 -- non LTS -- in this case). >>>>> >>>>> New LTS releases should be done carefully, with ample time for >>>>> announcements, testing and documentation. It's recommended >>>>> that one or two sprints are dedicated as preparations for a LTS >>>>> release, with a Sprint Release serving as a "LTS beta" release. >>>>> >>>>> Similarly, there should be announcements about the end-of-life >>>>> (EOL) of a LTS release once it approaches its 18 months of >>>>> life. >>>>> >>>>> >>>>> Deployment details >>>>> ================== >>>>> >>>>> Sprint and LTS releases, when packaged, whenever possible, will be >>>>> preferably distributed through different package channels >>>>> (repositories). >>>>> >>>>> This is possible for repository types such as `YUM/DNF repos`_. In >>>>> such cases, users can disable the regular channel, and enable the LTS >>>>> version. A request for the installation of Avocado packages will >>>>> fetch the latest version available in the enabled repository. If the >>>>> LTS repository channel is enabled, the packages will receive minor >>>>> updates (bugfixes only), until a new LTS version is released (roughly >>>>> every 12 months). >>>>> >>>>> If the non-LTS channel is enabled, users will receive updates every >>>>> 3-4 weeks. >>>>> >>>>> On other types of repos such as `PyPI`_ which have no concept of >>>>> "sub-repos" or "channels", users can request a version smaller than >>>>> the version that succeeds the current LTS to get the latest LTS >>>>> (including minor releases). Suppose the current LTS major version is >>>>> 52, but there have been minor releases 52.1 and 52.2. By running:: >>>>> >>>>> pip install 'avocado-framework<53.0' >>>>> >>>>> pip provide LTS version 52.2. If 52.3 gets released, they will be >>>>> automatically deployed instead. When a new LTS is released, users >>>>> would still get the latest minor release from the 52.0 series, unless >>>>> they update the version specification. >>>>> >>>>> The existence of LTS releases should never be used as an excuse >>>>> to break a Sprint Release or to introduce gratuitous >>>>> incompatibilities there. In other words, Sprint Releases should >>>>> still be taken seriously, just as they are today. >>>>> >>>>> >>>>> Timeline example >>>>> ================ >>>>> >>>>> Assume each sprint is taking 3 weeks. Notice how multiple LTS >>>>> releases can co-exist before EOL: >>>>> >>>>> 36.0 >>>>> * **LTS** release (the only LTS release available at the time of >>>>> writing) >>>>> >>>>> 37.0 .. 49.0 >>>>> * sprint releases >>>>> >>>>> 50.0 >>>>> * sprint release >>>>> * start preparing a LTS release, so 51.0 will be a **beta LTS** >>>>> >>>>> 51.0 >>>>> * sprint release >>>>> * **beta LTS** release >>>>> >>>>> 52.0 >>>>> * **LTS** release >>>>> * 52lts branch is created >>>>> * packages go into LTS repo >>>>> * both **36.x LTS** and **52.x LTS** maintained from this point on >>>>> >>>>> 53.0 >>>>> * sprint release >>>>> * bug that also affects 52.0 is found, fix gets added to master and >>>>> 52lts branches >>>> >>>> Fix gets added to master and backported to supported LTS branches (at >>>> this point, 52lts and 36lts). >>>> >>> >>> Yeah, good point (assuming it also applies to 36lts, as you mentioned on >>> the other reply). >> >> Yes, I think it makes sense to mention that here to make clear that >> all supported branches will receive the fix, as long as there are >> affected users. >> >> My proposal: >> >> 53.0 >> * sprint release >> * bug that also affects users of 52.0 and 36.0 is found, fix gets >> added to master and is backported to 52lts and 36lts branches. >> >> 54.0 >> * sprint release 54.0 **AND** LTS 52.1 release **AND** LTS 36.5 release. >> * minor bug that also affects users of 52.1 is found, fix gets added >> to master and is backported to 52lts branch. >> >> 55.0 >> * sprint release 55.0 **AND** LTS 52.2 release >> * critical bug that affects 52.2 *only* is found, fix gets added to >> 52lts and 52.3 LTS is immediately released > > Maybe I'm being pedantic, but how about some visual separation between > the sprint release and the things that follow? It looks to me as if > one could interpret everything under a numbered sprint release as > happening at the same time, when in fact we want to point out that > (a) bug discovery can happen at any time > (b) the bugfix occurs next > (c) the severity of the defect determines the timing of the release > (c.1) moderate and minor bugfixes to lts branches are held until the > next sprint release > (c.2) critical bugs are released asynchronously, without waiting for the > next sprint release. >
Those are all valid and true points. I've spent quite some time trying to fit the timeline into a easy to read format (that is also valid REstructuredText), so I won't promise that I'll be able to squeeze all that in. At the very least, I'll include your listing above as a summary of how things work. > 53 > * sprint release 53.0 > > * moderate bug that affects users of master, 52lts and 36lts is found > * fix is added to master and backported to 52lts and 36lts > > 54 > * sprint release 54.0 **AND** 36lts release 36.5 **AND** 52lts release 52.1 > > * minor bug that affects users of master and 52lts is found > * fix is added to master and backported to 52lts > > 55 > * sprint release 55.0 **AND** 52lts release 52.2 > > * critical bug is found that affects users of 52lts *only* > * fix is made to 52lts and 52lts release 52.3 is immediately announced > > I dunno; just something to think about. > > -Jeff > > Yep, thinking about it! Thanks for the feedback. - Cleber. >>> >>>>> >>>>> 54.0 >>>>> * sprint release 54.0 **AND** LTS 52.1 release >>>>> * minor bug that also affects 52.0 is found, fix gets added to >>>>> master and 52lts branches >>>> >>>> Fix gets added to master and backported to supported LTS branches (at >>>> this point, 52lts and 36lts). >>>> >>>>> >>>>> 55.0 >>>>> * sprint release >>>>> * critical bug that affects 52.1 *only* is found, fix gets added to >>>>> 52lts and 52.2 LTS is immediately released >>>>> >>>>> 56.0 >>>>> * sprint release >>>>> >>>>> 57.0 >>>>> * sprint release >>>>> >>>>> 58.0 >>>>> * sprint release >>>>> >>>>> 59.0 >>>>> * sprint release >>>>> * EOL for **36.x LTS** (18 months since the release of 36.0) >>>>> >>>>> Avocado-VT >>>>> ========== >>>>> >>>>> Avocado-VT is an Avocado plugin that allows "VT tests" to be run >>>>> inside Avocado. It's a third-party project maintained mostly by >>>>> Engineers from Red Hat QE with assistance from the Avocado team >>>>> and other community members. >>>>> >>>>> It's a general consensus that QE teams use Avocado-VT directly >>>>> from git, usually following the master branch, which they >>>>> control. >>>>> >>>>> There's no official maintenance or stability statement for >>>>> Avocado-VT. Even though the upstream community is quite >>>>> friendly and open to both contributions and bug reports, >>>>> Avocado-VT is made available without any promises for >>>>> compatibility or supportability. >>>>> >>>>> When packaged and versioned, Avocado-VT rpms should be considered >>>>> just snapshots, available in packaged form as a convenience to >>>>> users outside of the Avocado-VT development community. Again, >>>>> they are made available without any promises of compatibility or >>>>> stability. >>>>> >>>>> * Which Avocado version should be used by Avocado-VT? >>>>> >>>>> This is up to the Avocado-VT community to decide, but the >>>>> current consensus is that to guarantee some stability in >>>>> production environments, Avocado-VT should stick to a specific >>>>> LTS release of Avocado. In other words, the Avocado team >>>>> recommends production users of Avocado-VT not to install Avocado >>>>> from its master branch or upgrade it from Sprint Releases. >>>>> >>>>> Given each LTS release will be maintained for 18 months, it >>>>> should be reasonable to expect Avocado-VT to upgrade to a new >>>>> LTS release once a year or so. This process will be done with >>>>> support from the Avocado team to avoid disruptions, with proper >>>>> coordination via the avocado mailing lists. >>>>> >>>>> In practice the Avocado development team will keep watching >>>>> Avocado-VT to detect and document incompatibilities, so when >>>>> the time comes to do an upgrade in production, it's expected >>>>> that it should happen smoothly. >>>>> >>>>> * Will it be possible to use the latest Avocado and Avocado-VT >>>>> together? >>>>> >>>>> Users are welcome to *try* this combination. The Avocado >>>>> development team itself will do it internally as a way to monitor >>>>> incompatibilities and regressions. >>>>> >>>>> Whenever Avocado is released, a matching versioned snapshot of >>>>> Avocado-VT will be made. Packages containing those Avocado-VT >>>>> snapshots, for convenience only, will be made available in the >>>>> regular Avocado repository. >>>>> >>>>> .. _Version 1: >>>>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00006.html >>>>> .. _Version 2: >>>>> https://www.redhat.com/archives/avocado-devel/2016-April/msg00038.html >>>>> .. _Version 3: >>>>> https://www.redhat.com/archives/avocado-devel/2017-April/msg00032.html >>>>> .. _YUM/DNF repos: >>>>> http://avocado-framework.readthedocs.io/en/latest/GetStartedGuide.html#fedora-from-avocado-s-own-repo >>>>> .. _PyPI: https://pypi.python.org/pypi >>>>> >>>>> >>>>> -- >>>>> Cleber Rosa >>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ] >>>>> [ Avocado Test Framework - avocado-framework.github.io ] >>>>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] >>>>> >>> >>> -- >>> Cleber Rosa >>> [ Sr Software Engineer - Virtualization Team - Red Hat ] >>> [ Avocado Test Framework - avocado-framework.github.io ] >>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] >>> > -- Cleber Rosa [ Sr Software Engineer - Virtualization Team - Red Hat ] [ Avocado Test Framework - avocado-framework.github.io ] [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
signature.asc
Description: OpenPGP digital signature