Re: [openstack-dev] [oslo] versioning and releases
On 11 June 2014 04:24, Doug Hellmann doug.hellm...@dreamhost.com wrote: ... Incubated libraries have been released at the end of a release cycle, as with the rest of the integrated packages. Adopted libraries have historically been released as needed during their development. We would like to synchronize these so that all oslo libraries are officially released with the rest of the software created by OpenStack developers. So I see a fairly big problem with this - its going to impact heavily on anyone that isn't part of oslo using oslo.* as a library. For instance, the openstack clients which use oslo.config - they will not be able to get new features or bugfixes from trunk without waiting 6 months. And that means either: - carrying conditional code (version checking the version of oslo) in their trunk, - or not consuming new features until up to 6 months after release - or making their trunk *uninstallable* (because it will have either an dep on a non-existing version, or a dep on an old version that doesn't actually include the needed feature). The last option is what I suspect will happen, and that will be very sad, since it seems unneeded. It will also make version management hard for CD deployers (like TripleO's deploy-trunk default) because incorrect requirements files are anaethema. Markmc's mail about wanting the same rhythm doesn't make sense to me since more than half the openstack artifacts *do not have* the 6 month rhythm - client libraries have always been semver, release when ready. AIUI the stable branches thing in the server space exists to give us freedom to evolve the code super rapidly, in these large, fairly monolithic APIs; we need it because we are always backward-incompatible between server /releases/ (only of the server HTTP API) - if we do semver, we'd be doing a breaking release every 6 months. So stable branches is a valve to balance trunk velocity of independent codebases vs user demands for bugfixes and security fixes. The other basic model we have as software releasers is to do stable branches only when a backwards incompatible change has happened - thats what we demand from projects like setuptools *and our client libraries*: we want to be able to fork at that point, but the default behaviour is not to fork, and instead just not break things. Not break things is roughly proportional to the size of the codebase, and the oslo projects are typically small focused projects where this seems reasonable to expect. *More than that*, I believe its essential, or we'll have a very hard time propogating new releases out amongst the servers. tl;dr there seems to be no reason to do one release every 6 months of olso.*, and doing so seems likely to only incur costs, since the discipline needed to do regular semver with good backwards compat is a discipline we're going to have to exercise anyway. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] versioning and releases
On Tue, Jun 17, 2014 at 6:42 PM, Robert Collins robe...@robertcollins.net wrote: On 11 June 2014 04:24, Doug Hellmann doug.hellm...@dreamhost.com wrote: ... Incubated libraries have been released at the end of a release cycle, as with the rest of the integrated packages. Adopted libraries have historically been released as needed during their development. We would like to synchronize these so that all oslo libraries are officially released with the rest of the software created by OpenStack developers. So I see a fairly big problem with this - its going to impact heavily on anyone that isn't part of oslo using oslo.* as a library. For instance, the openstack clients which use oslo.config - they will not be able to get new features or bugfixes from trunk without waiting 6 months. And that means either: - carrying conditional code (version checking the version of oslo) in their trunk, - or not consuming new features until up to 6 months after release - or making their trunk *uninstallable* (because it will have either an dep on a non-existing version, or a dep on an old version that doesn't actually include the needed feature). The last option is what I suspect will happen, and that will be very sad, since it seems unneeded. It will also make version management hard for CD deployers (like TripleO's deploy-trunk default) because incorrect requirements files are anaethema. I don't follow that. What does synchronizing the release cycles have to do with whether or not things are installable? In order to make use of the library in unit tests, the project will have to have the correct versions in their requirements file so the package can be installed from pypi -- including allowing an alpha release if that's where the feature is. To allow unit tests to work on developer systems, we will tag alpha releases and publish them to pypi. AFAICT, the only thing that's special about those releases is that they will be wheels instead of sdists. Markmc's mail about wanting the same rhythm doesn't make sense to me since more than half the openstack artifacts *do not have* the 6 month rhythm - client libraries have always been semver, release when ready. AIUI the stable branches thing in the server space exists to give us freedom to evolve the code super rapidly, in these large, fairly monolithic APIs; we need it because we are always backward-incompatible between server /releases/ (only of the server HTTP API) - if we do semver, we'd be doing a breaking release every 6 months. So stable branches is a valve to balance trunk velocity of independent codebases vs user demands for bugfixes and security fixes. The other basic model we have as software releasers is to do stable branches only when a backwards incompatible change has happened - thats what we demand from projects like setuptools *and our client libraries*: we want to be able to fork at that point, but the default behaviour is not to fork, and instead just not break things. And that's what I plan to do with Oslo. I mentioned stable branches as a solution for security fixes mainly, though I wouldn't say only. I don't expect to be doing a lot of back-porting of insignificant fixes, since we hardly have enough reviewers to cover the changes we're trying to make now. Not break things is roughly proportional to the size of the codebase, and the oslo projects are typically small focused projects where this seems reasonable to expect. *More than that*, I believe its essential, or we'll have a very hard time propogating new releases out amongst the servers. Believe me, I know how hard it can be to update the version of a library used by more than one project in OpenStack. That's why I have been focusing so much attention on figuring out how to handle the cross-testing at the unit test level, so we know in the gate when a change in a library is going to break one of the servers, and can reject the patch. tl;dr there seems to be no reason to do one release every 6 months of olso.*, and doing so seems likely to only incur costs, since the discipline needed to do regular semver with good backwards compat is a discipline we're going to have to exercise anyway. As you point out, most of these libraries are small. I don't actually expect to create new releases of all of them every cycle. In fact, I expect most of them to be extremely stable over time. Doug -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] versioning and releases
Doug Hellmann wrote: On Tue, Jun 10, 2014 at 5:19 PM, Mark McLoughlin mar...@redhat.com wrote: On Tue, 2014-06-10 at 12:24 -0400, Doug Hellmann wrote: [...] Background: We have two types of oslo libraries. Libraries like oslo.config and oslo.messaging were created by extracting incubated code, updating the public API, and packaging it. Libraries like cliff and taskflow were created as standalone packages from the beginning, and later adopted by the oslo team to manage their development and maintenance. Incubated libraries have been released at the end of a release cycle, as with the rest of the integrated packages. Adopted libraries have historically been released as needed during their development. We would like to synchronize these so that all oslo libraries are officially released with the rest of the software created by OpenStack developers. Could you outline the benefits of syncing with the integrated release ? Personally I see a few drawbacks to this approach: We dump the new version on consumers usually around RC time, which is generally a bad time to push a new version of a dependency and detect potential breakage. Consumers just seem to get the new version at the worst possible time. It also prevents from spreading the work all over the cycle. For example it may have been more successful to have the oslo.messaging new release by milestone-1 to make sure it's adopted by projects in milestone-2 or milestone-3... rather than have it ready by milestone-3 and expect all projects to use it by consuming alphas during the cycle. Now if *all* projects were continuously consuming alpha versions, most of those drawbacks would go away. [...] Patch Releases: Updates to existing library releases can be made from stable branches. Checking out stable/icehouse of oslo.config for example would allow a release 1.3.1. We don't have a formal policy about whether we will create patch releases, or whether applications are better off using the latest release of the library. Do we need one? I'm not sure we need one, but if we did I'd expect them to be aligned with stable releases. Right now, I think they'd just be as-needed - if there's enough backported to the stable branch to warrant a release, we just cut one. That's pretty much what I thought, too. We shouldn't need to worry about alphas for patch releases, since we won't add features. Yes, I think we can be pretty flexible about it. But to come back to my above remark... should it be stable/icehouse or stable/1.3 ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] versioning and releases
On Thu, 2014-06-12 at 12:09 +0200, Thierry Carrez wrote: Doug Hellmann wrote: On Tue, Jun 10, 2014 at 5:19 PM, Mark McLoughlin mar...@redhat.com wrote: On Tue, 2014-06-10 at 12:24 -0400, Doug Hellmann wrote: [...] Background: We have two types of oslo libraries. Libraries like oslo.config and oslo.messaging were created by extracting incubated code, updating the public API, and packaging it. Libraries like cliff and taskflow were created as standalone packages from the beginning, and later adopted by the oslo team to manage their development and maintenance. Incubated libraries have been released at the end of a release cycle, as with the rest of the integrated packages. Adopted libraries have historically been released as needed during their development. We would like to synchronize these so that all oslo libraries are officially released with the rest of the software created by OpenStack developers. Could you outline the benefits of syncing with the integrated release ? Sure! http://lists.openstack.org/pipermail/openstack-dev/2012-November/003345.html :) Personally I see a few drawbacks to this approach: We dump the new version on consumers usually around RC time, which is generally a bad time to push a new version of a dependency and detect potential breakage. Consumers just seem to get the new version at the worst possible time. It also prevents from spreading the work all over the cycle. For example it may have been more successful to have the oslo.messaging new release by milestone-1 to make sure it's adopted by projects in milestone-2 or milestone-3... rather than have it ready by milestone-3 and expect all projects to use it by consuming alphas during the cycle. Now if *all* projects were continuously consuming alpha versions, most of those drawbacks would go away. Yes, that's the plan. Those issues are acknowledged and we're reasonably confident the alpha versions plan will address them. [...] Patch Releases: Updates to existing library releases can be made from stable branches. Checking out stable/icehouse of oslo.config for example would allow a release 1.3.1. We don't have a formal policy about whether we will create patch releases, or whether applications are better off using the latest release of the library. Do we need one? I'm not sure we need one, but if we did I'd expect them to be aligned with stable releases. Right now, I think they'd just be as-needed - if there's enough backported to the stable branch to warrant a release, we just cut one. That's pretty much what I thought, too. We shouldn't need to worry about alphas for patch releases, since we won't add features. Yes, I think we can be pretty flexible about it. But to come back to my above remark... should it be stable/icehouse or stable/1.3 ? It's a branch for bugfix releases of the icehouse version of the library, so I think stable/icehouse makes sense. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] versioning and releases
Mark McLoughlin wrote: On Thu, 2014-06-12 at 12:09 +0200, Thierry Carrez wrote: Doug Hellmann wrote: On Tue, Jun 10, 2014 at 5:19 PM, Mark McLoughlin mar...@redhat.com wrote: On Tue, 2014-06-10 at 12:24 -0400, Doug Hellmann wrote: [...] Background: We have two types of oslo libraries. Libraries like oslo.config and oslo.messaging were created by extracting incubated code, updating the public API, and packaging it. Libraries like cliff and taskflow were created as standalone packages from the beginning, and later adopted by the oslo team to manage their development and maintenance. Incubated libraries have been released at the end of a release cycle, as with the rest of the integrated packages. Adopted libraries have historically been released as needed during their development. We would like to synchronize these so that all oslo libraries are officially released with the rest of the software created by OpenStack developers. Could you outline the benefits of syncing with the integrated release ? Sure! http://lists.openstack.org/pipermail/openstack-dev/2012-November/003345.html :) Heh :) I know why *you* prefer it synced. Was just curious to see if Doug thought the same way :P Personally I see a few drawbacks to this approach: We dump the new version on consumers usually around RC time, which is generally a bad time to push a new version of a dependency and detect potential breakage. Consumers just seem to get the new version at the worst possible time. It also prevents from spreading the work all over the cycle. For example it may have been more successful to have the oslo.messaging new release by milestone-1 to make sure it's adopted by projects in milestone-2 or milestone-3... rather than have it ready by milestone-3 and expect all projects to use it by consuming alphas during the cycle. Now if *all* projects were continuously consuming alpha versions, most of those drawbacks would go away. Yes, that's the plan. Those issues are acknowledged and we're reasonably confident the alpha versions plan will address them. I agree that if we release alphas often and most projects consume them instead of jump from stable release to stable release, we have all the benefits without the drawbacks. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] versioning and releases
On Tue, Jun 10, 2014 at 5:19 PM, Mark McLoughlin mar...@redhat.com wrote: On Tue, 2014-06-10 at 12:24 -0400, Doug Hellmann wrote: As part of the push to release code from the oslo incubator in stand-alone libraries, we have had several different discussions about versioning and release schedules. This is an attempt to collect all of the decisions we have made in those discussions and to lay out the rationale for the approach we are taking. I don't expect any of this to be a surprise, since we've talked about it, but I haven't actually written it all down in one place before so some of you might not have seen all of the points. Please let me know if you see any issues with the proposal or have questions. If everyone agrees that this makes sense, I'll put it in the wiki. Nice work Doug, this is looking great. Background: We have two types of oslo libraries. Libraries like oslo.config and oslo.messaging were created by extracting incubated code, updating the public API, and packaging it. Libraries like cliff and taskflow were created as standalone packages from the beginning, and later adopted by the oslo team to manage their development and maintenance. Incubated libraries have been released at the end of a release cycle, as with the rest of the integrated packages. Adopted libraries have historically been released as needed during their development. We would like to synchronize these so that all oslo libraries are officially released with the rest of the software created by OpenStack developers. The first release of oslo.config was 1.1.0, as part of the grizzly release. The first release of oslo.messaging was 1.2.0, as part of the havana release. oslo.config was also updated to 1.2.0 during havana. All current adopted libraries have release numbers less than 1.0.0. In the past, alpha releases of oslo libraries have been distributed as tarballs on an openstack server, with official releases going to PyPI. Applications that required the alpha release specified the tarball in their requirements list, followed by a version specifier. This allowed us to prepare alpha releases, without worrying that their release would break continuous-deployment systems by making new library releases available to pip. This approach still made it difficult for an application developer to rely on new features of an oslo library, until an alpha version was produced. When the PyPI mirror was introduced in our CI system, relying on tarballs not available on the mirror conflicted with our desire to have the gate system install *only* from the package mirror. As we are now installing only from the mirror, we need to publish our alpha releases in a format that will work with the mirror. We already gate OpenStack applications and oslo libraries using integration tests using the normal devstack-gate jobs. During Icehouse we had a couple of oslo library releases that broke unit tests of applications after the library was released. We plan to address that with separate gating jobs during juno. In addition to that gating, we need to support developers who want to use new features of oslo libraries before official releases are available. A Version Numbering Scheme: At the Juno summit, Josh proposed that we use semantic versioning (SemVer) for oslo libraries [1]. Part of that proposal also included ideas for allowing breaking backwards compatibility at some release boundaries, and I am explicitly *not* addressing backwards-incompatible changes beyond saying that we do not expect to have any during Juno. We do need to solve the problem of breaking API compatibility, but I want to take one step at a time. The first step is choosing a rational release versioning scheme. SemVer is widely used and gives us relatively clear guidelines about choosing new version numbers. It supports alpha releases, which are going to be key to meeting some of our other requirements. I propose that we adopt pbr's modified SemVer [2] for new releases, beginning with Juno. The versions for existing libraries oslo.config and oslo.messaging will be incremented from their Icehouse versions but updating the minor number (1.x.0) at the end of the Juno cycle. All adopted libraries using numbers less than 1.0 will be released as 1.0.0 at the end of the Juno cycle, based on the fact that we expect deployers to use them in production. Perhaps not directly related to this proposal, but IMHO OpenStack should not depend on any semver-versioned libraries which are still on 0.x. i.e. we should not depend on API unstable projects (because of the pain it imposes on everyone), particularly ones that are likely to feel empowered to break their API at will because of their version number :) Releases during Juno should *all* be marked as alphas of the anticipated upcoming SemVer-based release number (1.0.0.0aN or 1.2.0.0aN or whatever). Why not simply 1.2.0aN btw? I read Robert's pbr-semver
Re: [openstack-dev] [oslo] versioning and releases
On Jun 10, 2014, at 5:19 PM, Mark McLoughlin mar...@redhat.com wrote: The new CI system can create packages as Python wheels and publish them to the appropriate servers, which means projects will no longer need to refer explicitly to pre-release tarballs. The details are a bit more nuanced here - pip won't install alpha libraries unless you explicitly request them with a command line flag to install any alphas available or you explicitly require the alpha version. It doesn’t have to explicitly require the alpha, it just has to include pre-releases so stuff like =1.2a0,1.3 would include 1.2a0 and higher, and “less than 1.3”. Unfortunately that’s a bit wonky still because 1.3a0 is technically 1.3 so it’d match too. At least until we finish implementing PEP 440. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
