Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
An initial WIP patch to show the direction I think this could materialize - https://review.openstack.org/125346 Early feedback appreciated so I can do this conversion across the board. My assumption is that all non integrated release projects should be converted to this model. On 09/26/2014 02:27 PM, Sean Dague wrote: > As we've been talking about the test disaggregation the hamster wheels > in the back of my brain have been churning on the library testing > problem I think we currently have. Namely, openstack components mostly > don't rely on released library versions, they rely on git master of > them. Right now olso and many clients are open masters (their > stable/juno versions are out), but we've still not cut RCs on servers. > So we're actually going to have a time rewind event as we start cutting > stables. > > We did this as a reaction to the fact that library releases were often > cratering the world. However, I think the current pattern leads us into > a much more dangerous world where basically the requirements.txt is invalid. > > So here is the particular unwind that I think would be useful here: > > 1) Change setup_library in devstack to be able to either setup the > library from git or install via pip. This would apply to all libraries > we are installing from oslo, the python clients, stackforge, etc. > Provide a mechanism to specify LIBRARIES_FROM_GIT (or something) so that > you can selectively decide to use libraries from git for development > purposes. > > 2) Default devstack to use pip released versions. > > 3) Change the job definition on the libraries to test against devstack > in check, not in gate. The library teams can decide if they want their > forward testing to be voting or not, but this is basically sniff testing > that when they release a new library they won't ruin the world for > everyone else. > > 4) If a ruin the world event happens, figure out how to prevent that > kind of event in local project testing, unit or functional. Basically an > unknown contract was broken. We should bring that contract back into the > project itself, or yell at the consuming project about why they were > using code in a crazy pants way. > > Additionally, I'd like us to consider: No more alpha libraries. The > moment we've bumped global requirements in projects we've actually > released these libraries to production, as people are CDing the servers. > We should just be honest about that and just give things a real version. > Version numbers are cheap. > > -Sean > -- Sean Dague http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On 30 September 2014 08:14, Doug Hellmann wrote: > > On Sep 29, 2014, at 4:22 PM, Robert Collins > wrote: > > > As far as pip goes, you may not know, but tox defaults to pip --pre, > > which means anyone using tox, like us all here, will be pulling the > > alphas down by default: so impacting folk doing e.g. devtest on > > icehouse. So I don't think the hack is working as intended. > > I dislike tox more every day. Is this why we have the installation command > override set in tox.ini? > Yes. This issue in the tox tracker has some background - feel free to upvote it! https://bitbucket.org/hpk42/tox/issue/193/remove-the-pre-pip-option-by-default Richard ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On Sep 29, 2014, at 4:22 PM, Robert Collins wrote: > On 30 September 2014 03:10, Doug Hellmann wrote: >> >> On Sep 28, 2014, at 5:00 PM, Robert Collins >> wrote: > >> As far as I know, the client libraries aren’t being released as alphas. The >> Oslo libraries are, but they aren’t “public” in the same sense — they’re an >> internal part of OpenStack, not something a user of a cloud is going to be >> interacting with. The parts that affect the deployers directly are usually >> limited to configuration options, and we have a strict deprecation policy >> for dealing with those, even within the development cycle. > > I'm now really confused. oslo.config for instance - its depended on by > the client libraries. Our strict backwards compatibility guarantees > for the clients are going to be transitive across our dependencies, > no? I would expect anyone using an experimental feature to be working with us on it, and to avoid releasing something “experimental" into a “stable" release stream. We do that automatically with the server projects because, irrespective of the fact that some people deploy from trunk, we create a formal release of the server and libraries at the end of a development cycle at which point the library’s API is declared stable. > >> We do reserve the right to change APIs for new features being added to the >> Oslo libraries during a development cycle. Because we want to support CD, >> and because of the way we gate test, those changes have to be able to roll >> out in a backwards-compatible way. (THIS is why the incubator is important >> for API evolution, by the way, because it mean the API of a module can >> change and not break any updates in any consuming project or CD >> environment.) Even if we change the way we gate test the libraries, we would >> still want to allow for backwards-compatibility, but still only within a >> development cycle. We do not want to support every API variation of every >> library for all time. If we have to tweak something, we try to get the >> consuming projects updated within a cycle so the old variation of the new >> feature can be removed. > > If we are only backwards compatible within a development cycle, we > can't use the new version of e.g. oslo.db with the stable icehouse API > servers. That means that distributors can't just distribute oslo.db... > they have to have very fixed sets of packages lockstepped together, > which seems unpleasant and fragile. Isn't that what the incubator was > for: that we release things from it once we're *ready* to do backwards > compatibility? We do not want anyone to use development versions of libraries for stable branches, right? We have processes in place to allow us to backport changes to patch releases of libraries, so that bug and security fixes can be released without the features being developed within the cycle. But deployers using stable branches should avoid mixing in development versions of libraries. As far as the incubator goes, we’ve been baking things there for some time. Much of the code is now ready to graduate, and some of it will require API changes as part of that process. Because we’re seeing more and more people just flatly refuse to work with incubated code at all, we are going to make some of those changes as we create the libraries. That means application changes during library adoption, but it also means fewer syncs, and those seem to be the rage trigger. We are trying to minimize API changes, but some are just unavoidable (circular dependencies, hidden globals, etc. are things we won’t support). Based on our experiences in Juno, I expect we’ll continue to get a few things wrong in the API changes, but that they will settle down after 1-2 cycles after a library graduates, depending on how well early adoption goes. In the future, I hope we can continue to use the incubator to avoid a lot of breaking churn. It does not seem like a community preference, though, and so the trade-off is less stability within each library as we figure out what the new APIs should look like. Again, we’re trying to minimize that, but the level of stability you seem to be asking for within a development cycle is beyond what I think we’re practically capable of providing while still maintaining the ability to actually graduate and improve the libraries. > >> Now, we’re not perfect and sometimes this doesn’t happen exactly as planned, >> but the intent is there. >> >> So I think we are actually doing all of the things you are asking us to do, >> with the exception of using the word “alpha” in the release version, and >> I’ve already given the technical reasons for doing that. > > As far as pip goes, you may not know, but tox defaults to pip --pre, > which means anyone using tox, like us all here, will be pulling the > alphas down by default: so impacting folk doing e.g. devtest on > icehouse. So I don't think the hack is working as intended. I dislike tox m
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On 30 September 2014 03:10, Doug Hellmann wrote: > > On Sep 28, 2014, at 5:00 PM, Robert Collins wrote: > As far as I know, the client libraries aren’t being released as alphas. The > Oslo libraries are, but they aren’t “public” in the same sense — they’re an > internal part of OpenStack, not something a user of a cloud is going to be > interacting with. The parts that affect the deployers directly are usually > limited to configuration options, and we have a strict deprecation policy for > dealing with those, even within the development cycle. I'm now really confused. oslo.config for instance - its depended on by the client libraries. Our strict backwards compatibility guarantees for the clients are going to be transitive across our dependencies, no? > We do reserve the right to change APIs for new features being added to the > Oslo libraries during a development cycle. Because we want to support CD, and > because of the way we gate test, those changes have to be able to roll out in > a backwards-compatible way. (THIS is why the incubator is important for API > evolution, by the way, because it mean the API of a module can change and not > break any updates in any consuming project or CD environment.) Even if we > change the way we gate test the libraries, we would still want to allow for > backwards-compatibility, but still only within a development cycle. We do not > want to support every API variation of every library for all time. If we have > to tweak something, we try to get the consuming projects updated within a > cycle so the old variation of the new feature can be removed. If we are only backwards compatible within a development cycle, we can't use the new version of e.g. oslo.db with the stable icehouse API servers. That means that distributors can't just distribute oslo.db... they have to have very fixed sets of packages lockstepped together, which seems unpleasant and fragile. Isn't that what the incubator was for: that we release things from it once we're *ready* to do backwards compatibility? > Now, we’re not perfect and sometimes this doesn’t happen exactly as planned, > but the intent is there. > > So I think we are actually doing all of the things you are asking us to do, > with the exception of using the word “alpha” in the release version, and I’ve > already given the technical reasons for doing that. As far as pip goes, you may not know, but tox defaults to pip --pre, which means anyone using tox, like us all here, will be pulling the alphas down by default: so impacting folk doing e.g. devtest on icehouse. So I don't think the hack is working as intended. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On Sep 28, 2014, at 5:00 PM, Robert Collins wrote: > On 27 September 2014 10:07, Robert Collins wrote: > >> TripleO has been running pip releases of clients in servers from the >> get go, and I've lost track of the number of bad dependency bugs we've >> encounted. We've hit many more of those than bad releases that crater >> the world [though those have happened]. >> >> And yes, alpha dependencies are a mistake - if we depend on it, its a >> release. Quod erat demonstratum. > > Doug has pointed out to me that this is perhaps a little shallow :). > > So let me try and sketch a little deeper. > > TripleO's upstream CI looks similar (if more aggressive) to the CD > deploy process some of our major contributors are using: we take > trunk, wrap it up into a production config and deploy it. There are > two major differences vis-a-vis what HP Cloud or Rackspace are doing. > Firstly, they're running off a fork which they sync at high frequency > - which exists to deal with the second thing, which is that they run > deployment specific tests against their tree before deployment, and > when that fails, they fix it simultaneously upstream and in the fork. > > So, more or less *every commit* in Nova, Cinder, etc is going into a > production release in a matter of days. From our hands to our users. > > What TripleO does - and I don't know the exact detail for Rackspace or > HP Cloud to say if this is the same) - is that we're driven by > requirements.txt files + what happens when things break. > > So if requirements.txt specifies a thing thats not on PyPI, that > causes things to break : we -really- don't like absolute URLs in > requirements.txt, and we -really- don't like having stale requirements > in requirements.txt. > > The current situation where (most) everything we consume is on PyPI is > a massive step forwards. Yay. > > But if requirements.txt specifies a library thats not released, thats > a little odd. It's odd because we're saying that each commit of the > API servers is at release quality (but we don't release because for > these big projects a release is much more than just quality - its > ecosystem, its documentation, its deployment support)... > > The other way things can be odd is if requirements.txt is stale: e.g. > say cinderclient adds an API to make life easier in Nova. If Nova > wants to use that, they could just use it - it will pass the > integrated gate which tests tip vs tip. But it will fail if any user > just 'pip install' installs Nova. It will fail for developers too. So > I think its far better to publish that cinderclient on PyPI so that > things do work. > > And here is where the discussion about alphas comes in. > > Should we publish that cinderclient as a release, or as a pre-release preview? > > If we publish it as a pre-release preview, we're saying that we > reserve the right to change the API anyway we want. We're not saying > that its not release quality: because the API servers can't be at > release quality if they depend on non-release quality components. > > And yet, we /cannot/ change that new cinderclient API anyway we want. > Deployers have already deployed the API server that depends on it; > they are pulling from pypi: if we push up a breaking cinderclient > alpha-2 or whatever, it will break our deployers. > > Thats why I say that if we depend on it, its released: because in all > ways that matter, the constraints that one expects to see on a full > release, apply to *every commit* we do within the transitive > dependency tree that is the integrated gate. > > And this isn't because we test together - its because the top level > drivers for that gate are the quality of the API server trees, which > are CD deployed. The testing strategy doesn't matter so much compared > to that basic starting point. > > To summarise the requirements I believe we have are: > - every commit of API servers is production quality and release-ready > [for existing features] > - we don't break public APIs in projects at the granularity of consumption > - Thats per commit for API servers, and > per-whatever-pip-grabs-when-installing-api-servers for library > projects > (e.g. per-release) > - requirements.txt should be pip installable at all times > - and faithfully represent actual dependencies: nothing should break > if one pip installs e.g. nova from git > > Requirements we *do not have*: > - new features within a project have to be production quality on day one > That is, projects are allowed to have code that isn't yet > supported, though for instance we don't have a good way to annotate > that X 'will be a public API but is not yet'. > > > So the alpha thing is a mistake IMO not because we're pushing to PyPI, > but because we're calling them alphas, which I don't believe > represents the actual state of the code nor our ability to alter > things. > > Right now we test tip vs tip, so some of this is hidden until it > breaks TripleO [which is more often than
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On 27 September 2014 10:07, Robert Collins wrote: > TripleO has been running pip releases of clients in servers from the > get go, and I've lost track of the number of bad dependency bugs we've > encounted. We've hit many more of those than bad releases that crater > the world [though those have happened]. > > And yes, alpha dependencies are a mistake - if we depend on it, its a > release. Quod erat demonstratum. Doug has pointed out to me that this is perhaps a little shallow :). So let me try and sketch a little deeper. TripleO's upstream CI looks similar (if more aggressive) to the CD deploy process some of our major contributors are using: we take trunk, wrap it up into a production config and deploy it. There are two major differences vis-a-vis what HP Cloud or Rackspace are doing. Firstly, they're running off a fork which they sync at high frequency - which exists to deal with the second thing, which is that they run deployment specific tests against their tree before deployment, and when that fails, they fix it simultaneously upstream and in the fork. So, more or less *every commit* in Nova, Cinder, etc is going into a production release in a matter of days. From our hands to our users. What TripleO does - and I don't know the exact detail for Rackspace or HP Cloud to say if this is the same) - is that we're driven by requirements.txt files + what happens when things break. So if requirements.txt specifies a thing thats not on PyPI, that causes things to break : we -really- don't like absolute URLs in requirements.txt, and we -really- don't like having stale requirements in requirements.txt. The current situation where (most) everything we consume is on PyPI is a massive step forwards. Yay. But if requirements.txt specifies a library thats not released, thats a little odd. It's odd because we're saying that each commit of the API servers is at release quality (but we don't release because for these big projects a release is much more than just quality - its ecosystem, its documentation, its deployment support)... The other way things can be odd is if requirements.txt is stale: e.g. say cinderclient adds an API to make life easier in Nova. If Nova wants to use that, they could just use it - it will pass the integrated gate which tests tip vs tip. But it will fail if any user just 'pip install' installs Nova. It will fail for developers too. So I think its far better to publish that cinderclient on PyPI so that things do work. And here is where the discussion about alphas comes in. Should we publish that cinderclient as a release, or as a pre-release preview? If we publish it as a pre-release preview, we're saying that we reserve the right to change the API anyway we want. We're not saying that its not release quality: because the API servers can't be at release quality if they depend on non-release quality components. And yet, we /cannot/ change that new cinderclient API anyway we want. Deployers have already deployed the API server that depends on it; they are pulling from pypi: if we push up a breaking cinderclient alpha-2 or whatever, it will break our deployers. Thats why I say that if we depend on it, its released: because in all ways that matter, the constraints that one expects to see on a full release, apply to *every commit* we do within the transitive dependency tree that is the integrated gate. And this isn't because we test together - its because the top level drivers for that gate are the quality of the API server trees, which are CD deployed. The testing strategy doesn't matter so much compared to that basic starting point. To summarise the requirements I believe we have are: - every commit of API servers is production quality and release-ready [for existing features] - we don't break public APIs in projects at the granularity of consumption - Thats per commit for API servers, and per-whatever-pip-grabs-when-installing-api-servers for library projects (e.g. per-release) - requirements.txt should be pip installable at all times - and faithfully represent actual dependencies: nothing should break if one pip installs e.g. nova from git Requirements we *do not have*: - new features within a project have to be production quality on day one That is, projects are allowed to have code that isn't yet supported, though for instance we don't have a good way to annotate that X 'will be a public API but is not yet'. So the alpha thing is a mistake IMO not because we're pushing to PyPI, but because we're calling them alphas, which I don't believe represents the actual state of the code nor our ability to alter things. Right now we test tip vs tip, so some of this is hidden until it breaks TripleO [which is more often than we'd like!] but the incremental, don't break things approach is enforced by the gate, which is good :). My proposal, FWIW, would be that we just call the things we push up to PyPI releases. We've already tested the code in fair anger in the ga
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On 27 September 2014 06:27, Sean Dague wrote: > As we've been talking about the test disaggregation the hamster wheels +1000 TripleO has been running pip releases of clients in servers from the get go, and I've lost track of the number of bad dependency bugs we've encounted. We've hit many more of those than bad releases that crater the world [though those have happened]. And yes, alpha dependencies are a mistake - if we depend on it, its a release. Quod erat demonstratum. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On Sep 26, 2014, at 4:50 PM, Sean Dague wrote: > On 09/26/2014 04:22 PM, Doug Hellmann wrote: >> >> On Sep 26, 2014, at 2:27 PM, Sean Dague wrote: >> >>> As we've been talking about the test disaggregation the hamster wheels >>> in the back of my brain have been churning on the library testing >>> problem I think we currently have. Namely, openstack components mostly >>> don't rely on released library versions, they rely on git master of >>> them. Right now olso and many clients are open masters (their >>> stable/juno versions are out), but we've still not cut RCs on servers. >>> So we're actually going to have a time rewind event as we start cutting >>> stables. >>> >>> We did this as a reaction to the fact that library releases were often >>> cratering the world. However, I think the current pattern leads us into >>> a much more dangerous world where basically the requirements.txt is invalid. >>> >>> So here is the particular unwind that I think would be useful here: >>> >>> 1) Change setup_library in devstack to be able to either setup the >>> library from git or install via pip. This would apply to all libraries >>> we are installing from oslo, the python clients, stackforge, etc. >>> Provide a mechanism to specify LIBRARIES_FROM_GIT (or something) so that >>> you can selectively decide to use libraries from git for development >>> purposes. >>> >>> 2) Default devstack to use pip released versions. >> >> These first 2 suggestions seem good. It makes our test configuration more >> accurate, and frees the Oslo team to have our dev cycle a little out of >> phase without worrying about breaking things. It means app developers will >> have to wait for fixes and features longer than they would otherwise, but >> Thierry has improved the automation for release tracking so we could release >> more often. >> >>> >>> 3) Change the job definition on the libraries to test against devstack >>> in check, not in gate. The library teams can decide if they want their >>> forward testing to be voting or not, but this is basically sniff testing >>> that when they release a new library they won't ruin the world for >>> everyone else. >> >> The other suggestions only work if we also have some job that does still use >> the source version of the libraries. A voting check job is good. Why would >> we not also do the test in the gate? > > Right, so the oslo-config devstack job sets > LIBRARIES_FROM_GIT=oslo.config. It tests with everything else from pip. > > So, honestly, we did this experiment with tempest. We voting check on a > lot more than we gate on. The only reason to rerun these same tests on > gate is if you believe compounding changes in your code can produce a > failure that was not found by testing the two patches independently. > Which, for something like Nova, has some really likelyhood. But I think > for libraries the likelihood is quite small. Small enough that you > relieve the pain of something completely outside of your control killing > a merge. > > If it was really an issue, then all future patches would fail check. So > you'd be stuck there. But that just means you just need to make that > your next fix patch. I actually think in the library case it would be a > good high throughput model. OK, that makes sense. > >>> 4) If a ruin the world event happens, figure out how to prevent that >>> kind of event in local project testing, unit or functional. Basically an >>> unknown contract was broken. We should bring that contract back into the >>> project itself, or yell at the consuming project about why they were >>> using code in a crazy pants way. >>> >>> Additionally, I'd like us to consider: No more alpha libraries. The >>> moment we've bumped global requirements in projects we've actually >>> released these libraries to production, as people are CDing the servers. >>> We should just be honest about that and just give things a real version. >>> Version numbers are cheap. >> >> Anyone following trunk is already expecting some pain, but the folks that >> follow the stable branches do it because we call them stable. We have been >> using alpha versions as a way to avoid throwing development versions of >> libraries into the stable branch test and production environments before >> they have had some runtime against master. It’s not so much an indication of >> the quality of the content of the package as a hack taking advantage of pip >> behaviors that don’t install pre-release packages unless specifically told >> do so, and a signal to distros that they don’t necessarily have to package >> those releases (I don’t know if they’re receiving that signal or not). If we >> stop using alpha version numbers, we need some other way to avoid having >> development releases end up in the stable test environments. Or to say >> explicitly that it’s OK if they do, I guess. > > So it seems to me that if you are running stable, you are probably not > going out of your way to run pip in
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On 2014-09-26 16:50:51 -0400 (-0400), Sean Dague wrote: [...] > Alternatively, we could actually semver cap on stable (yes we've > gone around this mulberry bush before, but I think it might be > time to again), so new feature releases aren't impacting stable. [...] Just to pause for a moment at that particular mulberry bush, recall that pip needs a real dependency solver and so we don't actually control the upper bounds of transitive dependencies. "That's all well and good," you say, "we can just worry about our direct dependencies and keep them all in sync in the requirements lists." But now think about how we're trying to make some of the projects we develop more broadly usable outside of OpenStack... just last week I noticed that one of our transitive dependencies uses pbr, and is not bound to the same version restrictions as dependencies we happen to develop ourselves. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On 09/26/2014 04:22 PM, Doug Hellmann wrote: > > On Sep 26, 2014, at 2:27 PM, Sean Dague wrote: > >> As we've been talking about the test disaggregation the hamster wheels >> in the back of my brain have been churning on the library testing >> problem I think we currently have. Namely, openstack components mostly >> don't rely on released library versions, they rely on git master of >> them. Right now olso and many clients are open masters (their >> stable/juno versions are out), but we've still not cut RCs on servers. >> So we're actually going to have a time rewind event as we start cutting >> stables. >> >> We did this as a reaction to the fact that library releases were often >> cratering the world. However, I think the current pattern leads us into >> a much more dangerous world where basically the requirements.txt is invalid. >> >> So here is the particular unwind that I think would be useful here: >> >> 1) Change setup_library in devstack to be able to either setup the >> library from git or install via pip. This would apply to all libraries >> we are installing from oslo, the python clients, stackforge, etc. >> Provide a mechanism to specify LIBRARIES_FROM_GIT (or something) so that >> you can selectively decide to use libraries from git for development >> purposes. >> >> 2) Default devstack to use pip released versions. > > These first 2 suggestions seem good. It makes our test configuration more > accurate, and frees the Oslo team to have our dev cycle a little out of phase > without worrying about breaking things. It means app developers will have to > wait for fixes and features longer than they would otherwise, but Thierry has > improved the automation for release tracking so we could release more often. > >> >> 3) Change the job definition on the libraries to test against devstack >> in check, not in gate. The library teams can decide if they want their >> forward testing to be voting or not, but this is basically sniff testing >> that when they release a new library they won't ruin the world for >> everyone else. > > The other suggestions only work if we also have some job that does still use > the source version of the libraries. A voting check job is good. Why would we > not also do the test in the gate? Right, so the oslo-config devstack job sets LIBRARIES_FROM_GIT=oslo.config. It tests with everything else from pip. So, honestly, we did this experiment with tempest. We voting check on a lot more than we gate on. The only reason to rerun these same tests on gate is if you believe compounding changes in your code can produce a failure that was not found by testing the two patches independently. Which, for something like Nova, has some really likelyhood. But I think for libraries the likelihood is quite small. Small enough that you relieve the pain of something completely outside of your control killing a merge. If it was really an issue, then all future patches would fail check. So you'd be stuck there. But that just means you just need to make that your next fix patch. I actually think in the library case it would be a good high throughput model. >> 4) If a ruin the world event happens, figure out how to prevent that >> kind of event in local project testing, unit or functional. Basically an >> unknown contract was broken. We should bring that contract back into the >> project itself, or yell at the consuming project about why they were >> using code in a crazy pants way. >> >> Additionally, I'd like us to consider: No more alpha libraries. The >> moment we've bumped global requirements in projects we've actually >> released these libraries to production, as people are CDing the servers. >> We should just be honest about that and just give things a real version. >> Version numbers are cheap. > > Anyone following trunk is already expecting some pain, but the folks that > follow the stable branches do it because we call them stable. We have been > using alpha versions as a way to avoid throwing development versions of > libraries into the stable branch test and production environments before they > have had some runtime against master. It’s not so much an indication of the > quality of the content of the package as a hack taking advantage of pip > behaviors that don’t install pre-release packages unless specifically told do > so, and a signal to distros that they don’t necessarily have to package those > releases (I don’t know if they’re receiving that signal or not). If we stop > using alpha version numbers, we need some other way to avoid having > development releases end up in the stable test environments. Or to say > explicitly that it’s OK if they do, I guess. So it seems to me that if you are running stable, you are probably not going out of your way to run pip install -U on all the dependencies every day. Honestly, you are probably not upgrading your dependencies unless we issue a security alert on one of them. So I'm not sure we should make the 'I'm run
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On Sep 26, 2014, at 2:27 PM, Sean Dague wrote: > As we've been talking about the test disaggregation the hamster wheels > in the back of my brain have been churning on the library testing > problem I think we currently have. Namely, openstack components mostly > don't rely on released library versions, they rely on git master of > them. Right now olso and many clients are open masters (their > stable/juno versions are out), but we've still not cut RCs on servers. > So we're actually going to have a time rewind event as we start cutting > stables. > > We did this as a reaction to the fact that library releases were often > cratering the world. However, I think the current pattern leads us into > a much more dangerous world where basically the requirements.txt is invalid. > > So here is the particular unwind that I think would be useful here: > > 1) Change setup_library in devstack to be able to either setup the > library from git or install via pip. This would apply to all libraries > we are installing from oslo, the python clients, stackforge, etc. > Provide a mechanism to specify LIBRARIES_FROM_GIT (or something) so that > you can selectively decide to use libraries from git for development > purposes. > > 2) Default devstack to use pip released versions. These first 2 suggestions seem good. It makes our test configuration more accurate, and frees the Oslo team to have our dev cycle a little out of phase without worrying about breaking things. It means app developers will have to wait for fixes and features longer than they would otherwise, but Thierry has improved the automation for release tracking so we could release more often. > > 3) Change the job definition on the libraries to test against devstack > in check, not in gate. The library teams can decide if they want their > forward testing to be voting or not, but this is basically sniff testing > that when they release a new library they won't ruin the world for > everyone else. The other suggestions only work if we also have some job that does still use the source version of the libraries. A voting check job is good. Why would we not also do the test in the gate? > > 4) If a ruin the world event happens, figure out how to prevent that > kind of event in local project testing, unit or functional. Basically an > unknown contract was broken. We should bring that contract back into the > project itself, or yell at the consuming project about why they were > using code in a crazy pants way. > > Additionally, I'd like us to consider: No more alpha libraries. The > moment we've bumped global requirements in projects we've actually > released these libraries to production, as people are CDing the servers. > We should just be honest about that and just give things a real version. > Version numbers are cheap. Anyone following trunk is already expecting some pain, but the folks that follow the stable branches do it because we call them stable. We have been using alpha versions as a way to avoid throwing development versions of libraries into the stable branch test and production environments before they have had some runtime against master. It’s not so much an indication of the quality of the content of the package as a hack taking advantage of pip behaviors that don’t install pre-release packages unless specifically told do so, and a signal to distros that they don’t necessarily have to package those releases (I don’t know if they’re receiving that signal or not). If we stop using alpha version numbers, we need some other way to avoid having development releases end up in the stable test environments. Or to say explicitly that it’s OK if they do, I guess. I would also like to hear some from the distro packaging folks about what it would mean for us to be releasing regular version libraries frequently (expect at least one release every week for the entire cycle). Would you package all of those, or would you wait until closer to the end of the cycle and package the “final” versions? Doug > > -Sean > > -- > Sean Dague > http://dague.net > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] concrete proposal on changing the library testing model with devstack-gate
On 2014-09-26 14:27:51 -0400 (-0400), Sean Dague wrote: > As we've been talking about the test disaggregation the hamster wheels > in the back of my brain have been churning on the library testing > problem I think we currently have. Namely, openstack components mostly > don't rely on released library versions, they rely on git master of > them. Right now olso and many clients are open masters (their > stable/juno versions are out), but we've still not cut RCs on servers. > So we're actually going to have a time rewind event as we start cutting > stables. [...] Agreed, I've brought this up a few times as well. Basically right now we have jobs which test things with branch tips of dependencies we also, as a community, develop--but we lack any tests against release versions. There is the very real risk that we could release a server which depends on unreleased features in one of its dependencies. The outstanding feature request against Gerrit to allow code review of tags could provide an easy place to catch that, but short of it getting some traction upstream we probably need actual gate jobs which validate the "releasability" of a project on each proposed commit. > Additionally, I'd like us to consider: No more alpha libraries. > The moment we've bumped global requirements in projects we've > actually released these libraries to production, as people are > CDing the servers. We should just be honest about that and just > give things a real version. Version numbers are cheap. "Release early. Release often. And listen to your customers." -ESR -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
