Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
On 08/26/2015 09:21 PM, Robert Collins wrote: Now, the most annoying one is with testtools (ie: #796542). I'd appreciate having help on that one. Twisted's latest releases moved a private symbol that testtools unfortunately depends on. https://github.com/testing-cabal/testtools/pull/149 - and I just noticed now that Colin has added the test matrix we need, so we can merge this and get a release out this week. Hum... this is for the latest testtools, right? Could you help me with fixing testtools 0.9.39 in Sid, so that Kilo can continue to build there? Or is this too much work? Liberty will require a newer testtools, but the patch to twisted should be easily backportable - nothing else has changed near it (I just did an inspection of all the patches over the last year) - so it should trivially apply. [Except .travis.yml, which is irrelevant to Debian]. Thanks a lot for pointing to it. Indeed, the backport work was really trivial, and now testtools 0.9.39 works perfectly in Sid. \o/ There's at least one issue which I know of: Horizon in Kilo needs testtools, but it is also incompatible (ie: build-conflicts) with python-unittest2, which is now a dependency of the newer testtools. So upgrading to the latest testtools would make it impossible to build Horizon with unit tests (which are by the way broken by mock 1.3). However, I'm not aware of any API breaks in testtools 1.8.0 that would affect kilo - we run the kilo tests in CI with latest testtools release; you may need the related dependencies updated as well though Yes, which maybe is the problem (as per above). I prefer to not attempt it right now, as it would be a lot of work. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
On 27 August 2015 at 02:15, Ihar Hrachyshka ihrac...@redhat.com wrote: I really wish that nothing of this kind was even possible. Adding such an upper cap is like hiding the dust under the carpet: it doesn't remove the issue, it just hides it. We really have too much of these in OpenStack. Fixing broken tests was the correct thing to do, IMO. I think whoever is interested in raising a cap is free to send patches and get it up. I don't think those patches would be blocked. So this is the situation. In Kilo we have only one mechanism to defend against new releases that are incompatible: version caps in project trees. Its possible but ugly and prone to confusing errors, to express 'only use versions up to Y to test this'. vs 'only use versions up to Y for this'[*] In Liberty we have constraints, which apply globally across all projects and give us a safety net, this will be available on stable/liberty once its created. So lifting the caps on kilo - won't work, please don't waste your own or others time trying. Lifing the caps on liberty - won't be needed, we're not putting them in in the first place. Folk like packagers that need to know the 'tested compatible versions of dependencies' will be advised to look in upper-constraints.txt. At the M summit I intend to drive discussion about getting the libraries back to a rolling release model, now that we're de-aggregating 'The Release' we're culturally aligning with these tree boundaries being contract boundaries, and working with any older supported release across those boundaries. If we can do that then one of the major drivers around caps will cease to be relevant (which was making sure that stable/server-X works with stable/library-Y even though there are newer releases of library-X up on PyPI. This stops being important if the statement we make isn't 'here is a release as a bundle' but is instead 'server-X uses library-Y and needed version a.b.c at release time'. There are obviously CI and API evolution factors at play: its not a simple or trivial discussion, but it is one strongly implied by non-lockstep releases of the servers. (IMO at least :)). *: The pitfalls are as follows: - projects can't express different rules in install_requires vs test-requirements.txt because of the requirements syncing checks - so the requirements have to be expressed in a non-synced file - even then, they can't be presented to pip twice or it will error, so you have to ensure that none of the testing does 'pip install -r requirements.txt -r test-requirements.txt -r thisnewfile ...' or it will bail, which means 'pip install -r test-requirements.txt -r thisnewfile' and let requirements.txt come in via the egg info reflection in pbr; which means url deps (deprecated anyway) won't work. - but devs can still be confused - and ordering issues can make surprising choices about dep versions done this way due to pips resolver limitations -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
On 27 August 2015 at 01:31, Thomas Goirand z...@debian.org wrote: On 08/25/2015 11:20 PM, Robert Collins wrote: So I can't upload PBR 1.3.0 to Sid. This has been dealt with because I am the maintainer of PBR, but really, it shouldn't have happen. How come for years, upgrading PBR always worked, and suddenly, when you start contributing to it, it breaks backward compat? I'm having a hard time to understand what's the need to break something which worked perfectly for so long. I'd appreciate more details. More of the ad hominens. Robert, I'm sorry you take it this way. Sure, I was kind of frustrated to see all the added work I have for dealing with issues with the newer mock. Though I was writing to you directly as we know each other for a long time. I didn't intend this as an attack. Thank you for explaining your intent. Unfortunately it very much came across as an attack. As a data point, a number of folk reached out to me privately expressing support for me after your email: I wasn't the only person to read it as one. I'm happy to work through why it came across as one if you wish - privately or publically. I'm equally happy to just move on - feeling better now that I know it was not intended as one. As I say above, its not a PBR problem. Its badly expressed defensive dependencies in kilo's runtime requirements. Fix that, and kilo will be happy with newer pbr. That'd be too much work to patch all of kilo's requirements.txt everywhere, I'm afraid I prefer to leave things as they are. It would be substantial work yes - thats why we haven't undertaken it in the OpenStack git repos either. 6/ Build something in Debian to deal with conflicting APIs of Python packages - we can do it with C ABIs (and do, all the time), but there's no infrastructure for doing it with Python. If we had that then Debian Python maintainers could treat this as a graceful transition rather than an awkward big-bang. Even if we had such a thing (which would be great), we'd still have to deal with transitions the way it's done in C, which is hugely painful. It is, though in principle at least it can be less disruptive. I'm not sure we (wearing DD hat) have made the right tradeoffs in our approach there. But thats very close to second guessing the folk driving the transitions, so I'm going to avoid such guessing until I have the time and inclination to directly help. One can't actually know that. Because one of the bugs in 1.0.1 is that many assertions appear to work even though they don't exist: tests are ilently broken with mock 1.0.1. FYI, I'm going for the option of not running the failed tests because of what you're explaining just above. Yes, sounds fine. Now, the most annoying one is with testtools (ie: #796542). I'd appreciate having help on that one. Twisted's latest releases moved a private symbol that testtools unfortunately depends on. https://github.com/testing-cabal/testtools/pull/149 - and I just noticed now that Colin has added the test matrix we need, so we can merge this and get a release out this week. Hum... this is for the latest testtools, right? Could you help me with fixing testtools 0.9.39 in Sid, so that Kilo can continue to build there? Or is this too much work? Liberty will require a newer testtools, but the patch to twisted should be easily backportable - nothing else has changed near it (I just did an inspection of all the patches over the last year) - so it should trivially apply. [Except .travis.yml, which is irrelevant to Debian]. However, I'm not aware of any API breaks in testtools 1.8.0 that would affect kilo - we run the kilo tests in CI with latest testtools release; you may need the related dependencies updated as well though - and I'm not certain we've updated the minimum versions of deps in the testtools requirements.txt - but as long as the test suite passes I expect it will be fine. Cheers, Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 08/26/2015 03:34 PM, Thomas Goirand wrote: On 08/25/2015 08:59 PM, Matt Riedemann wrote: On 8/25/2015 10:04 AM, Thomas Goirand wrote: On 08/25/2015 03:42 PM, Thomas Goirand wrote: Hi, [...] Anyway, the result is that mock 1.3 broke 9 packages at least in Kilo, currently in Sid [1]. Maybe, as packages gets rebuilt, I'll get more bug reports. This really, is a depressing situation. [...] Some ppl on IRC explained to me what the situation was, which is that the mock API has been wrongly used, and some tests were in fact wrongly passing, so indeed, this is one of the rare cases where breaking the API probably made sense. As it doesn't bring anything to repair these tests, I'm just not running them in Kilo from now on, using something like this: --subunit 'tests\.unit\.(?!.*foo.*) Please comment if you think that's the wrong way to go. Also, has some of these been repaired in the stable/kilo branch? I seem to remember some projects backporting the test fixes to stable/kilo but ultimately we just capped mock on the stable branches to avoid this issue there. I really wish that nothing of this kind was even possible. Adding such an upper cap is like hiding the dust under the carpet: it doesn't remove the issue, it just hides it. We really have too much of these in OpenStack. Fixing broken tests was the correct thing to do, IMO. I think whoever is interested in raising a cap is free to send patches and get it up. I don't think those patches would be blocked. Ihar -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJV3coRAAoJEC5aWaUY1u57FBsH/05W80HXQRLlGARiN4K5SA8T kC8dVlKx7OPcg/XY77GMHn/oacPErXcPQFreWW1EHwFpIFePNroE1mrwZjIkgy5L ehsn/I7B3lhKLq3yqlE+MdyoeCcgXBW/Hi4DzMGEu+Os59dYc+LrO5vAjEieoU50 SsqfsHBoJo4SjtgoJdp0Q/dlaVlXuetCF5I/DWvhvJVrYuJBHIFjORTjkc6RZOOU Ke+bBRjbxJFYcTDWlE8AHzssfIDCnYlDv9+pFv+JO+tCqxIhiOraVxq+sD60fJww pExbjkZikhrRaqzzdLnYm0/ZDNzPS/UO+JSEZPFwu/pUGc7ztB/7+1PFf2oyftI= =TEyG -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
On 08/25/2015 08:59 PM, Matt Riedemann wrote: On 8/25/2015 10:04 AM, Thomas Goirand wrote: On 08/25/2015 03:42 PM, Thomas Goirand wrote: Hi, [...] Anyway, the result is that mock 1.3 broke 9 packages at least in Kilo, currently in Sid [1]. Maybe, as packages gets rebuilt, I'll get more bug reports. This really, is a depressing situation. [...] Some ppl on IRC explained to me what the situation was, which is that the mock API has been wrongly used, and some tests were in fact wrongly passing, so indeed, this is one of the rare cases where breaking the API probably made sense. As it doesn't bring anything to repair these tests, I'm just not running them in Kilo from now on, using something like this: --subunit 'tests\.unit\.(?!.*foo.*) Please comment if you think that's the wrong way to go. Also, has some of these been repaired in the stable/kilo branch? I seem to remember some projects backporting the test fixes to stable/kilo but ultimately we just capped mock on the stable branches to avoid this issue there. I really wish that nothing of this kind was even possible. Adding such an upper cap is like hiding the dust under the carpet: it doesn't remove the issue, it just hides it. We really have too much of these in OpenStack. Fixing broken tests was the correct thing to do, IMO. Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
On 8/26/2015 9:15 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 08/26/2015 03:34 PM, Thomas Goirand wrote: On 08/25/2015 08:59 PM, Matt Riedemann wrote: On 8/25/2015 10:04 AM, Thomas Goirand wrote: On 08/25/2015 03:42 PM, Thomas Goirand wrote: Hi, [...] Anyway, the result is that mock 1.3 broke 9 packages at least in Kilo, currently in Sid [1]. Maybe, as packages gets rebuilt, I'll get more bug reports. This really, is a depressing situation. [...] Some ppl on IRC explained to me what the situation was, which is that the mock API has been wrongly used, and some tests were in fact wrongly passing, so indeed, this is one of the rare cases where breaking the API probably made sense. As it doesn't bring anything to repair these tests, I'm just not running them in Kilo from now on, using something like this: --subunit 'tests\.unit\.(?!.*foo.*) Please comment if you think that's the wrong way to go. Also, has some of these been repaired in the stable/kilo branch? I seem to remember some projects backporting the test fixes to stable/kilo but ultimately we just capped mock on the stable branches to avoid this issue there. I really wish that nothing of this kind was even possible. Adding such an upper cap is like hiding the dust under the carpet: it doesn't remove the issue, it just hides it. We really have too much of these in OpenStack. Fixing broken tests was the correct thing to do, IMO. I think whoever is interested in raising a cap is free to send patches and get it up. I don't think those patches would be blocked. Ihar -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBCAAGBQJV3coRAAoJEC5aWaUY1u57FBsH/05W80HXQRLlGARiN4K5SA8T kC8dVlKx7OPcg/XY77GMHn/oacPErXcPQFreWW1EHwFpIFePNroE1mrwZjIkgy5L ehsn/I7B3lhKLq3yqlE+MdyoeCcgXBW/Hi4DzMGEu+Os59dYc+LrO5vAjEieoU50 SsqfsHBoJo4SjtgoJdp0Q/dlaVlXuetCF5I/DWvhvJVrYuJBHIFjORTjkc6RZOOU Ke+bBRjbxJFYcTDWlE8AHzssfIDCnYlDv9+pFv+JO+tCqxIhiOraVxq+sD60fJww pExbjkZikhrRaqzzdLnYm0/ZDNzPS/UO+JSEZPFwu/pUGc7ztB/7+1PFf2oyftI= =TEyG -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Right, there is a very small team of people that actually care about and try to maintain stable branches for all projects and when things are completely wedged, our first response is to get things unwedged as soon as possible. This is generally because the longer you wait to get stable branches fixes, as soon as you fix the first problem there is a new problem and you're just constantly thrashing on digging out. If there was a more community-wide concerted all-hands-on-deck effort when it comes to gate breakages and stable branch maintenance, then we would maybe be less prone to cap dependencies. -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
On 08/25/2015 11:20 PM, Robert Collins wrote: This first happened with PBR. Kilo can't use = 1.x This is due to Kilo having *inappropriate* version caps on its dependencies. Which we've been busy unwinding and fixing infrastructure this cycle to avoid having it happen again on Liberty. Thanks for doing this. I've seen *many* occurrences of what you call defensive dependencies which I had a hard time dealing with. , and Liberty can't use = 1.x. This is because pbr 1.x offers features that Liberty needs. Thats how software moves forward: you add the feature, and someone else uses it and declares a dependency on your version. Sure, no problem with this. So I can't upload PBR 1.3.0 to Sid. This has been dealt with because I am the maintainer of PBR, but really, it shouldn't have happen. How come for years, upgrading PBR always worked, and suddenly, when you start contributing to it, it breaks backward compat? I'm having a hard time to understand what's the need to break something which worked perfectly for so long. I'd appreciate more details. More of the ad hominens. Robert, I'm sorry you take it this way. Sure, I was kind of frustrated to see all the added work I have for dealing with issues with the newer mock. Though I was writing to you directly as we know each other for a long time. I didn't intend this as an attack. As I say above, its not a PBR problem. Its badly expressed defensive dependencies in kilo's runtime requirements. Fix that, and kilo will be happy with newer pbr. That'd be too much work to patch all of kilo's requirements.txt everywhere, I'm afraid I prefer to leave things as they are. But for mock, that's another story. I'm not the maintainer, and the one who is, decided it was a good moment to upload to Sid. The result is 9 FTBFS (failures to build from source) so far, because mock = 1.1 is incompatible with Kilo (but does work well with Liberty, which *requires* it). Yes, Liberty requires it because we're porting to Python3.4 and up, and mock 1.1 is incompatible with Python3.4. Oh !!! Anyway, the result is that mock 1.3 broke 9 packages at least in Kilo, currently in Sid [1]. Maybe, as packages gets rebuilt, I'll get more bug reports. This really, is a depressing situation. Now, as the package maintainer for the failed packages, I have 4 solutions: 1/ Reassign these bugs to python-mock. 2/ Remove all of the unit tests which are currently failing because of the new python-mock version. This isn't great, but as I already ran these tests with mock 1.0.1, it should be ok. 3/ Completely remove unit tests for these Kilo packages (or at least allow them to fail). 4/ See what's been done in Liberty to fix these tests with the newer version of mock, and backport that to Kilo. 5/ update OpenStack in unstable to be Liberty This isn't an option: I haven't tested Liberty b2 enough, and b3 is soon out. I prefer to upload to Sid whatever is the latest stable, meaning I prefer to keep Liberty in Experimental until the final release, and leave the stable Kilo in Sid. 6/ Build something in Debian to deal with conflicting APIs of Python packages - we can do it with C ABIs (and do, all the time), but there's no infrastructure for doing it with Python. If we had that then Debian Python maintainers could treat this as a graceful transition rather than an awkward big-bang. Even if we had such a thing (which would be great), we'd still have to deal with transitions the way it's done in C, which is hugely painful. Unfortunately, 4/ isn't practical, because I'm also maintaining backports to Jessie, which means I'd have to write fixes so that the packages would work for both mock 1.0.1 and 1.3, plus it would take a very large amount of my time in a non-useful way (I know the package works as it passed unit tests with 1.0.1, so just fixing the tests is useless). One can't actually know that. Because one of the bugs in 1.0.1 is that many assertions appear to work even though they don't exist: tests are ilently broken with mock 1.0.1. FYI, I'm going for the option of not running the failed tests because of what you're explaining just above. So I'm left with either option 2/ and 3/. But really, I'd have preferred if mock didn't break things... :/ Now, the most annoying one is with testtools (ie: #796542). I'd appreciate having help on that one. Twisted's latest releases moved a private symbol that testtools unfortunately depends on. https://github.com/testing-cabal/testtools/pull/149 - and I just noticed now that Colin has added the test matrix we need, so we can merge this and get a release out this week. Hum... this is for the latest testtools, right? Could you help me with fixing testtools 0.9.39 in Sid, so that Kilo can continue to build there? Or is this too much work? I hope the message is heard and that it wont happen again. I certainly hope we won't have an email thread like this again :) Well, if I get a
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
On 08/25/2015 03:42 PM, Thomas Goirand wrote: Hi, [...] Anyway, the result is that mock 1.3 broke 9 packages at least in Kilo, currently in Sid [1]. Maybe, as packages gets rebuilt, I'll get more bug reports. This really, is a depressing situation. [...] Some ppl on IRC explained to me what the situation was, which is that the mock API has been wrongly used, and some tests were in fact wrongly passing, so indeed, this is one of the rare cases where breaking the API probably made sense. As it doesn't bring anything to repair these tests, I'm just not running them in Kilo from now on, using something like this: --subunit 'tests\.unit\.(?!.*foo.*) Please comment if you think that's the wrong way to go. Also, has some of these been repaired in the stable/kilo branch? Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
On 8/25/2015 10:04 AM, Thomas Goirand wrote: On 08/25/2015 03:42 PM, Thomas Goirand wrote: Hi, [...] Anyway, the result is that mock 1.3 broke 9 packages at least in Kilo, currently in Sid [1]. Maybe, as packages gets rebuilt, I'll get more bug reports. This really, is a depressing situation. [...] Some ppl on IRC explained to me what the situation was, which is that the mock API has been wrongly used, and some tests were in fact wrongly passing, so indeed, this is one of the rare cases where breaking the API probably made sense. As it doesn't bring anything to repair these tests, I'm just not running them in Kilo from now on, using something like this: --subunit 'tests\.unit\.(?!.*foo.*) Please comment if you think that's the wrong way to go. Also, has some of these been repaired in the stable/kilo branch? I seem to remember some projects backporting the test fixes to stable/kilo but ultimately we just capped mock on the stable branches to avoid this issue there. Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Matt Riedemann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
On 26 August 2015 at 03:04, Thomas Goirand z...@debian.org wrote: As it doesn't bring anything to repair these tests, I'm just not running them in Kilo from now on, using something like this: --subunit 'tests\.unit\.(?!.*foo.*) --subunit is for controlled test output logic - but a negative lookahead regex is the right way to disable known buggy tests, yes. Please comment if you think that's the wrong way to go. Also, has some of these been repaired in the stable/kilo branch? Few to none, but I think they'd be valid to backport. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
I feel your pain and where you are coming from. We're all of us trying to make things better in the world - OpenStack, Debian, Python and so on. And I understand the limitations that Debian has vs the upstream Python ecosystem, and the challenges you face working in that environment. But having said that, the tone and content of your email felt quite accusatory to me and thats unnecessary - it is to be blunt, nasty. We can do much better than that when resolving conflicts. I would greatly appreciate it if, should future conflicts come up, that you do so. I have seen your follow up email, but since you've raised the points here I feel compelled to proffer a different explanation of the issues you're reporting. On 26 August 2015 at 01:42, Thomas Goirand z...@debian.org wrote: Hi, This is a special message for Robert Collins, as I believe he's the one responsible for the breakage. If it's not your fault, then I'm sorry, and whoever does the breakage should read what's below carefully, so that it doesn't happen again. meta: I find it hard to read emails 'written specially to me because I broke something' in a calm and dispassionate manner. Whether I did it or not, it predisposes me to defensiveness and raises my heart rate and blood pressure. E.g. adrenaline. I think for my own health I'm going to add a rule to killfile such mailers in the future: life is too short. If someone wants to do a postmortem on something I'm involved in - great. If they decide to open with such a biased, blame based approach, then I'm not interested. - Ok, so onto the body. The mock API was broken by changes to the copy in the CPython standard library during 3.4 and 3.5. I and others worked hard to remediate the feature limits introduced by those changes when they were discovered by the backporting process to 'mock' in 1.1 and above. mock 1.1 was a minor version change rather than a major version change because at the time of the initial sync I did not realise how widespread the impact of the changes from the stdlb would be. I had personally reviewed them all, including test changes, and none seemed contentious. I was wrong. The Python users of the internet have already told me this in technicolour, with diagrams. However, having incurred that cost, we haven't had any /new/ gratuitous incompatibilities added, and we've rolled back the really big ones - by fixing the stdlib library to make it better. Except for the bad assert detection one - see below. Robert, while I do appreciate all of your work, and your technically sound contributions, I am having a hard time with your habit to regularly break backward AND forward API compatibility. Yes, sometimes we unfortunately must do it. But this should be a very rare exception, and you've been doing it over and over again, making package maintainer's life miserable. Mock has been an exceptional case in my experience. But where else have I done this? Backwards compat is 'deal with older inputs', which pbr does just fine for *defined inputs*. Mock 1.3 is also backwards compatible with *defined older inputs*. The one case where its not, assert methods on mocks that were not defined, has been hugely contentious, possibly burnt out a cPython code dev (who quite literally said 'I'm outta here' in the 100 message mailing thread about it), and has been widely *welcomed* by users because it finds actual genuine bugs in their test suite. Forwards compat is 'deal with newer inputs gracefully'. Both pbr and mock accept newer inputs gracefully: they error and callers can use that to detect an old version and provide whatsoever fallback they like. Just like handling of epoll on Linux versions that don't have it. This first happened with PBR. Kilo can't use = 1.x This is due to Kilo having *inappropriate* version caps on its dependencies. Which we've been busy unwinding and fixing infrastructure this cycle to avoid having it happen again on Liberty. The errors from projects in kilo running with pbr = 1.x are due to pkg_resources entry_points validating the declared dependencies from the package, and the packages having a pbr1 *defensive dependency*. This is not recognised as a mistaken pattern - see the requirements management spec where we're trying to avoid it. pbr's Python API and packaging behaviour is entirely compatible with all of kilo. The things that pbr 0.11 accepts and pbr 1.0+ doesn't is an empty set to the best of my knowledge. , and Liberty can't use = 1.x. This is because pbr 1.x offers features that Liberty needs. Thats how software moves forward: you add the feature, and someone else uses it and declares a dependency on your version. So I can't upload PBR 1.3.0 to Sid. This has been dealt with because I am the maintainer of PBR, but really, it shouldn't have happen. How come for years, upgrading PBR always worked, and suddenly, when you start contributing to it, it breaks backward compat? I'm having a hard time to understand what's the
[openstack-dev] mock 1.3 breaking all of Kilo in Sid (and other cases of this kind)
Hi, This is a special message for Robert Collins, as I believe he's the one responsible for the breakage. If it's not your fault, then I'm sorry, and whoever does the breakage should read what's below carefully, so that it doesn't happen again. Robert, while I do appreciate all of your work, and your technically sound contributions, I am having a hard time with your habit to regularly break backward AND forward API compatibility. Yes, sometimes we unfortunately must do it. But this should be a very rare exception, and you've been doing it over and over again, making package maintainer's life miserable. This first happened with PBR. Kilo can't use = 1.x, and Liberty can't use = 1.x. So I can't upload PBR 1.3.0 to Sid. This has been dealt with because I am the maintainer of PBR, but really, it shouldn't have happen. How come for years, upgrading PBR always worked, and suddenly, when you start contributing to it, it breaks backward compat? I'm having a hard time to understand what's the need to break something which worked perfectly for so long. I'd appreciate more details. But for mock, that's another story. I'm not the maintainer, and the one who is, decided it was a good moment to upload to Sid. The result is 9 FTBFS (failures to build from source) so far, because mock = 1.1 is incompatible with Kilo (but does work well with Liberty, which *requires* it). I am currently unsure why the maintainer of mock uploaded to Sid and not to experimental. What needs to be considered is that mock is used not only by OpenStack. Here's the result of an apt-rdepends -r python-mock in Sid, today: Reverse Depends: atheist (0.20110402-2.1) Reverse Depends: python-cookiecutter (1.0.0-2) Reverse Depends: python-jsonschema (2.4.0-1) Reverse Depends: python-lamson (1.0pre11-1.1) Reverse Depends: python-matplotlib (1.4.2-3.1) Reverse Depends: python-mockldap (0.2.5-1) Reverse Depends: python-model-mommy (1.2-1) Reverse Depends: python-oslo.versionedobjects (0.1.1-2) Reverse Depends: python-oslotest (= 1.5.1-1) Reverse Depends: python-responses (0.3.0-1) Reverse Depends: python-softlayer (4.0.4-1) Reverse Depends: python-vcr (1.6.1-1) Clearly, we're not alone using mock. And we should always consider that we aren't alone. So the usual yeah, but we have pinned the versions, so it's Debian's fault to have uploaded version 1.3 in Sid would be very naive in this case, and absolutely not valid. This is an ok-ish answer for OpenStack only components like Oslo libraries. And even so, I'm convince that we shouldn't break APIs there either. So the issue here, really, is backward and forward compatibility breakage in mock. Robert, you're a DD and you've been working for Canonical, so you must know about these. You just need to care more for this type of things. In the Linux kernel development space, they *never* break userland as a rule. Why are Python developers allowing themselves to do so? Worse case if we really want to break things: isn't there ways to keep the old API and write a new one, let everyone migrate, then eventually deprecate the old one? Anyway, the result is that mock 1.3 broke 9 packages at least in Kilo, currently in Sid [1]. Maybe, as packages gets rebuilt, I'll get more bug reports. This really, is a depressing situation. Now, as the package maintainer for the failed packages, I have 4 solutions: 1/ Reassign these bugs to python-mock. 2/ Remove all of the unit tests which are currently failing because of the new python-mock version. This isn't great, but as I already ran these tests with mock 1.0.1, it should be ok. 3/ Completely remove unit tests for these Kilo packages (or at least allow them to fail). 4/ See what's been done in Liberty to fix these tests with the newer version of mock, and backport that to Kilo. In the case of 1/, I don't think the python-mock package maintainer will be able to do anything about it, and eventually, python-mock will get AUTORM from Debian testing, which doesn't help me at all. Unfortunately, 4/ isn't practical, because I'm also maintaining backports to Jessie, which means I'd have to write fixes so that the packages would work for both mock 1.0.1 and 1.3, plus it would take a very large amount of my time in a non-useful way (I know the package works as it passed unit tests with 1.0.1, so just fixing the tests is useless). So I'm left with either option 2/ and 3/. But really, I'd have preferred if mock didn't break things... :/ Now, the most annoying one is with testtools (ie: #796542). I'd appreciate having help on that one. I hope the message is heard and that it wont happen again. Cheers, Thomas Goirand (zigo) [1] https://bugs.debian.org/795128 [src:python-barbicanclient] python-barbicanclient: FTBFS: test_delete_checks_status_code: AttributeError: assert_called https://bugs.debian.org/795587 [src:python-heatclient] python-heatclient: FTBFS: AttributeError: assert_called_once https://bugs.debian.org/795588 [src:python-glance-store]
