Re: [openstack-dev] [all] new testtools breaking gate
On 23 August 2014 00:55, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, this week is quite bumpy for unit testing in gate. First, it was upgrade to new 'tox' version that broke quite some branches. And today new testtools 0.9.36 were released and were caught by gate, which resulted in the following unit test failures in multiple projects: TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp. All branches are affected: havana, icehouse, and master. This is because the following check was released with the new version of the library: https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517 And the temporary fix is to merge the version pin patch in global requirements, backport it to stable branches, and merge the updates from Openstack Proposal Bot to all affected projects. The patch for master requirements is: https://review.openstack.org/#/c/116267/ In the meantime, projects will need to fix their tests not to call setUp() and tearDown() twice. This will be the requirement to unpin the version of the library. So, please review, backport, and make sure it lands in project requirements files. In hindsight I should have introduced a warning for a period of time then the actual error. Sorry! -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] new testtools breaking gate
On 8/22/2014 12:22 PM, Clark Boylan wrote: On Fri, Aug 22, 2014, at 05:55 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, this week is quite bumpy for unit testing in gate. First, it was upgrade to new 'tox' version that broke quite some branches. I did a ton of work to make the tox upgrade go smoothly because we knew it would be somewhat painful. About a month ago I sent mail to this list [0] describing the problem. This thread included a pointer to the bug filed to track this [1] and example work around changes [2] which I wrote and proposed for as many projects and branches as I had time to test at that point. Updating tox to 1.7.2 is important for a couple reasons. We get a lot of confused developers wondering why using tox doesn't work to run their tests when all of our documentation says just run tox. Well you needed a special version (1.6.1). Communicating that to everyone that tries to run tox is hard. It is also important because tox adds new features like the hashseed randomization. This is the cause of our problems but it is exposing real bugs in openstack [3]. We should be fixing these issues and hopefully my proposed workarounds are only temporary. I decided to push ahead [4] and upgrade tox a couple days ago for a couple reasons. This is an important change as illustrated above and feature freeze and stabilization are rapidly approaching and this probably needed to get in soon to have a chance at getting in at all. I felt this was appropriate because I had done a ton of work prior to make things go as smoothly as possible. Where things did not go smoothly was on the reviews for my workaround. Some changes were basically ignored [5] others ran into procedural paperwork associated with stable branches that are not quite appropriate for changes of this type [6][7]. I get that generally we only want to backport things from master and that we have some specific way to cherry pick things, but this type of change is to address issues with stable/foo directly and has nothing to do with master. I did eventually go through the backport dance for most of these changes despite this not actually being a true backport. [0] http://lists.openstack.org/pipermail/openstack-dev/2014-July/041283.html [1] https://bugs.launchpad.net/cinder/+bug/1348818 [2] https://review.openstack.org/#/c/109700/ [3] http://lists.openstack.org/pipermail/openstack-dev/2014-July/041496.html [4] http://lists.openstack.org/pipermail/openstack-dev/2014-August/042010.html [5] https://review.openstack.org/#/c/109749/ [6] https://review.openstack.org/#/c/109759/ [7] https://review.openstack.org/#/c/109750/ With all of that out of the way are there suggestions for how we can do this better next time? Do we need more time (I gave us about 4 weeks which seemed like plenty to me)? Perhaps I should send more reminder emails? Feedback is very welcome. Thanks, Clark And today new testtools 0.9.36 were released and were caught by gate, which resulted in the following unit test failures in multiple projects: TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp. All branches are affected: havana, icehouse, and master. This is because the following check was released with the new version of the library: https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517 And the temporary fix is to merge the version pin patch in global requirements, backport it to stable branches, and merge the updates from Openstack Proposal Bot to all affected projects. The patch for master requirements is: https://review.openstack.org/#/c/116267/ In the meantime, projects will need to fix their tests not to call setUp() and tearDown() twice. This will be the requirement to unpin the version of the library. So, please review, backport, and make sure it lands in project requirements files. /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT9z3cAAoJEC5aWaUY1u57DtsIAOFtK2i4zkMcC79nOrc5w9DW oO2b064eyLwwbQEaWeeIL2JBSLBxqNV5zeN0eZB3Sq7LQLv0oPaUNTMFG2+gvask JHCTAGKz776Rt7ptcfmpHURwcT9L//+1HXvd+ADtO0sYKwgmvaBF7aA4WFa4TseG JCnAsi5OiOZZgTo/6U1B55srHkZr0DWxqTkKKysZJbR2Pr/ZT9io8yu9uucaz9VH uNLfggtCcjGgccl7IqSUtVRf3lsSGuvBAxVqMszSFJQmFCjy2E26GfsTApp9KXtQ gbCpEns8QCnt6KF9rygjHLMbYikjbITuUfSL2okZelX9VpKNx0CS29K/tRg5/BA= =YavB -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Clark, given all the cat-herding involved I think you did a nice job. I actually thought the tox/hashseed thing was wrapped up until the other day when tempest/elastic-recheck
[openstack-dev] [all] new testtools breaking gate
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, this week is quite bumpy for unit testing in gate. First, it was upgrade to new 'tox' version that broke quite some branches. And today new testtools 0.9.36 were released and were caught by gate, which resulted in the following unit test failures in multiple projects: TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp. All branches are affected: havana, icehouse, and master. This is because the following check was released with the new version of the library: https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517 And the temporary fix is to merge the version pin patch in global requirements, backport it to stable branches, and merge the updates from Openstack Proposal Bot to all affected projects. The patch for master requirements is: https://review.openstack.org/#/c/116267/ In the meantime, projects will need to fix their tests not to call setUp() and tearDown() twice. This will be the requirement to unpin the version of the library. So, please review, backport, and make sure it lands in project requirements files. /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT9z3cAAoJEC5aWaUY1u57DtsIAOFtK2i4zkMcC79nOrc5w9DW oO2b064eyLwwbQEaWeeIL2JBSLBxqNV5zeN0eZB3Sq7LQLv0oPaUNTMFG2+gvask JHCTAGKz776Rt7ptcfmpHURwcT9L//+1HXvd+ADtO0sYKwgmvaBF7aA4WFa4TseG JCnAsi5OiOZZgTo/6U1B55srHkZr0DWxqTkKKysZJbR2Pr/ZT9io8yu9uucaz9VH uNLfggtCcjGgccl7IqSUtVRf3lsSGuvBAxVqMszSFJQmFCjy2E26GfsTApp9KXtQ gbCpEns8QCnt6KF9rygjHLMbYikjbITuUfSL2okZelX9VpKNx0CS29K/tRg5/BA= =YavB -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] new testtools breaking gate
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 UPD: probably pinning the version is really overkill here, and instead we should fix affected branches. Among them: Havana for neutron, Havana+Icehouse+Juno for Glance. Other projects may also be affected. For Neutron, it should be handled by the following backport: https://review.openstack.org/116271 /Ihar On 22/08/14 14:55, Ihar Hrachyshka wrote: Hi all, this week is quite bumpy for unit testing in gate. First, it was upgrade to new 'tox' version that broke quite some branches. And today new testtools 0.9.36 were released and were caught by gate, which resulted in the following unit test failures in multiple projects: TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp. All branches are affected: havana, icehouse, and master. This is because the following check was released with the new version of the library: https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517 And the temporary fix is to merge the version pin patch in global requirements, backport it to stable branches, and merge the updates from Openstack Proposal Bot to all affected projects. The patch for master requirements is: https://review.openstack.org/#/c/116267/ In the meantime, projects will need to fix their tests not to call setUp() and tearDown() twice. This will be the requirement to unpin the version of the library. So, please review, backport, and make sure it lands in project requirements files. /Ihar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT90TxAAoJEC5aWaUY1u57no0IAM4XUubdTyQB4siJuUSYZSFi k3ZtgydQtdXq8hAWEUMTKqlU4hN+0qRJHK8jhXJraOfWj1KPIyzmJkZQg5UGp7WZ aIlQiQGv2BcqPf+LoGX0QVOcrZWgHpB05iE3iCBOoPJmZeGX/iGG+xV7UVwI/ekV JwjIp6SAJB4IfS3+S4rB3KUKuDx1XDBqFz40aPA7LTXNBvkQq8Xjiy0K+bb7/YC/ n7R7OUCqvbYgSQRKEADbKlFMCCl7Z1lV4XIcwHL4/cvImZxVGhl3V8rY9dTiE/cE MVdZvXWZOKRVjfZxewbqT//v96jYGX1/v+RkmP85Vq9O2HW4VV5lwK8vjxLsA7k= =XSGB -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] new testtools breaking gate
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 [resending to include stable-maint.] UPD: probably pinning the version is really overkill here, and instead we should fix affected branches. Among them: Havana for neutron, Havana+Icehouse+Juno for Glance. Other projects may also be affected. For Neutron, it should be handled by the following backport: https://review.openstack.org/116271 /Ihar On 22/08/14 14:55, Ihar Hrachyshka wrote: Hi all, this week is quite bumpy for unit testing in gate. First, it was upgrade to new 'tox' version that broke quite some branches. And today new testtools 0.9.36 were released and were caught by gate, which resulted in the following unit test failures in multiple projects: TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp. All branches are affected: havana, icehouse, and master. This is because the following check was released with the new version of the library: https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517 And the temporary fix is to merge the version pin patch in global requirements, backport it to stable branches, and merge the updates from Openstack Proposal Bot to all affected projects. The patch for master requirements is: https://review.openstack.org/#/c/116267/ In the meantime, projects will need to fix their tests not to call setUp() and tearDown() twice. This will be the requirement to unpin the version of the library. So, please review, backport, and make sure it lands in project requirements files. /Ihar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT90ZfAAoJEC5aWaUY1u572VgH/1SgYU5aBBfKUmM3JSRdrgwn kq4dN+gb0VuQtm7EXL9lIRl7riTgR/Ok/An8wuhKehEtQhXzt313oMl1vag/90WP Wc0EZhb3X9zN5E1eCDtU+F/ssaIRMmx7onqfswemfDaXRmNHlcg9gtde2L7mSlUi lHRC1xZbIF8y0wfLhhQlfKP1bjh5x1XnpbN/Q3VT2hGnRLtWYWseSQuu5r2G0T8u 6oLOJXw2sGRvi9/ploAAVSGae/q2o+RukiM5t1b+brMtbZY0bn21MmJ09Mtz3hQl ctZt8dnE+jvNKg/D49kXLsb6SEN+TgYLCEC/YdVP0MdKDlxVn4qBFcFMly7zRfY= =br27 -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] new testtools breaking gate
At least for glance, the tox fix and the double setup problem are both blocking the gate, so it isn't possible to fix cleanly, since both issues need to be fixed in one commit - I think the correct thing is to merge https://review.openstack.org/#/c/116267/ and give projects time to fix up their issues cleanly. On 22 August 2014 13:55, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, this week is quite bumpy for unit testing in gate. First, it was upgrade to new 'tox' version that broke quite some branches. And today new testtools 0.9.36 were released and were caught by gate, which resulted in the following unit test failures in multiple projects: TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp. All branches are affected: havana, icehouse, and master. This is because the following check was released with the new version of the library: https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517 And the temporary fix is to merge the version pin patch in global requirements, backport it to stable branches, and merge the updates from Openstack Proposal Bot to all affected projects. The patch for master requirements is: https://review.openstack.org/#/c/116267/ In the meantime, projects will need to fix their tests not to call setUp() and tearDown() twice. This will be the requirement to unpin the version of the library. So, please review, backport, and make sure it lands in project requirements files. /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT9z3cAAoJEC5aWaUY1u57DtsIAOFtK2i4zkMcC79nOrc5w9DW oO2b064eyLwwbQEaWeeIL2JBSLBxqNV5zeN0eZB3Sq7LQLv0oPaUNTMFG2+gvask JHCTAGKz776Rt7ptcfmpHURwcT9L//+1HXvd+ADtO0sYKwgmvaBF7aA4WFa4TseG JCnAsi5OiOZZgTo/6U1B55srHkZr0DWxqTkKKysZJbR2Pr/ZT9io8yu9uucaz9VH uNLfggtCcjGgccl7IqSUtVRf3lsSGuvBAxVqMszSFJQmFCjy2E26GfsTApp9KXtQ gbCpEns8QCnt6KF9rygjHLMbYikjbITuUfSL2okZelX9VpKNx0CS29K/tRg5/BA= =YavB -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] new testtools breaking gate
On Fri, Aug 22, 2014, at 05:55 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, this week is quite bumpy for unit testing in gate. First, it was upgrade to new 'tox' version that broke quite some branches. I did a ton of work to make the tox upgrade go smoothly because we knew it would be somewhat painful. About a month ago I sent mail to this list [0] describing the problem. This thread included a pointer to the bug filed to track this [1] and example work around changes [2] which I wrote and proposed for as many projects and branches as I had time to test at that point. Updating tox to 1.7.2 is important for a couple reasons. We get a lot of confused developers wondering why using tox doesn't work to run their tests when all of our documentation says just run tox. Well you needed a special version (1.6.1). Communicating that to everyone that tries to run tox is hard. It is also important because tox adds new features like the hashseed randomization. This is the cause of our problems but it is exposing real bugs in openstack [3]. We should be fixing these issues and hopefully my proposed workarounds are only temporary. I decided to push ahead [4] and upgrade tox a couple days ago for a couple reasons. This is an important change as illustrated above and feature freeze and stabilization are rapidly approaching and this probably needed to get in soon to have a chance at getting in at all. I felt this was appropriate because I had done a ton of work prior to make things go as smoothly as possible. Where things did not go smoothly was on the reviews for my workaround. Some changes were basically ignored [5] others ran into procedural paperwork associated with stable branches that are not quite appropriate for changes of this type [6][7]. I get that generally we only want to backport things from master and that we have some specific way to cherry pick things, but this type of change is to address issues with stable/foo directly and has nothing to do with master. I did eventually go through the backport dance for most of these changes despite this not actually being a true backport. [0] http://lists.openstack.org/pipermail/openstack-dev/2014-July/041283.html [1] https://bugs.launchpad.net/cinder/+bug/1348818 [2] https://review.openstack.org/#/c/109700/ [3] http://lists.openstack.org/pipermail/openstack-dev/2014-July/041496.html [4] http://lists.openstack.org/pipermail/openstack-dev/2014-August/042010.html [5] https://review.openstack.org/#/c/109749/ [6] https://review.openstack.org/#/c/109759/ [7] https://review.openstack.org/#/c/109750/ With all of that out of the way are there suggestions for how we can do this better next time? Do we need more time (I gave us about 4 weeks which seemed like plenty to me)? Perhaps I should send more reminder emails? Feedback is very welcome. Thanks, Clark And today new testtools 0.9.36 were released and were caught by gate, which resulted in the following unit test failures in multiple projects: TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp. All branches are affected: havana, icehouse, and master. This is because the following check was released with the new version of the library: https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517 And the temporary fix is to merge the version pin patch in global requirements, backport it to stable branches, and merge the updates from Openstack Proposal Bot to all affected projects. The patch for master requirements is: https://review.openstack.org/#/c/116267/ In the meantime, projects will need to fix their tests not to call setUp() and tearDown() twice. This will be the requirement to unpin the version of the library. So, please review, backport, and make sure it lands in project requirements files. /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJT9z3cAAoJEC5aWaUY1u57DtsIAOFtK2i4zkMcC79nOrc5w9DW oO2b064eyLwwbQEaWeeIL2JBSLBxqNV5zeN0eZB3Sq7LQLv0oPaUNTMFG2+gvask JHCTAGKz776Rt7ptcfmpHURwcT9L//+1HXvd+ADtO0sYKwgmvaBF7aA4WFa4TseG JCnAsi5OiOZZgTo/6U1B55srHkZr0DWxqTkKKysZJbR2Pr/ZT9io8yu9uucaz9VH uNLfggtCcjGgccl7IqSUtVRf3lsSGuvBAxVqMszSFJQmFCjy2E26GfsTApp9KXtQ gbCpEns8QCnt6KF9rygjHLMbYikjbITuUfSL2okZelX9VpKNx0CS29K/tRg5/BA= =YavB -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] new testtools breaking gate
On 08/22/2014 05:30 PM, Duncan Thomas wrote: At least for glance, the tox fix and the double setup problem are both blocking the gate, so it isn't possible to fix cleanly, since both issues need to be fixed in one commit - I think the correct thing is to merge https://review.openstack.org/#/c/116267/ and give projects time to fix up their issues cleanly. We merged both fixes in a single patch. Once this[0] patch lands, glance shouldn't be blocked by any of these 2 issues anymore. [0] https://review.openstack.org/#/c/109749/ Flavio On 22 August 2014 13:55, Ihar Hrachyshka ihrac...@redhat.com wrote: Hi all, this week is quite bumpy for unit testing in gate. First, it was upgrade to new 'tox' version that broke quite some branches. And today new testtools 0.9.36 were released and were caught by gate, which resulted in the following unit test failures in multiple projects: TestCase.setUp was already called. Do not explicitly call setUp from your tests. In your own setUp, use super to call the base setUp. All branches are affected: havana, icehouse, and master. This is because the following check was released with the new version of the library: https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517 And the temporary fix is to merge the version pin patch in global requirements, backport it to stable branches, and merge the updates from Openstack Proposal Bot to all affected projects. The patch for master requirements is: https://review.openstack.org/#/c/116267/ In the meantime, projects will need to fix their tests not to call setUp() and tearDown() twice. This will be the requirement to unpin the version of the library. So, please review, backport, and make sure it lands in project requirements files. /Ihar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
