On 8/22/2014 12:22 PM, Clark Boylan wrote:
On Fri, Aug 22, 2014, at 05:55 AM, Ihar Hrachyshka wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hi all,
this week is quite bumpy for unit testing in gate. First, it was
upgrade to new 'tox' version that broke quite some branches.
I did a ton of work to make the tox upgrade go smoothly because we knew
it would be somewhat painful. About a month ago I sent mail to this list
[0] describing the problem. This thread included a pointer to the bug
filed to track this [1] and example work around changes [2] which I
wrote and proposed for as many projects and branches as I had time to
test at that point.
Updating tox to 1.7.2 is important for a couple reasons. We get a lot of
confused developers wondering why using tox doesn't work to run their
tests when all of our documentation says just run tox. Well you needed a
special version (1.6.1). Communicating that to everyone that tries to
run tox is hard.
It is also important because tox adds new features like the hashseed
randomization. This is the cause of our problems but it is exposing real
bugs in openstack [3]. We should be fixing these issues and hopefully my
proposed workarounds are only temporary.
I decided to push ahead [4] and upgrade tox a couple days ago for a
couple reasons. This is an important change as illustrated above and
feature freeze and stabilization are rapidly approaching and this
probably needed to get in soon to have a chance at getting in at all. I
felt this was appropriate because I had done a ton of work prior to make
things go as smoothly as possible.
Where things did not go smoothly was on the reviews for my workaround.
Some changes were basically ignored [5] others ran into procedural
paperwork associated with stable branches that are not quite appropriate
for changes of this type [6][7]. I get that generally we only want to
backport things from master and that we have some specific way to cherry
pick things, but this type of change is to address issues with
stable/foo directly and has nothing to do with master. I did eventually
go through the "backport" dance for most of these changes despite this
not actually being a true backport.
[0]
http://lists.openstack.org/pipermail/openstack-dev/2014-July/041283.html
[1] https://bugs.launchpad.net/cinder/+bug/1348818
[2] https://review.openstack.org/#/c/109700/
[3]
http://lists.openstack.org/pipermail/openstack-dev/2014-July/041496.html
[4]
http://lists.openstack.org/pipermail/openstack-dev/2014-August/042010.html
[5] https://review.openstack.org/#/c/109749/
[6] https://review.openstack.org/#/c/109759/
[7] https://review.openstack.org/#/c/109750/
With all of that out of the way are there suggestions for how we can do
this better next time? Do we need more time (I gave us about 4 weeks
which seemed like plenty to me)? Perhaps I should send more reminder
emails? Feedback is very welcome.
Thanks,
Clark
And today new testtools 0.9.36 were released and were caught by gate,
which resulted in the following unit test failures in multiple projects:
TestCase.setUp was already called. Do not explicitly call setUp from
your tests. In your own setUp, use super to call the base setUp.
All branches are affected: havana, icehouse, and master.
This is because the following check was released with the new version
of the library:
https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517
And the temporary fix is to merge the version pin patch in global
requirements, backport it to stable branches, and merge the updates
from Openstack Proposal Bot to all affected projects. The patch for
master requirements is: https://review.openstack.org/#/c/116267/
In the meantime, projects will need to fix their tests not to call
setUp() and tearDown() twice. This will be the requirement to unpin
the version of the library.
So, please review, backport, and make sure it lands in project
requirements files.
/Ihar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
iQEcBAEBCgAGBQJT9z3cAAoJEC5aWaUY1u57DtsIAOFtK2i4zkMcC79nOrc5w9DW
oO2b064eyLwwbQEaWeeIL2JBSLBxqNV5zeN0eZB3Sq7LQLv0oPaUNTMFG2+gvask
JHCTAGKz776Rt7ptcfmpHURwcT9L//+1HXvd+ADtO0sYKwgmvaBF7aA4WFa4TseG
JCnAsi5OiOZZgTo/6U1B55srHkZr0DWxqTkKKysZJbR2Pr/ZT9io8yu9uucaz9VH
uNLfggtCcjGgccl7IqSUtVRf3lsSGuvBAxVqMszSFJQmFCjy2E26GfsTApp9KXtQ
gbCpEns8QCnt6KF9rygjHLMbYikjbITuUfSL2okZelX9VpKNx0CS29K/tRg5/BA=
=YavB
-----END PGP SIGNATURE-----
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Clark, given all the cat-herding involved I think you did a nice job. I
actually thought the tox/hashseed thing was wrapped up until the other
day when tempest/elastic-recheck were blowing up, so I guess now I know
there was an infra flip switched. Whatever, that's fine, if someone asks
"wtf is going on with these since 8/21?" someone from infra is pretty
quick to point out the change, then it's just getting people that care
enough about fixing the bugs to fix them. I don't think you or the
infra team should be responsible for that in all projects affected, it
doesn't scale.
Maybe next time something like this comes up we get the PTLs to be the
ones assigning a person (Clark's Infra Czar?!?!) responsible for
coordinating these types of changes so they are ready.
--
Thanks,
Matt Riedemann
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
