Excerpts from Sascha Peilicke's message of 2013-06-18 00:48:23 -0700: > On 06/18/2013 01:01 AM, Joe Gordon wrote: > > Hi All, > > > > A patch to move the run_tests.sh script into oslo-incubator > > (https://review.openstack.org/#/c/32736/), has brought up the bigger > > question of what is the future of './run_tests.sh.' > > > > This seems like a topic that directly affects all developers, so it > > sounded like it should be brought to the Mailing list. > > > > Reasons to keep run_tests.sh: > > > > * there is no possibility to run tests with testtools instead of testr. > > This feature allows us to use the debugger. > > * in some projects tox doesn't use a testr wrapper to report progress > > while tests are running > > * building the sdist can be slow (slow == noticeable) > > > > > > Monty's Rant: > > > > "So, I'm SUPER torn on this. > > > > On the one hand, we use tox in the gate, and run_tests.sh hides that > > from people and then they get confused about why some change they made > > to run_tests.sh doesn't get reflected in the reality of the project. > > > > On the other hand, what Victor says is true. We need to land a couple of > > changes to upstream tox that would allow it to not run develop each time > > - but we seem to have a hard time doing that. > > > > On the third hand, we'd doing an INSANE amount of work to make working > > with testing in a venv "easy" and what we've wound up with is a > > situation where we have multiple competing incompatible ways of doing > > things. > > > > Here's what I do: source .tox/py27/bin/activate testr run --parallel > > testr run --parallel testr run some-test testr run --failing deactivate > > > > Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip > > install -r requirements.txt -r test-requirements.txt > > > > I believe vish just installs the depends directly on his box and skips > > venvs altogether. > > It's not only vish but basically how distros test their packages, > actually :-) > > > It turns out that testr is a powerful tool with a nice UI. Instead of > > putting wrappers around it, perhaps we should add support for our output > > stream thing we like to it upstream. We should also add support to it > > for dropping into a debugger directly, so that the testtools issue goes > > away. Then we should land a patch to upstream tox to get it to do > > setup.py develop instead of sdist+ install. > > > > If we did that, we could have "run tox" be the simple answer for anyone > > who just wants to run whatever the gate runs, because that will always > > work. And if they want to do fancier things, they should learn about > > venvs and testr. > > "run tox" would work for distros too, they only have to patch tox.ini with: > > [testenv] > sitepackages = True
Sounds to me like tox needs a switch that runs the tests without the virtualenv. This can be done right now by running tox --showconfig and parsing that to get the commands, but something like tox --no-venv should be easy to do and useful to all. > > to avoid > > > > > However, if we ARE going to keep a run_tests.sh file in our trees, we > > should certainly have a single copy and have it behave consistently. > > > > Please note that some projects actually do just have run_tests.sh as a > > thin wrapper around tox." > > > > > > What do you think? Should we keep run_tests as is and port to oslo, or > > should we revisit the role of run_tests.sh? > > I don't think the demands for testing on the gate are necessarily the > same for developers and/or packagers. For gating, we want fast, i.e. > parallel testing. Developers want PDB and running individual tests. > That's why nosetests is and remains so popular. However, packagers hate > everything that messes up their build environment. This includes stuff > that downloads from the interwebs or ignores the underlying systems. > Thus virtualenvs, setuptools_git, sphinx.ext.intersphinx & co. are a no > go and usually patched away. "./run_tests.sh -N -P" does a good job but > "nostests" is equally suitable for package testing. By contrast, tox > still creates a venv with "sitepackages = True", which is overhead for > no gain. When you test distro packages, you don't need several testenvs, > test against the Python interpreter your packages where build against > and don't care for PEP-8. > To be clear, tox runs individual tests just fine, simply pass in a module to run. Agreed that it takes quite a bit of digging to get to a point where you can use pdb. Trying to be everything to everyone is almost never a good idea. However, what is a good idea is loosely integrating and not repeating ourselves. .testr.conf and tox.ini have all of the info necessary to run tests. If run_tests.sh does need to stay, I would propose that it parse these files as much as possible so that when the gate changes, so does the way developers run tests. _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
