On 06/18/2013 01:01 AM, Joe Gordon wrote:
Hi All,
A patch to move the run_tests.sh script into oslo-incubator
(https://review.openstack.org/#/c/32736/), has brought up the bigger
question of what is the future of './run_tests.sh.'
This seems like a topic that directly affects all developers, so it
sounded like it should be brought to the Mailing list.
Reasons to keep run_tests.sh:
* there is no possibility to run tests with testtools instead of testr.
This feature allows us to use the debugger.
* in some projects tox doesn't use a testr wrapper to report progress
while tests are running
* building the sdist can be slow (slow == noticeable)
Monty's Rant:
"So, I'm SUPER torn on this.
On the one hand, we use tox in the gate, and run_tests.sh hides that
from people and then they get confused about why some change they made
to run_tests.sh doesn't get reflected in the reality of the project.
On the other hand, what Victor says is true. We need to land a couple of
changes to upstream tox that would allow it to not run develop each time
- but we seem to have a hard time doing that.
On the third hand, we'd doing an INSANE amount of work to make working
with testing in a venv "easy" and what we've wound up with is a
situation where we have multiple competing incompatible ways of doing
things.
Here's what I do: source .tox/py27/bin/activate testr run --parallel
testr run --parallel testr run some-test testr run --failing deactivate
Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip
install -r requirements.txt -r test-requirements.txt
I believe vish just installs the depends directly on his box and skips
venvs altogether.
It's not only vish but basically how distros test their packages,
actually :-)
It turns out that testr is a powerful tool with a nice UI. Instead of
putting wrappers around it, perhaps we should add support for our output
stream thing we like to it upstream. We should also add support to it
for dropping into a debugger directly, so that the testtools issue goes
away. Then we should land a patch to upstream tox to get it to do
setup.py develop instead of sdist+ install.
If we did that, we could have "run tox" be the simple answer for anyone
who just wants to run whatever the gate runs, because that will always
work. And if they want to do fancier things, they should learn about
venvs and testr.
"run tox" would work for distros too, they only have to patch tox.ini with:
[testenv]
sitepackages = True
to avoid
However, if we ARE going to keep a run_tests.sh file in our trees, we
should certainly have a single copy and have it behave consistently.
Please note that some projects actually do just have run_tests.sh as a
thin wrapper around tox."
What do you think? Should we keep run_tests as is and port to oslo, or
should we revisit the role of run_tests.sh?
I don't think the demands for testing on the gate are necessarily the
same for developers and/or packagers. For gating, we want fast, i.e.
parallel testing. Developers want PDB and running individual tests.
That's why nosetests is and remains so popular. However, packagers hate
everything that messes up their build environment. This includes stuff
that downloads from the interwebs or ignores the underlying systems.
Thus virtualenvs, setuptools_git, sphinx.ext.intersphinx & co. are a no
go and usually patched away. "./run_tests.sh -N -P" does a good job but
"nostests" is equally suitable for package testing. By contrast, tox
still creates a venv with "sitepackages = True", which is overhead for
no gain. When you test distro packages, you don't need several testenvs,
test against the Python interpreter your packages where build against
and don't care for PEP-8.
So, as long as we make sure all popular test runners can be used, I
don't see any issue in changing defaults.
--
Sascha Peilicke
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
