Re: [openstack-dev] The future of run_tests.sh
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/21/2013 01:44 PM, Joe Gordon wrote: > It sounds like the censuses in this thread is: > > In the long run, we want to kill run_tests.sh in favor of > explaining how to use the underlying tools in a TESTING file. I agree. I'd like to add that 'long run' here is potentially a couple of cycles away. I think we definitely don't want to get rid of a thing that a project is currently using without an answer for all of its use cases. > But in the short term, we should start moving toward using a > TESTING file (such as https://review.openstack.org/#/c/33456/) but > keep run_test.sh for the time being as there are things it does > that we don't have simple ways of doing yet. Since run_tests.sh > will be around for a while it does make sense to move it into > oslo. > > > best, Joe > > > On Tue, Jun 18, 2013 at 11:44 AM, Monty Taylor > mailto:mord...@inaugust.com>> wrote: > > > > On 06/18/2013 08:44 AM, Julien Danjou wrote: >> FWIW, I think we never really had a run_tests.sh in Ceilometer >> like other projects might have, and we don't have one anymore >> for weeks, and that never looked like a problem. > >> We just rely on tox and on a good working listing in >> requirements.txt and test-requirements.txt, so you can build a >> venv yourself if you'd like. > > A couple of followups to things in this thread so far: > > - Running tests consistently both in and out of virtualenv. > > Super important. Part of the problem is that setuptools "test" > command is a broken pile of garbage. So we have a patch coming to > pbr that will sort that out - and at least as a next step, tox and > run_tests.sh can both run python setup.py test and it will work > both in and out of a venv, regardless of whether the repo uses nose > or testr. > > - Individual tests > > nose and tox and testr and run_tests.sh all support running > individual tests just fine. The invocation is slightly different > for each. For me testr is hte friendliest because it defaults to > regexes - so "testr run test_foo" will happily run > nova.tests.integration.deep_directory.foo.TestFoo.test_foo. But - > all four mechanisms work here fine. > > - pbr > > Dropping in to a debugger while running via testr is currently > problematic, but is currently on the table to be sorted. In the > meantime, the workaround is to run testtools.run directly, which > run_tests.sh does for you if you specify a single test. I think > this is probably the single greatest current reason to keep > run_tests.sh at the moment - because as much as you can learn the > cantrips around doing it, it's not a good UI. > > - nova vs. testr > > In general, things are moving towards testr being the default. I > don't think there will be anybody cutting off people's hands for > using nose, but I strongly recommend taking a second to learn testr > a bit. It's got some great features and is built on top of a > completely machine parsable test result streaming protocol, which > means we can do some pretty cool stuff with it. > > -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHElUUACgkQ2Jv7/VK1RgGMggCfYIuErSqwiCUKhgCnZKSyjVlw 2gYAoNDkQR6VP8mP2w6rGY6WwRTpOwxy =svBU -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
It sounds like the censuses in this thread is: In the long run, we want to kill run_tests.sh in favor of explaining how to use the underlying tools in a TESTING file. But in the short term, we should start moving toward using a TESTING file (such as https://review.openstack.org/#/c/33456/) but keep run_test.sh for the time being as there are things it does that we don't have simple ways of doing yet. Since run_tests.sh will be around for a while it does make sense to move it into oslo. best, Joe On Tue, Jun 18, 2013 at 11:44 AM, Monty Taylor wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > > > On 06/18/2013 08:44 AM, Julien Danjou wrote: > > FWIW, I think we never really had a run_tests.sh in Ceilometer > > like other projects might have, and we don't have one anymore for > > weeks, and that never looked like a problem. > > > > We just rely on tox and on a good working listing in > > requirements.txt and test-requirements.txt, so you can build a venv > > yourself if you'd like. > > A couple of followups to things in this thread so far: > > - - Running tests consistently both in and out of virtualenv. > > Super important. Part of the problem is that setuptools "test" command > is a broken pile of garbage. So we have a patch coming to pbr that > will sort that out - and at least as a next step, tox and run_tests.sh > can both run python setup.py test and it will work both in and out of > a venv, regardless of whether the repo uses nose or testr. > > - - Individual tests > > nose and tox and testr and run_tests.sh all support running individual > tests just fine. The invocation is slightly different for each. For me > testr is hte friendliest because it defaults to regexes - so "testr > run test_foo" will happily run > nova.tests.integration.deep_directory.foo.TestFoo.test_foo. But - all > four mechanisms work here fine. > > - - pbr > > Dropping in to a debugger while running via testr is currently > problematic, but is currently on the table to be sorted. In the > meantime, the workaround is to run testtools.run directly, which > run_tests.sh does for you if you specify a single test. I think this > is probably the single greatest current reason to keep run_tests.sh at > the moment - because as much as you can learn the cantrips around > doing it, it's not a good UI. > > - - nova vs. testr > > In general, things are moving towards testr being the default. I don't > think there will be anybody cutting off people's hands for using nose, > but I strongly recommend taking a second to learn testr a bit. It's > got some great features and is built on top of a completely machine > parsable test result streaming protocol, which means we can do some > pretty cool stuff with it. > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.12 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlHAqn4ACgkQ2Jv7/VK1RgGZ9gCdHe8AhG8uQi7nkBz1UbZHUjvJ > KskAoKddVUPBZnXAtzNpBiwazRid0gu7 > =eGE3 > -END PGP SIGNATURE- > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 06/18/2013 08:44 AM, Julien Danjou wrote: > FWIW, I think we never really had a run_tests.sh in Ceilometer > like other projects might have, and we don't have one anymore for > weeks, and that never looked like a problem. > > We just rely on tox and on a good working listing in > requirements.txt and test-requirements.txt, so you can build a venv > yourself if you'd like. A couple of followups to things in this thread so far: - - Running tests consistently both in and out of virtualenv. Super important. Part of the problem is that setuptools "test" command is a broken pile of garbage. So we have a patch coming to pbr that will sort that out - and at least as a next step, tox and run_tests.sh can both run python setup.py test and it will work both in and out of a venv, regardless of whether the repo uses nose or testr. - - Individual tests nose and tox and testr and run_tests.sh all support running individual tests just fine. The invocation is slightly different for each. For me testr is hte friendliest because it defaults to regexes - so "testr run test_foo" will happily run nova.tests.integration.deep_directory.foo.TestFoo.test_foo. But - all four mechanisms work here fine. - - pbr Dropping in to a debugger while running via testr is currently problematic, but is currently on the table to be sorted. In the meantime, the workaround is to run testtools.run directly, which run_tests.sh does for you if you specify a single test. I think this is probably the single greatest current reason to keep run_tests.sh at the moment - because as much as you can learn the cantrips around doing it, it's not a good UI. - - nova vs. testr In general, things are moving towards testr being the default. I don't think there will be anybody cutting off people's hands for using nose, but I strongly recommend taking a second to learn testr a bit. It's got some great features and is built on top of a completely machine parsable test result streaming protocol, which means we can do some pretty cool stuff with it. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHAqn4ACgkQ2Jv7/VK1RgGZ9gCdHe8AhG8uQi7nkBz1UbZHUjvJ KskAoKddVUPBZnXAtzNpBiwazRid0gu7 =eGE3 -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On 2013-06-18 08:58:21 -0700 (-0700), Clint Byrum wrote: > Excerpts from Thierry Carrez's message of 2013-06-18 01:59:57 -0700: [...] > > "tox -epy27". > > Just to clarify, you're better off running 'tox'. [...] Though it's worth noting that tox will reuse existing virtual environments to save time. This can lead to false results if requirements change between test runs. To get around that in the CI infrastructure we use a thorough invocation of git clean (which has as a side effect clearing any existing tox environments), but this is of course dangerous if you have other uncommitted files lying around in your repo you want to keep. Instead the -r option to tox will clear and recreate any virtualenv it tries to use. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
Excerpts from Sascha Peilicke's message of 2013-06-18 00:48:23 -0700: > On 06/18/2013 01:01 AM, Joe Gordon wrote: > > Hi All, > > > > A patch to move the run_tests.sh script into oslo-incubator > > (https://review.openstack.org/#/c/32736/), has brought up the bigger > > question of what is the future of './run_tests.sh.' > > > > This seems like a topic that directly affects all developers, so it > > sounded like it should be brought to the Mailing list. > > > > Reasons to keep run_tests.sh: > > > > * there is no possibility to run tests with testtools instead of testr. > > This feature allows us to use the debugger. > > * in some projects tox doesn't use a testr wrapper to report progress > > while tests are running > > * building the sdist can be slow (slow == noticeable) > > > > > > Monty's Rant: > > > > "So, I'm SUPER torn on this. > > > > On the one hand, we use tox in the gate, and run_tests.sh hides that > > from people and then they get confused about why some change they made > > to run_tests.sh doesn't get reflected in the reality of the project. > > > > On the other hand, what Victor says is true. We need to land a couple of > > changes to upstream tox that would allow it to not run develop each time > > - but we seem to have a hard time doing that. > > > > On the third hand, we'd doing an INSANE amount of work to make working > > with testing in a venv "easy" and what we've wound up with is a > > situation where we have multiple competing incompatible ways of doing > > things. > > > > Here's what I do: source .tox/py27/bin/activate testr run --parallel > > testr run --parallel testr run some-test testr run --failing deactivate > > > > Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip > > install -r requirements.txt -r test-requirements.txt > > > > I believe vish just installs the depends directly on his box and skips > > venvs altogether. > > It's not only vish but basically how distros test their packages, > actually :-) > > > It turns out that testr is a powerful tool with a nice UI. Instead of > > putting wrappers around it, perhaps we should add support for our output > > stream thing we like to it upstream. We should also add support to it > > for dropping into a debugger directly, so that the testtools issue goes > > away. Then we should land a patch to upstream tox to get it to do > > setup.py develop instead of sdist+ install. > > > > If we did that, we could have "run tox" be the simple answer for anyone > > who just wants to run whatever the gate runs, because that will always > > work. And if they want to do fancier things, they should learn about > > venvs and testr. > > "run tox" would work for distros too, they only have to patch tox.ini with: > > [testenv] > sitepackages = True Sounds to me like tox needs a switch that runs the tests without the virtualenv. This can be done right now by running tox --showconfig and parsing that to get the commands, but something like tox --no-venv should be easy to do and useful to all. > > to avoid > > > > > However, if we ARE going to keep a run_tests.sh file in our trees, we > > should certainly have a single copy and have it behave consistently. > > > > Please note that some projects actually do just have run_tests.sh as a > > thin wrapper around tox." > > > > > > What do you think? Should we keep run_tests as is and port to oslo, or > > should we revisit the role of run_tests.sh? > > I don't think the demands for testing on the gate are necessarily the > same for developers and/or packagers. For gating, we want fast, i.e. > parallel testing. Developers want PDB and running individual tests. > That's why nosetests is and remains so popular. However, packagers hate > everything that messes up their build environment. This includes stuff > that downloads from the interwebs or ignores the underlying systems. > Thus virtualenvs, setuptools_git, sphinx.ext.intersphinx & co. are a no > go and usually patched away. "./run_tests.sh -N -P" does a good job but > "nostests" is equally suitable for package testing. By contrast, tox > still creates a venv with "sitepackages = True", which is overhead for > no gain. When you test distro packages, you don't need several testenvs, > test against the Python interpreter your packages where build against > and don't care for PEP-8. > To be clear, tox runs individual tests just fine, simply pass in a module to run. Agreed that it takes quite a bit of digging to get to a point where you can use pdb. Trying to be everything to everyone is almost never a good idea. However, what is a good idea is loosely integrating and not repeating ourselves. .testr.conf and tox.ini have all of the info necessary to run tests. If run_tests.sh does need to stay, I would propose that it parse these files as much as possible so that when the gate changes, so does the way developers run tests. ___ OpenStack-dev mailing list OpenStack-dev@lists.opens
Re: [openstack-dev] The future of run_tests.sh
Excerpts from Thierry Carrez's message of 2013-06-18 01:59:57 -0700: > Joe Gordon wrote: > > Reasons to keep run_tests.sh: > > > > * there is no possibility to run tests with testtools instead of testr. > > This feature allows us to use the debugger. > > * in some projects tox doesn't use a testr wrapper to report progress > > while tests are running > > * building the sdist can be slow (slow == noticeable) > > It also makes unit testing easily discoverable. After a repo clone it's > rather obvious that run_tests.sh will run tests. If you remove it, > people have to go through some effort to discover and run "tox -epy27". > Just to clarify, you're better off running 'tox'. Not having py26 around is not an error, it will just be noted and skipped. This will also run pep8 for you which is a fairly cheap operation and will catch any broken formatting early. This works fine with a passed in test module as well (though the global pep8 is still run w/o the arg) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On 13-06-18 11:14 AM, Anita Kuno wrote: On 13-06-18 09:34 AM, Anita Kuno wrote: On 13-06-18 04:59 AM, Thierry Carrez wrote: Personally I would introduce a TESTING file that would describe all available methods for running tests, Actually I should point out that in my approach, I went with the minimum instructions to get a newcomer up and running tests across (hopefully) all projects with links that can describe all available methods for running tests, including this link: https://wiki.openstack.org/wiki/Testing My motivation is to lower the barrier to entry for new contributors and encourage them to source more efficient testing methods on their project of choice AFTER they have experienced some form of success testing OpenStack code. Commentary on the validity of my direction welcome. Thanks, Anita. and rename/move run_tests.sh to something less discoverable... so that people looking for ways to run tests would find TESTING first. Then if developers care about any alternate method they will maintain it, and if it's unmaintained it can be removed. Oh, I do like this suggestion. I know it was also suggested last night in the -infra channel so thanks to who suggested it there as well. Since I am already creating something similar for internal purposes, I can take a run at this. Perhaps I should start with one for olso? Here is my initial patch in oslo for a TESTING file: https://review.openstack.org/#/c/33456 I welcome your help to improve this patch so that it can become accepted. Thanks, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On 13-06-18 11:14 AM, John Dennis wrote: On 06/18/2013 04:59 AM, Thierry Carrez wrote: The issue here is not really the burden of maintaining an alternate testing method in the tree... it's that by default newcomers use the alternate method rather than the one used by our CI infrastructure, which should be their default choice. Personally I would introduce a TESTING file that would describe all available methods for running tests, and rename/move run_tests.sh to something less discoverable... so that people looking for ways to run tests would find TESTING first. +1 As a developer newly assigned to keystone work I will echo the value of both the run_tests.sh script because of it's obvious presence and utility. A TESTING file should be mandatory as well. I immediately found the run_tests.sh script but fumbled around for a while until I found all the information I needed to run the tests (despite already being familiar with nose). It also took me a while to figure out how to run an individual test (mostly because I assumed one had to include the tests directory in either a test pathname or module path) but run_tests.sh apparently points nose into the test directory. Had there been a TESTING readme file it would have definitely saved me time and frustration. I had to learn about tox after a suggestion in IRC. Coming from a distro perspective I prefer not to see non-distro items being installed (venv has worked well though). And I definitely like being able to drop into the debugger, nose has served me well in the past and I like it. John Would you mind making your keystone testing notes available on https://wiki.openstack.org/wiki/Testing, John? I am having a heck of a time getting keystone and keystone client tests working on a default devstack installation and would welcome your guidance if you are willing. My thanks, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On 06/18/2013 04:59 AM, Thierry Carrez wrote: > The issue here is not really the burden of maintaining an alternate > testing method in the tree... it's that by default newcomers use the > alternate method rather than the one used by our CI infrastructure, > which should be their default choice. > > Personally I would introduce a TESTING file that would describe all > available methods for running tests, and rename/move run_tests.sh to > something less discoverable... so that people looking for ways to run > tests would find TESTING first. +1 As a developer newly assigned to keystone work I will echo the value of both the run_tests.sh script because of it's obvious presence and utility. A TESTING file should be mandatory as well. I immediately found the run_tests.sh script but fumbled around for a while until I found all the information I needed to run the tests (despite already being familiar with nose). It also took me a while to figure out how to run an individual test (mostly because I assumed one had to include the tests directory in either a test pathname or module path) but run_tests.sh apparently points nose into the test directory. Had there been a TESTING readme file it would have definitely saved me time and frustration. I had to learn about tox after a suggestion in IRC. Coming from a distro perspective I prefer not to see non-distro items being installed (venv has worked well though). And I definitely like being able to drop into the debugger, nose has served me well in the past and I like it. John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On 13-06-18 09:34 AM, Anita Kuno wrote: On 13-06-18 04:59 AM, Thierry Carrez wrote: Personally I would introduce a TESTING file that would describe all available methods for running tests, and rename/move run_tests.sh to something less discoverable... so that people looking for ways to run tests would find TESTING first. Then if developers care about any alternate method they will maintain it, and if it's unmaintained it can be removed. Oh, I do like this suggestion. I know it was also suggested last night in the -infra channel so thanks to who suggested it there as well. Since I am already creating something similar for internal purposes, I can take a run at this. Perhaps I should start with one for olso? Here is my initial patch in oslo for a TESTING file: https://review.openstack.org/#/c/33456 I welcome your help to improve this patch so that it can become accepted. Thanks, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On 13-06-18 04:59 AM, Thierry Carrez wrote: Personally I would introduce a TESTING file that would describe all available methods for running tests, and rename/move run_tests.sh to something less discoverable... so that people looking for ways to run tests would find TESTING first. Then if developers care about any alternate method they will maintain it, and if it's unmaintained it can be removed. Oh, I do like this suggestion. I know it was also suggested last night in the -infra channel so thanks to who suggested it there as well. Since I am already creating something similar for internal purposes, I can take a run at this. Perhaps I should start with one for olso? Thanks, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
FWIW, I think we never really had a run_tests.sh in Ceilometer like other projects might have, and we don't have one anymore for weeks, and that never looked like a problem. We just rely on tox and on a good working listing in requirements.txt and test-requirements.txt, so you can build a venv yourself if you'd like. -- Julien Danjou ;; Free Software hacker ; freelance consultant ;; http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
Joe Gordon wrote: > Reasons to keep run_tests.sh: > > * there is no possibility to run tests with testtools instead of testr. > This feature allows us to use the debugger. > * in some projects tox doesn't use a testr wrapper to report progress > while tests are running > * building the sdist can be slow (slow == noticeable) It also makes unit testing easily discoverable. After a repo clone it's rather obvious that run_tests.sh will run tests. If you remove it, people have to go through some effort to discover and run "tox -epy27". The issue here is not really the burden of maintaining an alternate testing method in the tree... it's that by default newcomers use the alternate method rather than the one used by our CI infrastructure, which should be their default choice. Personally I would introduce a TESTING file that would describe all available methods for running tests, and rename/move run_tests.sh to something less discoverable... so that people looking for ways to run tests would find TESTING first. Then if developers care about any alternate method they will maintain it, and if it's unmaintained it can be removed. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On 06/18/2013 01:01 AM, Joe Gordon wrote: Hi All, A patch to move the run_tests.sh script into oslo-incubator (https://review.openstack.org/#/c/32736/), has brought up the bigger question of what is the future of './run_tests.sh.' This seems like a topic that directly affects all developers, so it sounded like it should be brought to the Mailing list. Reasons to keep run_tests.sh: * there is no possibility to run tests with testtools instead of testr. This feature allows us to use the debugger. * in some projects tox doesn't use a testr wrapper to report progress while tests are running * building the sdist can be slow (slow == noticeable) Monty's Rant: "So, I'm SUPER torn on this. On the one hand, we use tox in the gate, and run_tests.sh hides that from people and then they get confused about why some change they made to run_tests.sh doesn't get reflected in the reality of the project. On the other hand, what Victor says is true. We need to land a couple of changes to upstream tox that would allow it to not run develop each time - but we seem to have a hard time doing that. On the third hand, we'd doing an INSANE amount of work to make working with testing in a venv "easy" and what we've wound up with is a situation where we have multiple competing incompatible ways of doing things. Here's what I do: source .tox/py27/bin/activate testr run --parallel testr run --parallel testr run some-test testr run --failing deactivate Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip install -r requirements.txt -r test-requirements.txt I believe vish just installs the depends directly on his box and skips venvs altogether. It's not only vish but basically how distros test their packages, actually :-) It turns out that testr is a powerful tool with a nice UI. Instead of putting wrappers around it, perhaps we should add support for our output stream thing we like to it upstream. We should also add support to it for dropping into a debugger directly, so that the testtools issue goes away. Then we should land a patch to upstream tox to get it to do setup.py develop instead of sdist+ install. If we did that, we could have "run tox" be the simple answer for anyone who just wants to run whatever the gate runs, because that will always work. And if they want to do fancier things, they should learn about venvs and testr. "run tox" would work for distros too, they only have to patch tox.ini with: [testenv] sitepackages = True to avoid However, if we ARE going to keep a run_tests.sh file in our trees, we should certainly have a single copy and have it behave consistently. Please note that some projects actually do just have run_tests.sh as a thin wrapper around tox." What do you think? Should we keep run_tests as is and port to oslo, or should we revisit the role of run_tests.sh? I don't think the demands for testing on the gate are necessarily the same for developers and/or packagers. For gating, we want fast, i.e. parallel testing. Developers want PDB and running individual tests. That's why nosetests is and remains so popular. However, packagers hate everything that messes up their build environment. This includes stuff that downloads from the interwebs or ignores the underlying systems. Thus virtualenvs, setuptools_git, sphinx.ext.intersphinx & co. are a no go and usually patched away. "./run_tests.sh -N -P" does a good job but "nostests" is equally suitable for package testing. By contrast, tox still creates a venv with "sitepackages = True", which is overhead for no gain. When you test distro packages, you don't need several testenvs, test against the Python interpreter your packages where build against and don't care for PEP-8. So, as long as we make sure all popular test runners can be used, I don't see any issue in changing defaults. -- Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
Please keep in mind that not everyone uses tox also ;) Some of us use VM's instead of tox+venv and throwaway the VM's when we are done testing whatever change. VM's seem more cloudy to me ;) I personally like 'run_tests.sh' since its a easy one-stop to running tests, making me not worry about tox/testr/nose syntax (and how the backing test impl seems to change about once a year). My 2 cents. From: Joe Gordon mailto:joe.gord...@gmail.com>> Reply-To: OpenStack Development Mailing List mailto:openstack-dev@lists.openstack.org>> Date: Monday, June 17, 2013 4:01 PM To: OpenStack Development Mailing List mailto:openstack-dev@lists.openstack.org>> Subject: [openstack-dev] The future of run_tests.sh Hi All, A patch to move the run_tests.sh script into oslo-incubator (https://review.openstack.org/#/c/32736/), has brought up the bigger question of what is the future of './run_tests.sh.' This seems like a topic that directly affects all developers, so it sounded like it should be brought to the Mailing list. Reasons to keep run_tests.sh: * there is no possibility to run tests with testtools instead of testr. This feature allows us to use the debugger. * in some projects tox doesn't use a testr wrapper to report progress while tests are running * building the sdist can be slow (slow == noticeable) Monty's Rant: "So, I'm SUPER torn on this. On the one hand, we use tox in the gate, and run_tests.sh hides that from people and then they get confused about why some change they made to run_tests.sh doesn't get reflected in the reality of the project. On the other hand, what Victor says is true. We need to land a couple of changes to upstream tox that would allow it to not run develop each time - but we seem to have a hard time doing that. On the third hand, we'd doing an INSANE amount of work to make working with testing in a venv "easy" and what we've wound up with is a situation where we have multiple competing incompatible ways of doing things. Here's what I do: source .tox/py27/bin/activate testr run --parallel testr run --parallel testr run some-test testr run --failing deactivate Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip install -r requirements.txt -r test-requirements.txt I believe vish just installs the depends directly on his box and skips venvs altogether. It turns out that testr is a powerful tool with a nice UI. Instead of putting wrappers around it, perhaps we should add support for our output stream thing we like to it upstream. We should also add support to it for dropping into a debugger directly, so that the testtools issue goes away. Then we should land a patch to upstream tox to get it to do setup.py develop instead of sdist+ install. If we did that, we could have "run tox" be the simple answer for anyone who just wants to run whatever the gate runs, because that will always work. And if they want to do fancier things, they should learn about venvs and testr. However, if we ARE going to keep a run_tests.sh file in our trees, we should certainly have a single copy and have it behave consistently. Please note that some projects actually do just have run_tests.sh as a thin wrapper around tox." What do you think? Should we keep run_tests as is and port to oslo, or should we revisit the role of run_tests.sh? best, Joe ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On Mon, Jun 17, 2013 at 6:02 PM, Henry Gessau wrote: > On Mon, Jun 17, at 7:01 pm, Joe Gordon wrote: > > Hi All, > > > > A patch to move the run_tests.sh script into oslo-incubator > > (https://review.openstack.org/#/c/32736/), has brought up the bigger > > question of what is the future of './run_tests.sh.' > > > > This seems like a topic that directly affects all developers, so it > > sounded like it should be brought to the Mailing list. > > > > Reasons to keep run_tests.sh: > > > > * there is no possibility to run tests with testtools instead of testr. > > This feature allows us to use the debugger. > > * in some projects tox doesn't use a testr wrapper to report progress > > while tests are running > > * building the sdist can be slow (slow == noticeable) > > > > > > Monty's Rant: > > > > "So, I'm SUPER torn on this. > > > > On the one hand, we use tox in the gate, and run_tests.sh hides that from > > people and then they get confused about why some change they made to > > run_tests.sh doesn't get reflected in the reality of the project. > > > > On the other hand, what Victor says is true. We need to land a couple of > > changes to upstream tox that would allow it to not run develop each time > - > > but we seem to have a hard time doing that. > > > > On the third hand, we'd doing an INSANE amount of work to make working > > with testing in a venv "easy" and what we've wound up with is a situation > > where we have multiple competing incompatible ways of doing things. > > > > Here's what I do: source .tox/py27/bin/activate testr run --parallel > testr > > run --parallel testr run some-test testr run --failing deactivate > > > > Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip > install > > -r requirements.txt -r test-requirements.txt > > > > I believe vish just installs the depends directly on his box and skips > > venvs altogether. > > > > It turns out that testr is a powerful tool with a nice UI. Instead of > > putting wrappers around it, perhaps we should add support for our output > > stream thing we like to it upstream. We should also add support to it for > > dropping into a debugger directly, so that the testtools issue goes away. > > Then we should land a patch to upstream tox to get it to do setup.py > > develop instead of sdist+ install. > > > > If we did that, we could have "run tox" be the simple answer for anyone > > who just wants to run whatever the gate runs, because that will always > > work. And if they want to do fancier things, they should learn about > venvs > > and testr. > > > > However, if we ARE going to keep a run_tests.sh file in our trees, we > > should certainly have a single copy and have it behave consistently. > > > > Please note that some projects actually do just have run_tests.sh as a > > thin wrapper around tox." > > > > > > What do you think? Should we keep run_tests as is and port to oslo, or > > should we revisit the role of run_tests.sh? > > > > > My vote is for run_tests.sh to contain just one line, "cat TESTING", and > ensure the instructions are detailed and up to date. If course, it would be > nice if all the projects gravitated towards the same TESTING. > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > Post conversation with Lifeless and Clarkb on IRC, I'll change my answer slightly: As long as I can do the following: 1. Run individual tests easily 2. Drop into PDB by setting a break-point 3. Run WITHOUT virtual env 4. Not have to download a bunch of stuff or run some monster init every time I run tests on a fresh clone of a repo (see item 3) I don't care if that's done via a wrapper script that calls a framework that supports the option (well I do care but we're already there so I'd prefer not to take it away), better documentation of how to run tests and how the frameworks operate or whatever. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On Mon, Jun 17, at 7:01 pm, Joe Gordon wrote: > Hi All, > > A patch to move the run_tests.sh script into oslo-incubator > (https://review.openstack.org/#/c/32736/), has brought up the bigger > question of what is the future of './run_tests.sh.' > > This seems like a topic that directly affects all developers, so it > sounded like it should be brought to the Mailing list. > > Reasons to keep run_tests.sh: > > * there is no possibility to run tests with testtools instead of testr. > This feature allows us to use the debugger. > * in some projects tox doesn't use a testr wrapper to report progress > while tests are running > * building the sdist can be slow (slow == noticeable) > > > Monty's Rant: > > "So, I'm SUPER torn on this. > > On the one hand, we use tox in the gate, and run_tests.sh hides that from > people and then they get confused about why some change they made to > run_tests.sh doesn't get reflected in the reality of the project. > > On the other hand, what Victor says is true. We need to land a couple of > changes to upstream tox that would allow it to not run develop each time - > but we seem to have a hard time doing that. > > On the third hand, we'd doing an INSANE amount of work to make working > with testing in a venv "easy" and what we've wound up with is a situation > where we have multiple competing incompatible ways of doing things. > > Here's what I do: source .tox/py27/bin/activate testr run --parallel testr > run --parallel testr run some-test testr run --failing deactivate > > Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip install > -r requirements.txt -r test-requirements.txt > > I believe vish just installs the depends directly on his box and skips > venvs altogether. > > It turns out that testr is a powerful tool with a nice UI. Instead of > putting wrappers around it, perhaps we should add support for our output > stream thing we like to it upstream. We should also add support to it for > dropping into a debugger directly, so that the testtools issue goes away. > Then we should land a patch to upstream tox to get it to do setup.py > develop instead of sdist+ install. > > If we did that, we could have "run tox" be the simple answer for anyone > who just wants to run whatever the gate runs, because that will always > work. And if they want to do fancier things, they should learn about venvs > and testr. > > However, if we ARE going to keep a run_tests.sh file in our trees, we > should certainly have a single copy and have it behave consistently. > > Please note that some projects actually do just have run_tests.sh as a > thin wrapper around tox." > > > What do you think? Should we keep run_tests as is and port to oslo, or > should we revisit the role of run_tests.sh? > > My vote is for run_tests.sh to contain just one line, "cat TESTING", and ensure the instructions are detailed and up to date. If course, it would be nice if all the projects gravitated towards the same TESTING. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] The future of run_tests.sh
On Mon, Jun 17, 2013 at 5:01 PM, Joe Gordon wrote: > Hi All, > > A patch to move the run_tests.sh script into oslo-incubator ( > https://review.openstack.org/#/c/32736/), has brought up the bigger > question of what is the future of './run_tests.sh.' > > This seems like a topic that directly affects all developers, so it > sounded like it should be brought to the Mailing list. > > Reasons to keep run_tests.sh: > > * there is no possibility to run tests with testtools instead of testr. > This feature allows us to use the debugger. > * in some projects tox doesn't use a testr wrapper to report progress > while tests are running > * building the sdist can be slow (slow == noticeable) > > > Monty's Rant: > > "So, I'm SUPER torn on this. > > On the one hand, we use tox in the gate, and run_tests.sh hides that from > people and then they get confused about why some change they made to > run_tests.sh doesn't get reflected in the reality of the project. > > On the other hand, what Victor says is true. We need to land a couple of > changes to upstream tox that would allow it to not run develop each time - > but we seem to have a hard time doing that. > > On the third hand, we'd doing an INSANE amount of work to make working > with testing in a venv "easy" and what we've wound up with is a situation > where we have multiple competing incompatible ways of doing things. > > Here's what I do: source .tox/py27/bin/activate testr run --parallel testr > run --parallel testr run some-test testr run --failing deactivate > > Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip install > -r requirements.txt -r test-requirements.txt > > I believe vish just installs the depends directly on his box and skips > venvs altogether. > > It turns out that testr is a powerful tool with a nice UI. Instead of > putting wrappers around it, perhaps we should add support for our output > stream thing we like to it upstream. We should also add support to it for > dropping into a debugger directly, so that the testtools issue goes away. > Then we should land a patch to upstream tox to get it to do setup.py > develop instead of sdist+ install. > > If we did that, we could have "run tox" be the simple answer for anyone > who just wants to run whatever the gate runs, because that will always > work. And if they want to do fancier things, they should learn about venvs > and testr. > > However, if we ARE going to keep a run_tests.sh file in our trees, we > should certainly have a single copy and have it behave consistently. > > Please note that some projects actually do just have run_tests.sh as a > thin wrapper around tox." > > > What do you think? Should we keep run_tests as is and port to oslo, or > should we revisit the role of run_tests.sh? > > > best, > > Joe > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > As far as the question of whether run_tests.sh lives or dies, I have no intention of killing it (or letting it die in Cinder). With respect to porting it to OSLO, I'm torn a bit here. There are some things I like WRT the direction of testing in other projects, and some things not so much. The ability to have run_tests in the project allows me to tweak things and keep it usable (granted this capability probably wouldn't go away). I'm just a bit uneasy with all of the testing "enhancements and improvements". We've been trying to pull said changes in to Cinder but to be honest most of it from a dev perspective has been far more trouble than it's been worth. One change we loose ability to run individual tests, get it back... loose ability to run debugger... get it back... loose color output (I know horror or horrors), and so on. Understood if it was common or removed we'd all be in the same boat, but I'm not sure other care about some things like no virtual env and debugging tests as much as I do. Anyway, run_tests is extremely valuable to me and I believe others as is the ability to run outside of venv etc. Granted there is a ton of overlap and OSLO does have meritt, I just don't want unwanted and unnecessary change as we've finally got things stabilized again. Thanks, John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] The future of run_tests.sh
Hi All, A patch to move the run_tests.sh script into oslo-incubator ( https://review.openstack.org/#/c/32736/), has brought up the bigger question of what is the future of './run_tests.sh.' This seems like a topic that directly affects all developers, so it sounded like it should be brought to the Mailing list. Reasons to keep run_tests.sh: * there is no possibility to run tests with testtools instead of testr. This feature allows us to use the debugger. * in some projects tox doesn't use a testr wrapper to report progress while tests are running * building the sdist can be slow (slow == noticeable) Monty's Rant: "So, I'm SUPER torn on this. On the one hand, we use tox in the gate, and run_tests.sh hides that from people and then they get confused about why some change they made to run_tests.sh doesn't get reflected in the reality of the project. On the other hand, what Victor says is true. We need to land a couple of changes to upstream tox that would allow it to not run develop each time - but we seem to have a hard time doing that. On the third hand, we'd doing an INSANE amount of work to make working with testing in a venv "easy" and what we've wound up with is a situation where we have multiple competing incompatible ways of doing things. Here's what I do: source .tox/py27/bin/activate testr run --parallel testr run --parallel testr run some-test testr run --failing deactivate Sometimes, if I'm being fancy, I'll do: virtualenv foo foo/bin/pip install -r requirements.txt -r test-requirements.txt I believe vish just installs the depends directly on his box and skips venvs altogether. It turns out that testr is a powerful tool with a nice UI. Instead of putting wrappers around it, perhaps we should add support for our output stream thing we like to it upstream. We should also add support to it for dropping into a debugger directly, so that the testtools issue goes away. Then we should land a patch to upstream tox to get it to do setup.py develop instead of sdist+ install. If we did that, we could have "run tox" be the simple answer for anyone who just wants to run whatever the gate runs, because that will always work. And if they want to do fancier things, they should learn about venvs and testr. However, if we ARE going to keep a run_tests.sh file in our trees, we should certainly have a single copy and have it behave consistently. Please note that some projects actually do just have run_tests.sh as a thin wrapper around tox." What do you think? Should we keep run_tests as is and port to oslo, or should we revisit the role of run_tests.sh? best, Joe ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
