Re: [openstack-dev] [nova] fastest way to run individual tests ?
On 07/09/2014 10:51 PM, Matt Riedemann wrote: > > > On 6/12/2014 6:17 AM, Daniel P. Berrange wrote: >> On Thu, Jun 12, 2014 at 07:07:37AM -0400, Sean Dague wrote: >>> On 06/12/2014 06:59 AM, Daniel P. Berrange wrote: Does anyone have any tip on how to actually run individual tests in an efficient manner. ie something that adds no more than 1 second penalty over & above the time to run the test itself. NB, assume that i've primed the virtual env with all prerequisite deps already. >>> >>> The overhead is in the fact that we have to discover the world, then >>> throw out the world. >>> >>> You can actually run an individual test via invoking the testtools.run >>> directly: >>> python -m testtools.run nova.tests.test_versions >>> >>> (Also, when testr explodes because of an import error this is about the >>> only way to debug what's going on). >> >> Most excellent, thankyou. I knew someone must know a way to do it :-) >> >> Regards, >> Daniel >> > > I've been beating my head against the wall a bit on unit tests too this > week, and here is another tip that just uncovered something for me when > python -m testtools.run and nosetests didn't help. > > I sourced the tox virtualenv and then ran the test from there, which > gave me the actual error, so something like this: > > source .tox/py27/bin/activate > python -m testtools.run > > Props to Matt Odden for helping me with the source of the venv tip. > FWIW - this is what ./run_tests.sh -d does but also prepends the lockutils invocation Vish mentioned that is needed for some tests to run. I've also noticed several bugs when running functional tests this way, especially those that start services and parse config options when we don't load the whole world, which no one seems to be fixing, so I assumed that usage of this was not very widespread (I assume bcz ppl used tox). N. PS. Now that I think about it - I've also not submitted fixes for them... :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fastest way to run individual tests ?
On Jul 9, 2014, at 13:51, Matt Riedemann wrote: > I've been beating my head against the wall a bit on unit tests too this week, > and here is another tip that just uncovered something for me when python -m > testtools.run and nosetests didn't help. > > I sourced the tox virtualenv and then ran the test from there, which gave me > the actual error, so something like this: > > source .tox/py27/bin/activate > python -m testtools.run > > Props to Matt Odden for helping me with the source of the venv tip. This is awesome, thank you for sharing. signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fastest way to run individual tests ?
On 7/10/14, 12:08 PM, Chris Dent wrote: > On Thu, 10 Jul 2014, Mike Bayer wrote: > >> I typically never use tox or testtools at the commandline until I'm >> ready to commit and want to see what the jenkins builds will see. I >> start up the whole thing and then it's time to take a break while it >> reinvents the whole world. > > Me tool I've been squeezing py.test into my testing as well. It > allows me to do TDD off the test file of the thing that I'm creating > or changing and focus on just that without the incredibly long round > trip time for tox and friends. > > I do some variation on: > >py.test -svx path/to/test/file.py > > with a pre-warmed virtualenv. > > My next hope is to get rid of unittest and just do the plain asserts > that py.test makes so nice and lovely. i think this is an important issue. When I first got on the IRC channel and was asking around about some testing issues, everyone was like, "just run tox. why aren't you running tox? Run tox, it just works!". well of course it works, but it takes five minutes. This is a serious productivity drain and also having to wait five minutes just to test some small thing means you lose your focus and concentration, leads to more mistakes and less insight on the best way to do something. I've been working with openstack for six weeks now and I am extremely impressed by the infrastructure of stackforge, gerritt, all of that. But at the same time there seems to be a bit of "default to heavyweight" in all cases; when I ask around about allowing things to work in more ways than just "how the jenkins builds do it", while it's not always the case, people have sometimes reacted with puzzlement. I think when changes are proposed that allow more flexibility in how tests are run, databases are built up, and environments are configured, they should be welcomed. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fastest way to run individual tests ?
On Thu, 10 Jul 2014, Mike Bayer wrote: I typically never use tox or testtools at the commandline until I'm ready to commit and want to see what the jenkins builds will see. I start up the whole thing and then it's time to take a break while it reinvents the whole world. Me tool I've been squeezing py.test into my testing as well. It allows me to do TDD off the test file of the thing that I'm creating or changing and focus on just that without the incredibly long round trip time for tox and friends. I do some variation on: py.test -svx path/to/test/file.py with a pre-warmed virtualenv. My next hope is to get rid of unittest and just do the plain asserts that py.test makes so nice and lovely. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fastest way to run individual tests ?
On 7/9/14, 4:51 PM, Matt Riedemann wrote: > > > On 6/12/2014 6:17 AM, Daniel P. Berrange wrote: >> On Thu, Jun 12, 2014 at 07:07:37AM -0400, Sean Dague wrote: >>> On 06/12/2014 06:59 AM, Daniel P. Berrange wrote: Does anyone have any tip on how to actually run individual tests in an efficient manner. ie something that adds no more than 1 second penalty over & above the time to run the test itself. NB, assume that i've primed the virtual env with all prerequisite deps already. >>> >>> The overhead is in the fact that we have to discover the world, then >>> throw out the world. >>> >>> You can actually run an individual test via invoking the testtools.run >>> directly: >>> python -m testtools.run nova.tests.test_versions >>> >>> (Also, when testr explodes because of an import error this is about the >>> only way to debug what's going on). >> >> Most excellent, thankyou. I knew someone must know a way to do it :-) >> >> Regards, >> Daniel >> > > I've been beating my head against the wall a bit on unit tests too > this week, and here is another tip that just uncovered something for > me when python -m testtools.run and nosetests didn't help. > > I sourced the tox virtualenv and then ran the test from there, which > gave me the actual error, so something like this: > > source .tox/py27/bin/activate > python -m testtools.run > > Props to Matt Odden for helping me with the source of the venv tip. Being kind of an outlaw in the testing world, when hitting individual tests I typically am using py.test. It has a way way nicer interface than the testtools stuff, when things don't import correctly you get an actual informative error message rather than ten screenfuls of garbage, stuff like that.You can point it at the .py file of the test, instead of the module path, so that shell completion can be used to find what you're looking for, e.g. py.test tests/ --> tests/sqlalchemy/ --> tests/sqlalchemy/test_migrations.py. I typically never use tox or testtools at the commandline until I'm ready to commit and want to see what the jenkins builds will see. I start up the whole thing and then it's time to take a break while it reinvents the whole world. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fastest way to run individual tests ?
On 6/12/2014 6:17 AM, Daniel P. Berrange wrote: On Thu, Jun 12, 2014 at 07:07:37AM -0400, Sean Dague wrote: On 06/12/2014 06:59 AM, Daniel P. Berrange wrote: Does anyone have any tip on how to actually run individual tests in an efficient manner. ie something that adds no more than 1 second penalty over & above the time to run the test itself. NB, assume that i've primed the virtual env with all prerequisite deps already. The overhead is in the fact that we have to discover the world, then throw out the world. You can actually run an individual test via invoking the testtools.run directly: python -m testtools.run nova.tests.test_versions (Also, when testr explodes because of an import error this is about the only way to debug what's going on). Most excellent, thankyou. I knew someone must know a way to do it :-) Regards, Daniel I've been beating my head against the wall a bit on unit tests too this week, and here is another tip that just uncovered something for me when python -m testtools.run and nosetests didn't help. I sourced the tox virtualenv and then ran the test from there, which gave me the actual error, so something like this: source .tox/py27/bin/activate python -m testtools.run Props to Matt Odden for helping me with the source of the venv tip. -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fastest way to run individual tests ?
This isn’t an officially supported method, but i tend to use: python -m nova.openstack.common.lockutils nosetests for example: python -m nova.openstack.common.lockutils nosetests nova.tests.integrated.test_api_samples:CloudPipeSampleJsonTest.test_cloud_pipe_create I think this is a little bit less flexible than testtools run but old habits. Vish On Jun 12, 2014, at 3:59 AM, Daniel P. Berrange wrote: > When in the middle of developing code for nova I'll typically not wish to > the run the entire Nova test suite every time I have a bit of code to > verify. I'll just want to run the single test case that deals with the > code I'm hacking on. > > I'm currently writing a 'test_hardware.py' test case for the NUMA work > I'm dealing with. I can run that using 'run_tests.sh' or 'tox' by just > passing the name of the test case. The test case in question takes a tiny > fraction of a second to run, but the tox command above wastes 32 seconds > faffing about before it runs the test itself, while run_tests.sh is not > much better wasting 22 seconds. > > # tox -e py27 tests.virt.test_hardware > ...snip... > real0m32.923s > user0m22.083s > sys 0m4.377s > > > # time ./run_tests.sh tests.virt.test_hardware > ...snip... > real0m22.075s > user0m14.282s > sys 0m1.407s > > > This is a really severe time penalty to incurr each time I want to run > this tiny test (which is very frequently during dev). > > Does anyone have any tip on how to actually run individual tests in an > efficient manner. ie something that adds no more than 1 second penalty > over & above the time to run the test itself. NB, assume that i've primed > the virtual env with all prerequisite deps already. > > Regards, > Daniel > -- > |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| > |: http://libvirt.org -o- http://virt-manager.org :| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fastest way to run individual tests ?
On Thu, Jun 12, 2014 at 07:07:37AM -0400, Sean Dague wrote: > On 06/12/2014 06:59 AM, Daniel P. Berrange wrote: > > Does anyone have any tip on how to actually run individual tests in an > > efficient manner. ie something that adds no more than 1 second penalty > > over & above the time to run the test itself. NB, assume that i've primed > > the virtual env with all prerequisite deps already. > > > > The overhead is in the fact that we have to discover the world, then > throw out the world. > > You can actually run an individual test via invoking the testtools.run > directly: > > > python -m testtools.run nova.tests.test_versions > > (Also, when testr explodes because of an import error this is about the > only way to debug what's going on). Most excellent, thankyou. I knew someone must know a way to do it :-) Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] fastest way to run individual tests ?
On 06/12/2014 06:59 AM, Daniel P. Berrange wrote: > When in the middle of developing code for nova I'll typically not wish to > the run the entire Nova test suite every time I have a bit of code to > verify. I'll just want to run the single test case that deals with the > code I'm hacking on. > > I'm currently writing a 'test_hardware.py' test case for the NUMA work > I'm dealing with. I can run that using 'run_tests.sh' or 'tox' by just > passing the name of the test case. The test case in question takes a tiny > fraction of a second to run, but the tox command above wastes 32 seconds > faffing about before it runs the test itself, while run_tests.sh is not > much better wasting 22 seconds. > ># tox -e py27 tests.virt.test_hardware >...snip... >real 0m32.923s >user 0m22.083s >sys0m4.377s > > ># time ./run_tests.sh tests.virt.test_hardware >...snip... >real 0m22.075s >user 0m14.282s >sys0m1.407s > > > This is a really severe time penalty to incurr each time I want to run > this tiny test (which is very frequently during dev). > > Does anyone have any tip on how to actually run individual tests in an > efficient manner. ie something that adds no more than 1 second penalty > over & above the time to run the test itself. NB, assume that i've primed > the virtual env with all prerequisite deps already. > > Regards, > Daniel The overhead is in the fact that we have to discover the world, then throw out the world. You can actually run an individual test via invoking the testtools.run directly: > python -m testtools.run nova.tests.test_versions (Also, when testr explodes because of an import error this is about the only way to debug what's going on). -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [nova] fastest way to run individual tests ?
When in the middle of developing code for nova I'll typically not wish to the run the entire Nova test suite every time I have a bit of code to verify. I'll just want to run the single test case that deals with the code I'm hacking on. I'm currently writing a 'test_hardware.py' test case for the NUMA work I'm dealing with. I can run that using 'run_tests.sh' or 'tox' by just passing the name of the test case. The test case in question takes a tiny fraction of a second to run, but the tox command above wastes 32 seconds faffing about before it runs the test itself, while run_tests.sh is not much better wasting 22 seconds. # tox -e py27 tests.virt.test_hardware ...snip... real 0m32.923s user 0m22.083s sys 0m4.377s # time ./run_tests.sh tests.virt.test_hardware ...snip... real 0m22.075s user 0m14.282s sys 0m1.407s This is a really severe time penalty to incurr each time I want to run this tiny test (which is very frequently during dev). Does anyone have any tip on how to actually run individual tests in an efficient manner. ie something that adds no more than 1 second penalty over & above the time to run the test itself. NB, assume that i've primed the virtual env with all prerequisite deps already. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
