[issue7897] Support parametrized tests in unittest

Change by Phil Connell : -- nosy: +pconnell ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue7897] Support parametrized tests in unittest

Andre Herbst added the comment: +1 for the feature Subtests make the test results of all asserts visible at test execution time but decrease the readability of a test: @parameterized([2,4,6]) def test_method_whenCalled_returnsNone(self, a): # 1) arrange something = Something()

[issue7897] Support parametrized tests in unittest

Change by Mark Lawrence : -- nosy: -BreamoreBoy ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue7897] Support parametrized tests in unittest

Changes by Alexander Belopolsky : -- versions: +Python 3.6 -Python 3.5 ___ Python tracker ___

[issue7897] Support parametrized tests in unittest

Changes by Martin Panter vadmium...@gmail.com: -- nosy: +vadmium ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list

[issue7897] Support parametrized tests in unittest

Nick Coghlan added the comment: I don't believe the how to request specific parameters aspect has been clearly addressed so far, and the subTest() API in Python 3.4 already allows subtests that aren't individually addressable. Any new API (if any) should likely also be based on the subtest

[issue7897] Support parametrized tests in unittest

Michael Foord added the comment: I agree with Nick. There is a potential use case for parameterized tests as well as sub tests, but it's not something we're going to rush into. -- ___ Python tracker rep...@bugs.python.org

[issue7897] Support parametrized tests in unittest

Mark Lawrence added the comment: Is there any possibility of getting this into 3.5? If it helps I've always got time on my hands so if nothing else I could do testing on Windows 8.1. -- nosy: +BreamoreBoy, zach.ware versions: +Python 3.5 -Python 3.4

[issue7897] Support parametrized tests in unittest

Changes by Brian Curtin br...@python.org: -- nosy: -brian.curtin ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list

[issue7897] Support parametrized tests in unittest

Ezio Melotti added the comment: Since we now got subtests, can this be closed? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___

[issue7897] Support parametrized tests in unittest

R. David Murray added the comment: subtests don't satisfy my use cases. You can't run an individual subtest by name, and I find that to be a very important thing to be able to do during development and debugging. At the moment at least I'm fine with just having my parameterize decorator in

[issue7897] Support parametrized tests in unittest

Nick Coghlan added the comment: Right, subtests are about improving reporting without adding selectivity. Explicitly parameterized tests require more structural changes to tests, but give the selectivity that subtests don't. -- ___ Python tracker

[issue7897] Support parametrized tests in unittest

Changes by Terry J. Reedy tjre...@udel.edu: -- nosy: +terry.reedy ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list

[issue7897] Support parametrized tests in unittest

Michael Foord added the comment: Looks like we're going to get subtests ( issue #16997 ) instead of parameterized tests. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___

[issue7897] Support parametrized tests in unittest

Miklós Fazekas added the comment: http://gist.github.com/mfazekas/1710455 I have a parametric delclarator which works is similiar to sweepargs in concept. It can be either applied at class or method level. And it mutates testnames so failure should be nice, and filters can be applied too.

[issue7897] Support parametrized tests in unittest

Changes by Santoso Wijaya santoso.wij...@gmail.com: -- nosy: +santa4nt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list

[issue7897] Support parametrized tests in unittest

Changes by Chris Jerdonek chris.jerdo...@gmail.com: -- nosy: +chris.jerdonek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___

[issue7897] Support parametrized tests in unittest

Changes by Bradley Froehle brad.froe...@gmail.com: -- nosy: +bfroehle ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list

[issue7897] Support parametrized tests in unittest

Changes by florian-rathgeber florian.rathge...@gmail.com: -- nosy: +florian-rathgeber ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___

[issue7897] Support parametrized tests in unittest

Nick Coghlan ncogh...@gmail.com added the comment: As another in-the-standard-library uses case: my additions to the ipaddress test suite are really crying out for parameterised test support. My current solution is adequate for coverage and debugging purposes (a custom assert applied to

[issue7897] Support parametrized tests in unittest

Borja Ruiz Castro br...@alienvault.com added the comment: Hi Murray! I use a lot od parametrized tests. I usually use the ENV to pass these parameters and/or a custon configuration file. What is your approach to parametrize all the test stuff? Regards, Borja. On 31 May 2012 03:57, R. David

[issue7897] Support parametrized tests in unittest

Borja Ruiz Castro br...@alienvault.com added the comment: Sorry, I failed to mention that I use Testify to launch all my tests! On 2 July 2012 13:23, Borja Ruiz Castro rep...@bugs.python.org wrote: Borja Ruiz Castro br...@alienvault.com added the comment: Hi Murray! I use a lot od

[issue7897] Support parametrized tests in unittest

Éric Araujo mer...@netwok.org added the comment: I like test_email’s decorator. It looks like https://github.com/wolever/nose-parameterized which I’m using. (The implementation and generation of test method names may be less nice than what was discussed here, but the API (decorator + list

[issue7897] Support parametrized tests in unittest

R. David Murray rdmur...@bitdance.com added the comment: People interested in this issue might be interested in changeset e6a33938b03f. I use parameterized unit tests in email a lot, and was annoyed by the fact that I couldn't run the tests individually using the unittest CLI. The fix for

[issue7897] Support parametrized tests in unittest

Nick Coghlan ncogh...@gmail.com added the comment: I just remembered that many of the urllib.urlparse tests are guilty of only reporting the first case that fails, instead of testing everything and reporting all failures: http://hg.python.org/cpython/file/default/Lib/test/test_urlparse.py

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: FWIW I think nose2 is going to have test load time parameterized tests rather than run time parameterized tests, which is what I think we should do for unit test. The API should be as simple as possible for basic cases, but suitable

[issue7897] Support parametrized tests in unittest

Changes by holger krekel holger.kre...@gmail.com: -- nosy: +hpk ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list

[issue7897] Support parametrized tests in unittest

Changes by Julian Berman julian+python@grayvines.com: -- nosy: +Julian ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___

[issue7897] Support parametrized tests in unittest

Éric Araujo mer...@netwok.org added the comment: Another nice API: http://feldboris.alwaysdata.net/blog/unittest-template.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: Well, pyflakes will tell you about name clashes within a TestCase (unless you're shadowing a test on a base class which I guess is rarely the case)... When we generate the tests we could add the parameter reprs to the docstring. A

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: Note that name clashes *would* result in non-unique testcase ids, so we need to prevent that. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897

[issue7897] Support parametrized tests in unittest

R. David Murray rdmur...@bitdance.com added the comment: Please implement name+argtuple first and build auto-naming on top of that. Nick's approach would not allow me to specify a custom (hand coded) name for each set of arguments, which is my normal use case. I also would not like the

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: David, I don't understand - it looks like Nick's suggestion would allow you to create a name per case, that's the point of it! You could even do this: def _name_from_case(name, cases): for idx, case in enumerate(cases, start=1):

[issue7897] Support parametrized tests in unittest

R. David Murray rdmur...@bitdance.com added the comment: Oh, I see. Make the name the first element of the argtuple and then strip it off. Well, that will work, it just seems bass-ackwards to me :) And why is it 'case'? I thought we were talking about tests. --

[issue7897] Support parametrized tests in unittest

Changes by Nick Coghlan ncogh...@gmail.com: -- Removed message: http://bugs.python.org/msg140819 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___

[issue7897] Support parametrized tests in unittest

Nick Coghlan ncogh...@gmail.com added the comment: Sorry, misclicked and removed this comment from David: Oh, I see. Make the name the first element of the argtuple and then strip it off. Well, that will work, it just seems bass-ackwards to me :) And why is it 'case'? I thought we were

[issue7897] Support parametrized tests in unittest

Nick Coghlan ncogh...@gmail.com added the comment: In my example, I needed a word to cover each entry in the collection of parameter tuples. 'case' fit the bill. The reason I like the builder approach is that it means the simplest usage is to just create a list (or other iterable) of

[issue7897] Support parametrized tests in unittest

Éric Araujo mer...@netwok.org added the comment: Well, pyflakes will tell you about name clashes within a TestCase Seen that :) (unless you're shadowing a test on a base class which I guess is rarely the case)... That too, in our own test suite (test_list or test_tuple, I have that change in

[issue7897] Support parametrized tests in unittest

Andrew Bennetts s...@users.sourceforge.net added the comment: You may be interested an existing, unittest-compatible library that provides this: http://pypi.python.org/pypi/testscenarios -- nosy: +spiv ___ Python tracker rep...@bugs.python.org

[issue7897] Support parametrized tests in unittest

Austin Bingham austin.bing...@gmail.com added the comment: Has a decision been made to implement some form of parametric tests? Is working being done? Along with parameterizing individual test methods, I'd also like to throw out a request for parametric TestCases. In some cases I find that I

[issue7897] Support parametrized tests in unittest

Brian Curtin br...@python.org added the comment: By this issue existing, that's the decision that we should probably do this, and I think the discussion shows we agree it should happen. How it's done is another way, and we have roughly a year to get it figured out before 3.3 gets closer. I

[issue7897] Support parametrized tests in unittest

Changes by R. David Murray rdmur...@bitdance.com: -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___

[issue7897] Support parametrized tests in unittest

R. David Murray rdmur...@bitdance.com added the comment: Brian, if you don't have time to work on it in the next little while, maybe you could post your partial patch in case someone else wants to work on it? Might be a good project for someone on the mentoring list. Unless someone sees a

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: *If* we add this to unittest then we need to decide between test load time parameterised tests and test run time parameterisation. Load time is more backwards compatible / easier (all tests can be generated at load time and the number

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: And yes, parameterising test cases is a different issue. bzr does this IIRC. This is easier in some ways, and can be done through load_tests, or any other test load time mechanism. -- ___

[issue7897] Support parametrized tests in unittest

R. David Murray rdmur...@bitdance.com added the comment: Michael, would your single test clearly indicate all the individual failures by name? If not, then I would not find it useful. I can already easily parameterize inside a single test using a loop, it's the detailed reporting piece that

[issue7897] Support parametrized tests in unittest

R. David Murray rdmur...@bitdance.com added the comment: The reporting piece, and ideally being able to use the arguments to unittest to run a single one of the parameterized tests. (I can get the reporting piece now using the locals() hack, but that doesn't support test selection). Does

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: Test selection would require load time parameterisation - although the current test selection mechanism is through importing which would probably *not* work without a specific fix. Same for run time parameterisation. Well how *exactly*

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: Dammit, I've reversed my thinking in some of those messages. Load time parameterisation *does* give you separate test reporting. It is run time parameterisation that doesn't. Depending on how you do it (i.e. if the decorator generates

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: So if we do import time *or* test load time parameterisation then we can do separate failure reporting. We may still want to improve test selection for parameterised tests. There are use cases for run time parameterisation (for example

[issue7897] Support parametrized tests in unittest

Changes by Éric Araujo mer...@netwok.org: -- nosy: +eric.araujo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list

[issue7897] Support parametrized tests in unittest

Nick Coghlan ncogh...@gmail.com added the comment: Load time parameterisation seems more of a worthwhile addition to me, too. As David noted, runtime parameterisation is pretty easy to do by looping and consolidating failures into the one error message via self.fail(). For test naming, trying

[issue7897] Support parametrized tests in unittest

Changes by Eric Snow ericsnowcurren...@gmail.com: -- nosy: +ericsnow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: That all sounds good to me Nick. Some notes / questions. How should parameterised tests be marked? I'm happy with a unittest.parameterized decorator (it would do no work other than mark the test method, with the parameterisation being

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: Oh, and if we're not going to get clever with naming, how is the TestResult going to include the parameter repr in the failure report? That information will have to be stored on the TestCase. I would prefer this feature not to touch

[issue7897] Support parametrized tests in unittest

Nick Coghlan ncogh...@gmail.com added the comment: Here's a sketch for a possible decorator factory: def parameters(params, *, builder=_default_param_builder): def make_parameterized_test(f): return ParameterizedTest(f, params, builder) return make_parameterized_test (default

[issue7897] Support parametrized tests in unittest

R. David Murray rdmur...@bitdance.com added the comment: Personally I would be happy if I could pass in a dictionary that maps names to argument tuples, or an iterator yielding (name, (argtuple)) pairs, and just have the failure report the name. That is, make me responsible for generating

[issue7897] Support parametrized tests in unittest

Changes by Ezio Melotti ezio.melo...@gmail.com: -- nosy: +ezio.melotti ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___ ___ Python-bugs-list

[issue7897] Support parametrized tests in unittest

Austin Bingham austin.bing...@gmail.com added the comment: OK, I created issue 12600 for dealing with parameterized TestCases as opposed to individual tests. On Wed, Jul 20, 2011 at 4:03 PM, R. David Murray rep...@bugs.python.org wrote: Unless someone sees a clever way to implement both with

[issue7897] Support parametrized tests in unittest

nchauvat nicolas.chau...@logilab.fr added the comment: In case it could be useful, here is how generative/parametrized tests are handled in logilab.common.testlib http://hg.logilab.org/logilab/common/file/a6b5fe18df99/testlib.py#l1137 -- nosy: +nchauvat

[issue7897] Support parametrized tests in unittest

Nick Coghlan ncogh...@gmail.com added the comment: No, I'm saying I don't see summarising the parameterised tests separately from the overall test run as a particularly important feature, since you can test multiple parameters in a single test manually now. The important part is for the

[issue7897] Support parametrized tests in unittest

Nick Coghlan ncogh...@gmail.com added the comment: I agree with Michael - one test that covers multiple settings can easily be done by collecting results within the test itself and then checking at the end that no failures were detected (e.g. I've done this myself with a test that needed to

[issue7897] Support parametrized tests in unittest

Yaroslav Halchenko yarikop...@gmail.com added the comment: Hi Nick, Am I reading your right, Are you suggesting to implement this manual looping/collecting/reporting separately in every unittest which needs that? On Tue, 11 May 2010, Nick Coghlan wrote: Nick Coghlan ncogh...@gmail.com added

[issue7897] Support parametrized tests in unittest

Fernando Perez fdo.pe...@gmail.com added the comment: Hey Yarick, On Thu, Apr 8, 2010 at 18:53, Yaroslav Halchenko rep...@bugs.python.org w= rote: In PyMVPA we have our little decorator as an alternative to Fernando's ge= nerators, =A0and which is closer, I think, to what Michael was wishing

[issue7897] Support parametrized tests in unittest

Yaroslav Halchenko yarikop...@gmail.com added the comment: Fernando, I agree... somewhat ;-) At some point (whenever everything works fine and no unittests fail) I wanted to merry sweepargs to nose and make it spit out a dot (or animate a spinning wheel ;)) for every passed unittest, so

[issue7897] Support parametrized tests in unittest

Fernando Perez fdo.pe...@gmail.com added the comment: Yarick: Yes, I do actually see the value of the summary view. When I have a parametric test that fails, I tend to just run nose with -x so it stops at the first error and with the --pdb options to study it, so I simply ignore all the

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: If we provide builtin support for parameterized tests it will have to report each test separately otherwise there is no point. You can already add support for running tests with multiple parameters yourself - the *only* advantage of

[issue7897] Support parametrized tests in unittest

Yaroslav Halchenko yarikop...@gmail.com added the comment: In PyMVPA we have our little decorator as an alternative to Fernando's generators, and which is closer, I think, to what Michael was wishing for: @sweepargs http://github.com/yarikoptic/PyMVPA/blob/master/mvpa/testing/sweepargs.py NB

[issue7897] Support parametrized tests in unittest

Antoine Pitrou pit...@free.fr added the comment: I'm not sure what this brings. It is easy to write a loop iterating over test data. What parametric testing could bring is precise progress and error display (displaying the parameters for each run), but this doesn't seem to be the case here,

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: By the way - I have no opinion on whether or not using yield is the right way to support parameterized tests. It may be better for the test method to take arguments, and be decorated as a parameterized test, with the decorator

[issue7897] Support parametrized tests in unittest

Jean-Paul Calderone exar...@divmod.com added the comment: With paramterized tests *all* the tests are run and *all* failures reported. With testing in a loop the tests stop at the first failure. +1 to this justification. Parameterized tests are a big win over a simple for loop in a test.

[issue7897] Support parametrized tests in unittest

Brian Curtin cur...@acm.org added the comment: It may be better for the test method to take arguments, and be decorated as a parameterized test, with the decorator providing the parameters. +1 on something like this. That's also how NUnit supports parameterized tests. -- nosy:

[issue7897] Support parametrized tests in unittest

Antoine Pitrou pit...@free.fr added the comment: With paramterized tests *all* the tests are run and *all* failures reported. With testing in a loop the tests stop at the first failure. +1 to this justification. Parameterized tests are a big win over a simple for loop in a test. Ah,

[issue7897] Support parametrized tests in unittest

Michael Foord mich...@voidspace.org.uk added the comment: Antoine: the failure message would include a repr of the parameters used in the particular test that failed. So you can tell which test failed and with what parameters. -- ___ Python tracker

[issue7897] Support parametrized tests in unittest

Jean-Paul Calderone exar...@divmod.com added the comment: Something else I think it would be nice to consider is what the id() (and shortDescription(), heh) of the resulting tests will be. It would be great if the id were sufficient to identify a particular test *and* data combination. In

[issue7897] Support parametrized tests in unittest

Fernando Perez fdo.pe...@gmail.com added the comment: I should probably have clarified better our reasons for using this type of code. The first is the one Michael pointed out, where such parametric tests all execute; it's very common in scientific computing to have algorithms that only fail

[issue7897] Support parametrized tests in unittest

Changes by Michael Foord mich...@voidspace.org.uk: -- assignee: - michael.foord nosy: +michael.foord ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7897 ___

[issue7897] Support parametrized tests in unittest

New submission from Fernando Perez fdo.pe...@gmail.com: IPython has unittest-based parametric testing (something nose has but which produces effectively undebuggable tests, while this approach gives perfectly debuggable ones). The code lives here for 2.x and 3.x: