[issue7897] Support parametrized tests in unittest

[Phil Connell]

Change by Phil Connell :


--
nosy: +pconnell

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Andre Herbst]

Andre Herbst  added the comment:

+1 for the feature

Subtests make the test results of all asserts visible at test execution time 
but decrease the readability of a test:

@parameterized([2,4,6])
def test_method_whenCalled_returnsNone(self, a):
# 1) arrange
something = Something()

# 2) act
result = something.method(a)

# 3) assert
self.assertIsNone(result)

When using subtests the phases of 1) arrange, 2) act, 3) assert are not clearly 
separated, the unit test contains logic and two additional indentation levels 
that could be avoided with parameterized tests:

def test_method_whenCalled_returnsNone(self, a):
# 1) arrange
something = Something()

for a in [2,4,6]:
with self.subTest(a=a):
# 2) act
result = something.method(a)

# 3) assert
self.assertIsNone(result)

--
nosy: +moormaster

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Mark Lawrence]

Change by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Alexander Belopolsky]

Changes by Alexander Belopolsky :


--
versions: +Python 3.6 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Martin Panter]

Changes by Martin Panter vadmium...@gmail.com:


--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan added the comment:

I don't believe the how to request specific parameters aspect has been
clearly addressed so far, and the subTest() API in Python 3.4 already
allows subtests that aren't individually addressable.

Any new API (if any) should likely also be based on the subtest feature,
and I don't think the community has enough experience with that yet to be
confident in how best to integrate it with parameterised testing.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord added the comment:

I agree with Nick. There is a potential use case for parameterized tests as 
well as sub tests, but it's not something we're going to rush into.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Mark Lawrence]

Mark Lawrence added the comment:

Is there any possibility of getting this into 3.5?  If it helps I've always got 
time on my hands so if nothing else I could do testing on Windows 8.1.

--
nosy: +BreamoreBoy, zach.ware
versions: +Python 3.5 -Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Brian Curtin]

Changes by Brian Curtin br...@python.org:


--
nosy:  -brian.curtin

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Ezio Melotti]

Ezio Melotti added the comment:

Since we now got subtests, can this be closed?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[R. David Murray]

R. David Murray added the comment:

subtests don't satisfy my use cases.  You can't run an individual subtest by 
name, and I find that to be a very important thing to be able to do during 
development and debugging.  At the moment at least I'm fine with just having my 
parameterize decorator in the email module, so I'm not motivated to move this 
forward right now.  I would like to come back to it eventually, though.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan added the comment:

Right, subtests are about improving reporting without adding selectivity.
Explicitly parameterized tests require more structural changes to tests,
but give the selectivity that subtests don't.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Terry J. Reedy]

Changes by Terry J. Reedy tjre...@udel.edu:


--
nosy: +terry.reedy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord added the comment:

Looks like we're going to get subtests ( issue #16997 ) instead of 
parameterized tests.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Miklós Fazekas]

Miklós Fazekas added the comment:

http://gist.github.com/mfazekas/1710455

I have a parametric delclarator which works is similiar to sweepargs in 
concept. It can be either applied at class or method level. And it mutates 
testnames so failure should be nice, and filters can be applied too.

@parametric
class MyTest(unittest.TestCase):
@parametric(foo=[1,2],bar=[3,4])
def testWithParams(self,foo,bar):
self.assertLess(foo,bar)
def testNormal(self):
self.assertEqual('foo','foo')

@parametric(foo=[1,2],bar=[3,4])
class MyTest(unittest.TestCase):
 def testPositive(self,foo,bar):
 self.assertLess(foo,bar)
 def testNegative(self,foo,bar):
 self.assertLess(-foo,-bar)

Sample failures:
==
FAIL: testNegative_bar_3_foo_1 (__main__.MyTest)
--
Traceback (most recent call last):
  File /Work/temp/parametric.py, line 63, in f
self.fun(*args,**v)
  File /Work/temp/parametric.py, line 158, in testNegative
self.assertLess(-foo,-bar)
AssertionError: -1 not less than -3

==
FAIL: testNegative_bar_3_foo_2 (__main__.MyTest)
--
Traceback (most recent call last):
  File /Work/temp/parametric.py, line 63, in f
self.fun(*args,**v)
  File /Work/temp/parametric.py, line 158, in testNegative
self.assertLess(-foo,-bar)
AssertionError: -2 not less than -3

--
nosy: +Miklós.Fazekas

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Santoso Wijaya]

Changes by Santoso Wijaya santoso.wij...@gmail.com:


--
nosy: +santa4nt

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Chris Jerdonek]

Changes by Chris Jerdonek chris.jerdo...@gmail.com:


--
nosy: +chris.jerdonek

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Bradley Froehle]

Changes by Bradley Froehle brad.froe...@gmail.com:


--
nosy: +bfroehle

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[florian-rathgeber]

Changes by florian-rathgeber florian.rathge...@gmail.com:


--
nosy: +florian-rathgeber

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

As another in-the-standard-library uses case: my additions to the ipaddress 
test suite are really crying out for parameterised test support.

My current solution is adequate for coverage and debugging purposes (a custom 
assert applied to multiple values in a test case), but has the known 
limitations of that approach (specifically, only the first failing case gets 
reported rather than all failing cases, which can sometimes slow down the 
debugging process).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Borja Ruiz Castro]

Borja Ruiz Castro br...@alienvault.com added the comment:

Hi Murray!

I use a lot od parametrized tests. I usually use the ENV to pass these
parameters and/or a custon configuration file.

What is your approach to parametrize all the test stuff?

Regards,

Borja.

On 31 May 2012 03:57, R. David Murray rep...@bugs.python.org wrote:


 R. David Murray rdmur...@bitdance.com added the comment:

 People interested in this issue might be interested in changeset
 e6a33938b03f.  I use parameterized unit tests in email a lot, and was
 annoyed by the fact that I couldn't run the tests individually using the
 unittest CLI.  The fix for that turned out to be trivial, but by the time I
 figured it out, I'd already written most of the metaclass.  So since the
 metaclass reduces the boilerplate (albeit at the cost of looking like black
 magic), I decided to go with it.  And the metaclass at least avoids the
 rather questionable use of the modify locals() hack I was using before.

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue7897
 ___
 ___
 Python-bugs-list mailing list
 Unsubscribe:
 http://mail.python.org/mailman/options/python-bugs-list/isec%40alienvault.com



--
nosy: +borja.ruiz

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Borja Ruiz Castro]

Borja Ruiz Castro br...@alienvault.com added the comment:

Sorry, I failed to mention that I use Testify to launch all my tests!

On 2 July 2012 13:23, Borja Ruiz Castro rep...@bugs.python.org wrote:


 Borja Ruiz Castro br...@alienvault.com added the comment:

 Hi Murray!

 I use a lot od parametrized tests. I usually use the ENV to pass these
 parameters and/or a custon configuration file.

 What is your approach to parametrize all the test stuff?

 Regards,

 Borja.

 On 31 May 2012 03:57, R. David Murray rep...@bugs.python.org wrote:

 
  R. David Murray rdmur...@bitdance.com added the comment:
 
  People interested in this issue might be interested in changeset
  e6a33938b03f.  I use parameterized unit tests in email a lot, and was
  annoyed by the fact that I couldn't run the tests individually using the
  unittest CLI.  The fix for that turned out to be trivial, but by the
 time I
  figured it out, I'd already written most of the metaclass.  So since the
  metaclass reduces the boilerplate (albeit at the cost of looking like
 black
  magic), I decided to go with it.  And the metaclass at least avoids the
  rather questionable use of the modify locals() hack I was using before.
 
  --
 
  ___
  Python tracker rep...@bugs.python.org
  http://bugs.python.org/issue7897
  ___
  ___
  Python-bugs-list mailing list
  Unsubscribe:
 
 http://mail.python.org/mailman/options/python-bugs-list/isec%40alienvault.com
 
 

 --
 nosy: +borja.ruiz

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue7897
 ___


--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

I like test_email’s decorator.  It looks like 
https://github.com/wolever/nose-parameterized which I’m using.   (The 
implementation and generation of test method names may be less nice than what 
was discussed here, but the API (decorator + list of arguments) is something I 
like (much better than nose’s built-in parametrization).

--
versions: +Python 3.4 -Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

People interested in this issue might be interested in changeset e6a33938b03f.  
I use parameterized unit tests in email a lot, and was annoyed by the fact that 
I couldn't run the tests individually using the unittest CLI.  The fix for that 
turned out to be trivial, but by the time I figured it out, I'd already written 
most of the metaclass.  So since the metaclass reduces the boilerplate (albeit 
at the cost of looking like black magic), I decided to go with it.  And the 
metaclass at least avoids the rather questionable use of the modify locals() 
hack I was using before.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

I just remembered that many of the urllib.urlparse tests are guilty of only 
reporting the first case that fails, instead of testing everything and 
reporting all failures:

http://hg.python.org/cpython/file/default/Lib/test/test_urlparse.py

IMO, it would make a good guinea pig for any upgrade to the stdlib support for 
parameterised unit tests.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

FWIW I think nose2 is going to have test load time parameterized tests rather 
than run time parameterized tests, which is what I think we should do for 
unit test. The API should be as simple as possible for basic cases, but 
suitable for more complex cases too.

I quite liked Nicks idea of allowing a custom case builder but providing a 
sane default one.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[holger krekel]

Changes by holger krekel holger.kre...@gmail.com:


--
nosy: +hpk

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Julian Berman]

Changes by Julian Berman julian+python@grayvines.com:


--
nosy: +Julian

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

Another nice API: http://feldboris.alwaysdata.net/blog/unittest-template.html

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

Well, pyflakes will tell you about name clashes within a TestCase (unless 
you're shadowing a test on a base class which I guess is rarely the case)...

When we generate the tests we could add the parameter reprs to the docstring. A 
decorator factor that takes arguments and an optional name builder seem like a 
reasonable api to me.

Actually, name clashes *aren't* a problem - as we're generating TestCase 
instances these generated tests won't shadow existing tests (you'll just have 
two tests with the same name - does this mean id() may not be unique - worth 
checking).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

Note that name clashes *would* result in non-unique testcase ids, so we need to 
prevent that.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Please implement name+argtuple first and build auto-naming on top of that.  
Nick's approach would not allow me to specify a custom (hand coded) name for 
each set of arguments, which is my normal use case.  I also would not like the 
arguments auto-generated into the docstring unless that was optional, since I 
often have quite substantial argument tuples and it would just clutter the 
output to have them echoed in the docstring.  In my use cases the custom name 
is more useful than seeing the actual parameters.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

David, I don't understand - it looks like Nick's suggestion would allow you to 
create a name per case, that's the point of it!

You could even do this:

def _name_from_case(name, cases):
for idx, case in enumerate(cases, start=1):
test_name = case[0]
params = case[1:]
yield (test_name, params)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Oh, I see.  Make the name the first element of the argtuple and then strip it 
off.  Well, that will work, it just seems bass-ackwards to me :)

And why is it 'case'?  I thought we were talking about tests.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Changes by Nick Coghlan ncogh...@gmail.com:


--
Removed message: http://bugs.python.org/msg140819

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Sorry, misclicked and removed this comment from David:

Oh, I see.  Make the name the first element of the argtuple and then strip it 
off.  Well, that will work, it just seems bass-ackwards to me :)

And why is it 'case'?  I thought we were talking about tests.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

In my example, I needed a word to cover each entry in the collection of 
parameter tuples. 'case' fit the bill.

The reason I like the builder approach is that it means the simplest usage is 
to just create a list (or other iterable) of parameter tuples to be tested, 
then pass that list to the decorator factory. The sequential naming will then 
let you find the failing test cases in the sequence.

Custom builders then cover any cases where better naming is possible and 
desirable (such as explicitly naming each case as part of the parameters).

One refinement that may be useful is for the builders to produce (name, 
description, parameters) 3-tuple rather than 2-tuples, though. Then the default 
builder could just insert repr(params) as the description, while David's custom 
builder could either leave the description blank, or include a relevant subset 
of the parameters.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

 Well, pyflakes will tell you about name clashes within a TestCase
Seen that :)

 (unless you're shadowing a test on a base class which I guess is
 rarely the case)...
That too, in our own test suite (test_list or test_tuple, I have that change in 
a Subversion or old hg clone somewhere).  I agree about “rarely”, however.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Andrew Bennetts]

Andrew Bennetts s...@users.sourceforge.net added the comment:

You may be interested an existing, unittest-compatible library that provides 
this: http://pypi.python.org/pypi/testscenarios

--
nosy: +spiv

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Austin Bingham]

Austin Bingham austin.bing...@gmail.com added the comment:

Has a decision been made to implement some form of parametric tests? Is working 
being done?

Along with parameterizing individual test methods, I'd also like to throw out a 
request for parametric TestCases. In some cases I find that I want the behavior 
of setUp() or tearDown() to vary based on some set of parameters, but the 
individual tests are not parametric per se.

--
nosy: +abingham

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Brian Curtin]

Brian Curtin br...@python.org added the comment:

By this issue existing, that's the decision that we should probably do this, 
and I think the discussion shows we agree it should happen. How it's done is 
another way, and we have roughly a year to get it figured out before 3.3 gets 
closer.

I have a patch that implements this as a decorator, but it's out of date by now 
and not feature complete. It's on my todo list of patches to revive.

--
components: +Tests
stage:  - needs patch
versions: +Python 3.3 -Python 2.7, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Brian, if you don't have time to work on it in the next little while, maybe you 
could post your partial patch in case someone else wants to work on it?  Might 
be a good project for someone on the mentoring list.

Unless someone sees a clever way to implement both with the same mechanism, 
parameterizing test cases should probably be a separate issue.  Doing it via 
subclassing is straightforward, which also argues for a separate issue, since 
an 'is this worth it' discussion should be done separately for that feature.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

*If* we add this to unittest then we need to decide between test load time 
parameterised tests and test run time parameterisation. 

Load time is more backwards compatible / easier (all tests can be generated at 
load time and the number of tests can be known). Run time is more useful. (With 
load time parameterisation the danger is that test generation can fail so we 
need to have the test run not bomb out in this case.)

A hack for run time parameterisation is to have all tests represented by a 
single test but generate a single failure that represents all the failures. I 
think this would be an acceptable approach. It could still be done with a 
decorator.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

And yes, parameterising test cases is a different issue. bzr does this IIRC. 
This is easier in some ways, and can be done through load_tests, or any other 
test load time mechanism.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Michael, would your single test clearly indicate all the individual failures 
by name?  If not, then I would not find it useful.  I can already easily 
parameterize inside a single test using a loop, it's the detailed reporting 
piece that I want support for.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

The reporting piece, and ideally being able to use the arguments to unittest to 
run a single one of the parameterized tests.  (I can get the reporting piece 
now using the locals() hack, but that doesn't support test selection).  Does 
test selection support required load time parameterization?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

Test selection would require load time parameterisation - although the current 
test selection mechanism is through importing which would probably *not* work 
without a specific fix. Same for run time parameterisation.

Well how *exactly* you generate the names is an open question, and once you've 
solved that problem it should be no harder to show them clearly in the failure 
message with a single test report than with multiple test reports.

The way to generate the names is to number each test *and* show the 
parameterised data as part of the name (which can lead to *huge* names if 
you're not careful - or just object reprs in names which isn't necessarily 
useful). I have a decorator example that does runtime parameterisation, 
concatenating failures to a single report but still keeping the generated name 
for each failure.

Another issue is whether or not parameterised tests share a TestCase instance 
or have separate ones. If you're doing load time generation it makes sense to 
have a separate test case instance, with setUp and tearDown run individually. 
This needs to be clearly documented as the parameter generation would run 
against an uninitialised (not setUp) testcase.

Obviously reporting multiple test failures separately (run time 
parameterisation) is a bit nicer, but runtime test generation doesn't play well 
with anything that works with test suites - where you expect all tests to be 
represented by a single test case instance in the suite. I'm not sure that's a 
solveable problem.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

Dammit, I've reversed my thinking in some of those messages. Load time 
parameterisation *does* give you separate test reporting. It is run time 
parameterisation that doesn't.

Depending on how you do it (i.e. if the decorator generates the tests and 
attaches them as TestCase methods) you can do it at import time. And then 
normal test selection works if you know the test name.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

So if we do import time *or* test load time parameterisation then we can do 
separate failure reporting. We may still want to improve test selection for 
parameterised tests.

There are use cases for run time parameterisation (for example generating tests 
from a database that is only available during the test run was one use case 
someone has that can't be met by early parameterisation).

If we do run time parameterisation then I think we can only maintain 
compatibility with existing uses of test suites by having a single test case 
instance per parameterised test, and single combined failure report.

I would prefer *not* to add run time parameterisation.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Éric Araujo]

Changes by Éric Araujo mer...@netwok.org:


--
nosy: +eric.araujo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Load time parameterisation seems more of a worthwhile addition to me, too. As 
David noted, runtime parameterisation is pretty easy to do by looping and 
consolidating failures into the one error message via self.fail().

For test naming, trying to get too clever seems fraught with peril. I'd be 
happy with something like:
1. Parameterised tests are just numbered sequentially by default
2. The repr of the test parameters is included when displaying detailed error 
messages
3. A hook is provided to allow customisation of the naming scheme and test 
parameters. This hook receives an iterable of test parameters and is expected 
to create a new iterable producing 2-tuples of test names and parameters.

The default behaviour of the parameterised test generator could then be 
something like:

def parameterised_test_info(name, cases):
for idx, case in enumerate(cases, start=1):
yield ({}_{}.format(name, idx), case)

The new machinery (however provided) would then take care of checking the names 
are unique, adding those methods to the test case, and storing the parameters 
somewhere convenient (likely as attributes of the created methods) for use when 
running the test and when displaying error messages.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +ericsnow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

That all sounds good to me Nick. Some notes / questions.

How should parameterised tests be marked? I'm happy with a 
unittest.parameterized decorator (it would do no work other than mark the test 
method, with the parameterisation being done in the TestLoader).

What could the name customisation hook look like? A module level hook (yuck) 
/ a class method hook on the TestCase that must handle every parameterised test 
on that TestCase / a decorator for the parameterised test method?

If we do it at load time should we require parameterised methods to be class 
methods? The alternative is to instantiate the test case when loading and 
collecting the tests. Class methods won't play well with the other unittest 
decorators as you can't attach attributes to classmethods. (So I guess instance 
methods it is!)

If collecting tests fails part way through we should generate a failing test 
that reports the exception. Should the tests collected so far be kept?

Should skip / expectedFail decorators still work with them? If so they'll need 
custom support I expect.

If we're generating test names and then attaching these tests to TestCase 
instances (so that normal test case execution will run them), should we check 
for name clashes? What should we do if there is a name clash? (The generated 
name would shadow an existing name.) We could prevent that by using a 
non-identifier name - e.g. ${}_{}.format(name, idx)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

Oh, and if we're not going to get clever with naming, how is the TestResult 
going to include the parameter repr in the failure report? That information 
will have to be stored on the TestCase. 

I would prefer this feature not to touch the TestResult - so any failure 
message cleverness should be done in the TestCase. It will be harder to 
maintain if this feature is spread out all the way through unittest.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Here's a sketch for a possible decorator factory:

def parameters(params, *, builder=_default_param_builder):
def make_parameterized_test(f):
return ParameterizedTest(f, params, builder)
return make_parameterized_test

(default builder would be the one I sketched earlier)

Example usage:

ExampleParams = namedtuple('ExampleParams', 'x y')
all_cases = [ExampleParams(x, y) for x in range(3) for y in range(3)]

def x_y_naming(name, cases):
for idx, case in enumerate(cases, start=1):
x, y = case
yield ({}_{}_{}_{}.format(name, idx, x, y), case)

class ExampleTestCase(TestCase):
  @parameters(all_cases)
  def test_example_auto_naming(self, x, y):
  self.assertNotEqual(x, y) # fails sometimes :)

  @parameters(all_cases, builder=x_y_naming)
  def test_example_custom_naming(self, x, y):
  self.assertNotEqual(x, y) # fails sometimes :)

Once defined, you're far better equipped than I am to decide how TestLoader and 
TestCase can best cooperate to generate appropriate test operations and error 
messages.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

Personally I would be happy if I could pass in a dictionary that maps names to 
argument tuples, or an iterator yielding (name, (argtuple)) pairs, and just 
have the failure report the name.  That is, make me responsible for generating 
meaningful names, don't autogenerate them.  Then someone could provide a name 
autogenerate helper on top of that base interface. 

Checking for name clashes would be nice, but it would be nice to have that for 
non-parameterized test cases, too (I've been bitten by that a number of time ;)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Ezio Melotti]

Changes by Ezio Melotti ezio.melo...@gmail.com:


--
nosy: +ezio.melotti

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Austin Bingham]

Austin Bingham austin.bing...@gmail.com added the comment:

OK, I created issue 12600 for dealing with parameterized TestCases as
opposed to individual tests.

On Wed, Jul 20, 2011 at 4:03 PM, R. David Murray rep...@bugs.python.org wrote:
 Unless someone sees a clever way to implement both with the same mechanism, 
 parameterizing test cases should probably be a separate issue.  Doing it via 
 subclassing is straightforward, which also argues for a separate issue, since 
 an 'is this worth it' discussion should be done separately for that feature.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[nchauvat]

nchauvat nicolas.chau...@logilab.fr added the comment:

In case it could be useful, here is how generative/parametrized tests are 
handled in logilab.common.testlib 
http://hg.logilab.org/logilab/common/file/a6b5fe18df99/testlib.py#l1137

--
nosy: +nchauvat

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

No, I'm saying I don't see summarising the parameterised tests separately from 
the overall test run as a particularly important feature, since you can test 
multiple parameters in a single test manually now.

The important part is for the framework to be able to generate multiple tests 
from a single test function with a selection of arguments.  Doing a separate 
run with just those tests will give you any information that a summary would 
give you.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

I agree with Michael - one test that covers multiple settings can easily be 
done by collecting results within the test itself and then checking at the end 
that no failures were detected (e.g. I've done this myself with a test that 
needed to be run against multiple input files - the test knew the expected 
results and maintained lists of filenames where the result was incorrect. At 
the end of the test, if any of those lists contained entries, the test was 
failed, with the error message giving details of which files had failed and 
why).

What parameterised tests could add which is truly unique is for each of those 
files to be counted and tracked as a separate test. Sometimes the 
single-test-with-internal-failure-recording will still make more sense, but 
tracking the tests individually will typically give a better indication of 
software health (e.g. in my example above, the test ran against a few dozen 
files, but the only way to tell if it was one specific file that was failing or 
all of them was to go look at the error message).

--
nosy: +ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Yaroslav Halchenko]

Yaroslav Halchenko yarikop...@gmail.com added the comment:

Hi Nick,

Am I reading your right, Are you suggesting to implement this
manual looping/collecting/reporting separately in every unittest
which needs that?

On Tue, 11 May 2010, Nick Coghlan wrote:
 Nick Coghlan ncogh...@gmail.com added the comment:

 I agree with Michael - one test that covers multiple settings can easily be 
 done by collecting results within the test itself and then checking at the 
 end that no failures were detected (e.g. I've done this myself with a test 
 that needed to be run against multiple input files - the test knew the 
 expected results and maintained lists of filenames where the result was 
 incorrect. At the end of the test, if any of those lists contained entries, 
 the test was failed, with the error message giving details of which files had 
 failed and why).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Fernando Perez]

Fernando Perez fdo.pe...@gmail.com added the comment:

Hey Yarick,

On Thu, Apr 8, 2010 at 18:53, Yaroslav Halchenko rep...@bugs.python.org w=
rote:
 In PyMVPA we have our little decorator as an alternative to Fernando's ge=
nerators, =A0and which is closer, I think, to what Michael was wishing for:
 @sweepargs

 http://github.com/yarikoptic/PyMVPA/blob/master/mvpa/testing/sweepargs.py

 NB it has some minor PyMVPA specificity which could be easily wiped out, =
and since it was at most 4 eyes looking at it and it bears evolutionary c=
hanges, it is far from being the cleanest/best piece of code, BUT:

 * it is very easy to use, just decorate a test method/function and give a=
n argument which to vary within the function call, e.g smth like

 @sweepargs(arg=3Drange(5))
 def test_sweepargs_demo(arg):
 =A0 =A0ok_(arg  5)
 =A0 =A0ok_(arg  3)
 =A0 =A0ok_(arg  2)

 For nose/unittest it would still look like a single test

Thanks for the post; I obviously defer to Michael on the final
decision, but I *really* would like a solution that reports an
'argument sweep' as multiple tests, not as one.  They are truly
multiple tests (since they can pass/fail independently), so I think
they should be treated as such.

On the other hand, your code does have nifty features that could be
used as well, so perhaps the best of both can be used in the end.

Cheers,

f

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Yaroslav Halchenko]

Yaroslav Halchenko yarikop...@gmail.com added the comment:

Fernando, I agree... somewhat ;-)

At some point (whenever everything works fine and no unittests fail) I wanted 
to merry sweepargs to nose and make it spit out a dot (or animate a spinning 
wheel ;)) for every passed unittest, so instead of 300 dots I got a picturesque 
field of thousands dots and Ss and also saw how many were skipped for some 
parametrizations.  But I became Not sure of such feature since field became 
quite large and hard to grasp visually although it gave me better idea indeed 
of what was the total number of testings were done and skipped.  So may be it 
would be helpful to separate notions of tests and testings and provide user 
ability to control the level of verbosity (1 -- tests, 2 -- testings, 3 -- 
verbose listing of testings (test(parametrization)))

But I blessed sweepargs every time whenever something goes nuts and a test 
starts failing for (nearly) all parametrization at the same point.  And that is 
where I really enjoy the concise summary.
Also I observe that often an ERROR bug reveals itself through multiple tests.  
So, may be it would be worth developing a generic 'summary' output which would 
collect all tracebacks and then groups them by the location of the actual 
failure and tests/testings which hit it?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Fernando Perez]

Fernando Perez fdo.pe...@gmail.com added the comment:

Yarick: Yes, I do actually see the value of the summary view.  When I have a 
parametric test that fails, I tend to just run nose with -x so it stops at the 
first error and with the --pdb options to study it, so I simply ignore all the 
other failures.  To me, test failures are quite often like compiler error 
messages: if there's a lot of them, it's best to look only at the first few, 
fix those and try again, because the rest could be coming from the same cause.

I don't know if Michael has plans/bandwidth to add the summary support as well, 
but I agree it would be very nice to have, just not at the expense of 
individual reporting.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

If we provide builtin support for parameterized tests it will have to report 
each test separately otherwise there is no point. You can already add support 
for running tests with multiple parameters yourself - the *only* advantage of 
building support into unittest (as I see it) is for better reporting of 
individual tests.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Yaroslav Halchenko]

Yaroslav Halchenko yarikop...@gmail.com added the comment:

In PyMVPA we have our little decorator as an alternative to Fernando's 
generators,  and which is closer, I think, to what Michael was wishing for:
@sweepargs

http://github.com/yarikoptic/PyMVPA/blob/master/mvpa/testing/sweepargs.py

NB it has some minor PyMVPA specificity which could be easily wiped out, and 
since it was at most 4 eyes looking at it and it bears evolutionary changes, 
it is far from being the cleanest/best piece of code, BUT:

* it is very easy to use, just decorate a test method/function and give an 
argument which to vary within the function call, e.g smth like

@sweepargs(arg=range(5))
def test_sweepargs_demo(arg):
ok_(arg  5)
ok_(arg  3)
ok_(arg  2)

For nose/unittest it would still look like a single test

* if failures occur, sweepargs groups failures by the type/location of the 
failures and spits out a backtrace for one of failures + summary (instead of 
detailed backtraces for each failure) specifying which arguments lead to what 
error... here is the output for example above:

$ nosetests -s test_sweepargs_demo.py
F
==
FAIL: mvpa.tests.test_sweepargs_demo.test_sweepargs_demo
--
Traceback (most recent call last):
  File /usr/lib/pymodules/python2.5/nose/case.py, line 183, in runTest
self.test(*self.arg)
  File /usr/lib/pymodules/python2.5/nose/util.py, line 630, in newfunc
return func(*arg, **kw)
  File /home/yoh/proj/pymvpa/pymvpa/mvpa/tests/test_sweepargs_demo.py, line 
11, in test_sweepargs_demo
ok_(arg  2)
  File /usr/lib/pymodules/python2.5/nose/tools.py, line 25, in ok_
assert expr, msg
AssertionError: 
 Different scenarios lead to failures of unittest test_sweepargs_demo (specific 
tracebacks are below):
  File /home/yoh/proj/pymvpa/pymvpa/mvpa/tests/test_sweepargs_demo.py, line 
10, in test_sweepargs_demo
ok_(arg  3)
File /usr/lib/pymodules/python2.5/nose/tools.py, line 25, in ok_
assert expr, msg
  on
arg=3 
arg=4 

  File /home/yoh/proj/pymvpa/pymvpa/mvpa/tests/test_sweepargs_demo.py, line 
11, in test_sweepargs_demo
ok_(arg  2)
File /usr/lib/pymodules/python2.5/nose/tools.py, line 25, in ok_
assert expr, msg
  on
arg=2 

--
Ran 1 test in 0.003s

FAILED (failures=1)

* obviousely multiple decorators could be attached to the same test, to test on 
all combinations of more than 1 argument but output atm is a bit cryptic ;-)

--
nosy: +Yaroslav.Halchenko

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

I'm not sure what this brings. It is easy to write a loop iterating over test 
data.
What parametric testing could bring is precise progress and error display 
(displaying the parameters for each run), but this doesn't seem to be the case 
here, since you are not making the parameters available to unittest. Your 
examples just boil down to writing yield is_smaller(x, y) instead of 
self.assertLess(x, y)...

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

By the way - I have no opinion on whether or not using yield is the right way 
to support parameterized tests. It may be better for the test method to take 
arguments, and be decorated as a parameterized test, with the decorator 
providing the parameters. When I come to look at it I will look at how py.test 
and nose do it and solicit advice on the Testing in Python list. We had a 
useful discussion there previously that would be good to refer to.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Jean-Paul Calderone]

Jean-Paul Calderone exar...@divmod.com added the comment:

 With paramterized tests *all* the tests are run and *all* failures reported. 
 With testing in a loop the tests stop at the first failure.

+1 to this justification.  Parameterized tests are a big win over a simple for 
loop in a test.

(However, I haven't looked at the IPython code at all, and Antoine's objection 
seemed to have something in particular to do with the IPython code?)

--
nosy: +exarkun

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Brian Curtin]

Brian Curtin cur...@acm.org added the comment:

 It may be better for the test method to take arguments, and 
 be decorated as a parameterized test, with the decorator 
 providing the parameters.

+1 on something like this. That's also how NUnit supports parameterized tests.

--
nosy: +brian.curtin

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

  With paramterized tests *all* the tests are run and *all* failures 
  reported. With testing in a loop the tests stop at the first failure.
 
 +1 to this justification.  Parameterized tests are a big win over a
 simple for loop in a test.

Ah, thank you. Looks better indeed.

 (However, I haven't looked at the IPython code at all, and Antoine's
 objection seemed to have something in particular to do with the
 IPython code?)

No, it has to do that you need to be able to distinguish the different
runs of your parameterized test (the same way you distinguish between
different test methods by their names, or line numbers, in the test
report).

If I have 500 runs in my parameterized (*) test, and the only failure
reported is that 2 is not greater than 3, I don't know where to start
looking because the traceback won't give me the information of *which*
parameters were currently in use.

(*) (this is horrible to type)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Michael Foord mich...@voidspace.org.uk added the comment:

Antoine: the failure message would include a repr of the parameters used in the 
particular test that failed. So you can tell which test failed and with what 
parameters.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Jean-Paul Calderone]

Jean-Paul Calderone exar...@divmod.com added the comment:

Something else I think it would be nice to consider is what the id() (and 
shortDescription(), heh) of the resulting tests will be.

It would be great if the id were sufficient to identify a particular test *and* 
data combination.

In trial, we're trying to use test ids to support distributed test running.  If 
the above property holds for parameterized tests, then we'll be able to 
automatically distribute them to different processes or hosts to be run.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Fernando Perez]

Fernando Perez fdo.pe...@gmail.com added the comment:

I should probably have clarified better our reasons for using this type of 
code.  The first is the one Michael pointed out, where such parametric tests 
all execute; it's very common in scientific computing to have algorithms that 
only fail for certain values, so it's important to identify these points of 
failure easily while still running the entire test suite.  

The second is that the approach nose uses produces on failure the nose stack, 
not the stack of the test. Nose consumes the test generators at test discovery 
time, and then simply calls the stored assertions at test execution time.  If a 
test fails, you see a nose traceback which is effectively useless for debugging 
and with which using --pdb for interactive debugging doesn't help much (all you 
can do is print the values, as your own stack is gone).  This code, in 
contrast, evaluates the full test at execution time, so a failure can be 
inspected 'live'.  In practice this makes an enormous difference in a test 
suite being actively useful for ongoing development where changes may send you 
into debugging often.

I hope this helps clarify the intent of the code better, I'd be happy to 
provide further details.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Michael Foord]

Changes by Michael Foord mich...@voidspace.org.uk:


--
assignee:  - michael.foord
nosy: +michael.foord

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7897] Support parametrized tests in unittest

[Fernando Perez]

New submission from Fernando Perez fdo.pe...@gmail.com:

IPython has unittest-based parametric testing (something nose has but
which produces effectively undebuggable tests, while this approach
gives perfectly debuggable ones).  The code lives here for 2.x and
3.x:

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/testing/_paramtestpy2.py

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/testing/_paramtestpy3.py

we import them into our public decorators module for actual use:

http://bazaar.launchpad.net/~ipython-dev/ipython/trunk/annotate/head%3A/IPython/testing/decorators.py

Simple tests showing them in action are here:

http://bazaar.launchpad.net/%7Eipython-dev/ipython/trunk/annotate/head%3A/IPython/testing/tests/test_decorators.py#L45

The code is all BSD and we'd be more than happy to see it used
upstream; the less we have to carry the better.

If there is interest in this code, I'm happy to sign a PSF contributor 
agreement, the code is mostly my authorship. I received help for the 3.x 
version on the Testing in Python mailing list, I would handle asking for 
permission on-list if there is interest in including this.

--
messages: 99149
nosy: fperez
severity: normal
status: open
title: Support parametrized tests in unittest
type: feature request
versions: Python 2.7, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7897
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com