[openstack-dev] [Tempest] Which is the best way for skipping tests?

2013-12-04 Thread Daisuke Morita
Hi, everyone. Which do you think is the best way of coding test skipping, writing cls.skipException statement in setUpClass method or skipIf annotation for each test method ? This question comes to me in reviewing https://review.openstack.org/#/c/59759/ . I think that work itself is great and I

Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?

2013-12-04 Thread Sean Dague
I agree, preference should be to the function level skip vs. the class exception. Especially as I have some sample fixture code from Robert that will remove setUpClass in the future (we do have a long term goal of getting rid of it in favor of proper fixtures). It also gives us an actual count of

Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?

2013-12-04 Thread Joe Hakim Rahme
I am in favor of class level exceptions for the obvious reasons: + It reduces code duplication. Copy/pasting a SkipIf decorator on every test method in the class is tedious and possibly error prone. Adding the exception as a guard in the setUpClass() makes for a more elegant solution +

Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?

2013-12-04 Thread Sean Dague
On 12/04/2013 09:24 AM, Joe Hakim Rahme wrote: I am in favor of class level exceptions for the obvious reasons: + It reduces code duplication. Copy/pasting a SkipIf decorator on every test method in the class is tedious and possibly error prone. Adding the exception as a guard in the

Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?

2013-12-04 Thread Joe Hakim Rahme
On 04 Dec 2013, at 17:05, Sean Dague s...@dague.net wrote: That will require someone signing up to writing that though. I could do that. Since you know the code better than me, can you confirm that tempest/test.py is the best place to define this decorator? Thanks. --- Joe H. Rahme

Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?

2013-12-04 Thread Sean Dague
On 12/04/2013 11:32 AM, Joe Hakim Rahme wrote: On 04 Dec 2013, at 17:05, Sean Dague s...@dague.net wrote: That will require someone signing up to writing that though. I could do that. Since you know the code better than me, can you confirm that tempest/test.py is the best place to define

Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?

2013-12-04 Thread Brant Knudson
In Keystone, we've got some tests that raise self.skipTest('...') in the test class setUp() method (not setUpClass). My testing shows that if there's several tests in the class then it shows all of those tests as skipped (not just 1 skip). Does this do what you want? Here's an example:

Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?

2013-12-04 Thread Daisuke Morita
Thanks for your suggestion, Brant. And, I experimented multiple approaches like that in Tempest, annotating testtools.skipIf above setUp() method can deliver a similar result. How about this annotation approach, Joe? This approach can meet both requirements we said, avoiding code

Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?

2013-12-04 Thread Robert Collins
On 5 December 2013 03:24, Joe Hakim Rahme joe.hakim.ra...@enovance.com wrote: I am in favor of class level exceptions for the obvious reasons: ... If proper fixtures are meant to replace setUpClass in the future (something I would really love to see in Tempest), we still need to take into