[openstack-dev] [Tempest] Which is the best way for skipping tests?
Hi, everyone. Which do you think is the best way of coding test skipping, writing cls.skipException statement in setUpClass method or skipIf annotation for each test method ? This question comes to me in reviewing https://review.openstack.org/#/c/59759/ . I think that work itself is great and I hope this patch is merged to Tempest. I just want to focus on coding styles and explicitness of test outputs. If skipIf annotation is used, test output of Swift is as follows. --- tempest.api.object_storage.test_account_quotas.AccountQuotasTest test_admin_modify_quota[gate,smoke] SKIP 1.15 test_upload_large_object[gate,negative,smoke] SKIP 0.03 test_upload_valid_object[gate,smoke] SKIP 0.03 test_user_modify_quota[gate,negative,smoke] SKIP 0.03 tempest.api.object_storage.test_account_services.AccountTest test_create_and_delete_account_metadata[gate,smoke] OK 0.32 test_list_account_metadata[gate,smoke]OK 0.02 test_list_containers[gate,smoke] OK 0.02 ...(SKIP)... Ran 54 tests in 85.977s OK --- On the other hand, if cls.skipException is used, an output is changed as follows. --- setUpClass (tempest.api.object_storage.test_account_quotas AccountQuotasTest) SKIP 0.00 tempest.api.object_storage.test_account_services.AccountTest test_create_and_delete_account_metadata[gate,smoke] OK 0.48 test_list_account_metadata[gate,smoke]OK 0.02 test_list_containers[gate,smoke] OK 0.02 ...(SKIP)... Ran 49 tests in 81.475s OK --- I believe the output of the code using skipIf annotation is better. Since the coverage of tests is displayed more definitely, it is easier to find out what tests are really skipped. I scanned the whole code of Tempest. The count of cls.skipException statements is 63, and the count of skipIf annotations is 24. Replacing them is not trivial task, but I think the most impportant for testing is to output consistent and accurate log. Am I missing something? Or, this kind of discussion has been done already in the past? If so, could you let me know? Best Regards, -- Daisuke Morita morita.dais...@lab.ntt.co.jp NTT Software Innovation Center, NTT Corporation ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?
I agree, preference should be to the function level skip vs. the class exception. Especially as I have some sample fixture code from Robert that will remove setUpClass in the future (we do have a long term goal of getting rid of it in favor of proper fixtures). It also gives us an actual count of the number of tests skipped in the configuration, which is nice. -Sean On 12/04/2013 06:46 AM, Daisuke Morita wrote: Hi, everyone. Which do you think is the best way of coding test skipping, writing cls.skipException statement in setUpClass method or skipIf annotation for each test method ? This question comes to me in reviewing https://review.openstack.org/#/c/59759/ . I think that work itself is great and I hope this patch is merged to Tempest. I just want to focus on coding styles and explicitness of test outputs. If skipIf annotation is used, test output of Swift is as follows. --- tempest.api.object_storage.test_account_quotas.AccountQuotasTest test_admin_modify_quota[gate,smoke] SKIP 1.15 test_upload_large_object[gate,negative,smoke] SKIP 0.03 test_upload_valid_object[gate,smoke] SKIP 0.03 test_user_modify_quota[gate,negative,smoke] SKIP 0.03 tempest.api.object_storage.test_account_services.AccountTest test_create_and_delete_account_metadata[gate,smoke] OK 0.32 test_list_account_metadata[gate,smoke]OK 0.02 test_list_containers[gate,smoke] OK 0.02 ...(SKIP)... Ran 54 tests in 85.977s OK --- On the other hand, if cls.skipException is used, an output is changed as follows. --- setUpClass (tempest.api.object_storage.test_account_quotas AccountQuotasTest) SKIP 0.00 tempest.api.object_storage.test_account_services.AccountTest test_create_and_delete_account_metadata[gate,smoke] OK 0.48 test_list_account_metadata[gate,smoke]OK 0.02 test_list_containers[gate,smoke] OK 0.02 ...(SKIP)... Ran 49 tests in 81.475s OK --- I believe the output of the code using skipIf annotation is better. Since the coverage of tests is displayed more definitely, it is easier to find out what tests are really skipped. I scanned the whole code of Tempest. The count of cls.skipException statements is 63, and the count of skipIf annotations is 24. Replacing them is not trivial task, but I think the most impportant for testing is to output consistent and accurate log. Am I missing something? Or, this kind of discussion has been done already in the past? If so, could you let me know? Best Regards, -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?
I am in favor of class level exceptions for the obvious reasons: + It reduces code duplication. Copy/pasting a SkipIf decorator on every test method in the class is tedious and possibly error prone. Adding the exception as a guard in the setUpClass() makes for a more elegant solution + function level skips will waste unnecessary time in the setup/teardown methods. If I know I'm skipping all the tests in a class, why should I bother executing all the boilerplate preliminary actions? In the context of heavy use, like the CI gate, this can accumulate and be a pain. + Using function level skips requires importing an extra module (from testtools import SkipIf) that would be otherwise unnecessary. If the output of the class level skipException needs to be improved, maybe there should be a patch there to list all the methods skipped. If proper fixtures are meant to replace setUpClass in the future (something I would really love to see in Tempest), we still need to take into account that setUpClass might do more than just fixtures, and certain guards are expected to be found in there. What do you guys think? --- Joe H. Rahme ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?
On 12/04/2013 09:24 AM, Joe Hakim Rahme wrote: I am in favor of class level exceptions for the obvious reasons: + It reduces code duplication. Copy/pasting a SkipIf decorator on every test method in the class is tedious and possibly error prone. Adding the exception as a guard in the setUpClass() makes for a more elegant solution + function level skips will waste unnecessary time in the setup/teardown methods. If I know I'm skipping all the tests in a class, why should I bother executing all the boilerplate preliminary actions? In the context of heavy use, like the CI gate, this can accumulate and be a pain. + Using function level skips requires importing an extra module (from testtools import SkipIf) that would be otherwise unnecessary. If the output of the class level skipException needs to be improved, maybe there should be a patch there to list all the methods skipped. If proper fixtures are meant to replace setUpClass in the future (something I would really love to see in Tempest), we still need to take into account that setUpClass might do more than just fixtures, and certain guards are expected to be found in there. What do you guys think? So I'd be ok with a compromise, which would build a decorator for the setUpClass method, at least that would make it easier to refactor out later. That will require someone signing up to writing that though. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?
On 04 Dec 2013, at 17:05, Sean Dague s...@dague.net wrote: That will require someone signing up to writing that though. I could do that. Since you know the code better than me, can you confirm that tempest/test.py is the best place to define this decorator? Thanks. --- Joe H. Rahme ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?
On 12/04/2013 11:32 AM, Joe Hakim Rahme wrote: On 04 Dec 2013, at 17:05, Sean Dague s...@dague.net wrote: That will require someone signing up to writing that though. I could do that. Since you know the code better than me, can you confirm that tempest/test.py is the best place to define this decorator? Yes, that would be the right place to add it. And thanks for signing up for this! -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?
In Keystone, we've got some tests that raise self.skipTest('...') in the
test class setUp() method (not setUpClass). My testing shows that if
there's several tests in the class then it shows all of those tests as
skipped (not just 1 skip). Does this do what you want?
Here's an example:
http://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/test_ipv6.py?id=73dbc00e6ac049f19d0069ecb07ca8ed75627dd5#n30
http://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/core.py?id=73dbc00e6ac049f19d0069ecb07ca8ed75627dd5#n500
- Brant
On Wed, Dec 4, 2013 at 5:46 AM, Daisuke Morita morita.dais...@lab.ntt.co.jp
wrote:
Hi, everyone.
Which do you think is the best way of coding test skipping, writing
cls.skipException statement in setUpClass method or skipIf annotation
for each test method ?
This question comes to me in reviewing
https://review.openstack.org/#/c/59759/ . I think that work itself is
great and I hope this patch is merged to Tempest. I just want to focus
on coding styles and explicitness of test outputs.
If skipIf annotation is used, test output of Swift is as follows.
---
tempest.api.object_storage.test_account_quotas.AccountQuotasTest
test_admin_modify_quota[gate,smoke]
SKIP 1.15
test_upload_large_object[gate,negative,smoke]
SKIP 0.03
test_upload_valid_object[gate,smoke]
SKIP 0.03
test_user_modify_quota[gate,negative,smoke]
SKIP 0.03
tempest.api.object_storage.test_account_services.AccountTest
test_create_and_delete_account_metadata[gate,smoke] OK
0.32
test_list_account_metadata[gate,smoke]OK
0.02
test_list_containers[gate,smoke] OK
0.02
...(SKIP)...
Ran 54 tests in 85.977s
OK
---
On the other hand, if cls.skipException is used, an output is changed as
follows.
---
setUpClass (tempest.api.object_storage.test_account_quotas
AccountQuotasTest)
SKIP 0.00
tempest.api.object_storage.test_account_services.AccountTest
test_create_and_delete_account_metadata[gate,smoke] OK
0.48
test_list_account_metadata[gate,smoke]OK
0.02
test_list_containers[gate,smoke] OK
0.02
...(SKIP)...
Ran 49 tests in 81.475s
OK
---
I believe the output of the code using skipIf annotation is better.
Since the coverage of tests is displayed more definitely, it is easier
to find out what tests are really skipped.
I scanned the whole code of Tempest. The count of cls.skipException
statements is 63, and the count of skipIf annotations is 24. Replacing
them is not trivial task, but I think the most impportant for testing is
to output consistent and accurate log.
Am I missing something? Or, this kind of discussion has been done
already in the past? If so, could you let me know?
Best Regards,
--
Daisuke Morita morita.dais...@lab.ntt.co.jp
NTT Software Innovation Center, NTT Corporation
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?
Thanks for your suggestion, Brant. And, I experimented multiple
approaches like that in Tempest, annotating testtools.skipIf above
setUp() method can deliver a similar result.
How about this annotation approach, Joe? This approach can meet both
requirements we said, avoiding code duplications and accurately logging
what are skipped. The demerit of this approach seems less intuitive than
annotating above setUpClass method, but you do not need to spend any
time to implement additional annotation.
Best Regards,
Daisuke Morita
(2013/12/05 8:59), Brant Knudson wrote:
In Keystone, we've got some tests that raise self.skipTest('...') in
the test class setUp() method (not setUpClass). My testing shows that if
there's several tests in the class then it shows all of those tests as
skipped (not just 1 skip). Does this do what you want?
Here's an example:
http://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/test_ipv6.py?id=73dbc00e6ac049f19d0069ecb07ca8ed75627dd5#n30
http://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/core.py?id=73dbc00e6ac049f19d0069ecb07ca8ed75627dd5#n500
- Brant
On Wed, Dec 4, 2013 at 5:46 AM, Daisuke Morita
morita.dais...@lab.ntt.co.jp mailto:morita.dais...@lab.ntt.co.jp wrote:
Hi, everyone.
Which do you think is the best way of coding test skipping, writing
cls.skipException statement in setUpClass method or skipIf annotation
for each test method ?
This question comes to me in reviewing
https://review.openstack.org/#/c/59759/ . I think that work itself is
great and I hope this patch is merged to Tempest. I just want to focus
on coding styles and explicitness of test outputs.
If skipIf annotation is used, test output of Swift is as follows.
---
tempest.api.object_storage.test_account_quotas.AccountQuotasTest
test_admin_modify_quota[gate,smoke]
SKIP 1.15
test_upload_large_object[gate,negative,smoke]
SKIP 0.03
test_upload_valid_object[gate,smoke]
SKIP 0.03
test_user_modify_quota[gate,negative,smoke]
SKIP 0.03
tempest.api.object_storage.test_account_services.AccountTest
test_create_and_delete_account_metadata[gate,smoke]
OK
0.32
test_list_account_metadata[gate,smoke]
OK
0.02
test_list_containers[gate,smoke]
OK
0.02
...(SKIP)...
Ran 54 tests in 85.977s
OK
---
On the other hand, if cls.skipException is used, an output is changed as
follows.
---
setUpClass (tempest.api.object_storage.test_account_quotas
AccountQuotasTest)
SKIP 0.00
tempest.api.object_storage.test_account_services.AccountTest
test_create_and_delete_account_metadata[gate,smoke]
OK
0.48
test_list_account_metadata[gate,smoke]
OK
0.02
test_list_containers[gate,smoke]
OK
0.02
...(SKIP)...
Ran 49 tests in 81.475s
OK
---
I believe the output of the code using skipIf annotation is better.
Since the coverage of tests is displayed more definitely, it is easier
to find out what tests are really skipped.
I scanned the whole code of Tempest. The count of cls.skipException
statements is 63, and the count of skipIf annotations is 24. Replacing
them is not trivial task, but I think the most impportant for testing is
to output consistent and accurate log.
Am I missing something? Or, this kind of discussion has been done
already in the past? If so, could you let me know?
Best Regards,
--
Daisuke Morita morita.dais...@lab.ntt.co.jp
mailto:morita.dais...@lab.ntt.co.jp
NTT Software Innovation Center, NTT Corporation
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Daisuke Morita morita.dais...@lab.ntt.co.jp
NTT Software Innovation Center, NTT Corporation
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Tempest] Which is the best way for skipping tests?
On 5 December 2013 03:24, Joe Hakim Rahme joe.hakim.ra...@enovance.com wrote: I am in favor of class level exceptions for the obvious reasons: ... If proper fixtures are meant to replace setUpClass in the future (something I would really love to see in Tempest), we still need to take into account that setUpClass might do more than just fixtures, and certain guards are expected to be found in there. So the point is to remove setUpClass use entirely. Please bear that in mind in whatever you come up with. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
