Hi, everyone. Which do you think is the best way of coding test skipping, "writing cls.skipException statement in setUpClass method" or "skipIf annotation for each test method" ?
This question comes to me in reviewing https://review.openstack.org/#/c/59759/ . I think that work itself is great and I hope this patch is merged to Tempest. I just want to focus on coding styles and explicitness of test outputs. If skipIf annotation is used, test output of Swift is as follows. --- tempest.api.object_storage.test_account_quotas.AccountQuotasTest test_admin_modify_quota[gate,smoke] SKIP 1.15 test_upload_large_object[gate,negative,smoke] SKIP 0.03 test_upload_valid_object[gate,smoke] SKIP 0.03 test_user_modify_quota[gate,negative,smoke] SKIP 0.03 tempest.api.object_storage.test_account_services.AccountTest test_create_and_delete_account_metadata[gate,smoke] OK 0.32 test_list_account_metadata[gate,smoke] OK 0.02 test_list_containers[gate,smoke] OK 0.02 ...(SKIP)... Ran 54 tests in 85.977s OK --- On the other hand, if cls.skipException is used, an output is changed as follows. --- setUpClass (tempest.api.object_storage.test_account_quotas AccountQuotasTest) SKIP 0.00 tempest.api.object_storage.test_account_services.AccountTest test_create_and_delete_account_metadata[gate,smoke] OK 0.48 test_list_account_metadata[gate,smoke] OK 0.02 test_list_containers[gate,smoke] OK 0.02 ...(SKIP)... Ran 49 tests in 81.475s OK --- I believe the output of the code using skipIf annotation is better. Since the coverage of tests is displayed more definitely, it is easier to find out what tests are really skipped. I scanned the whole code of Tempest. The count of cls.skipException statements is 63, and the count of skipIf annotations is 24. Replacing them is not trivial task, but I think the most impportant for testing is to output consistent and accurate log. Am I missing something? Or, this kind of discussion has been done already in the past? If so, could you let me know? Best Regards, -- Daisuke Morita <[email protected]> NTT Software Innovation Center, NTT Corporation _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
