Re: [openstack-dev] [qa] negative test generation and testscenarios
Hi Robert, https://review.openstack.org/#/c/64733 is updated and is using testscenarios now. I was wondering if it would make sense to use a decorator instead of a class attribute to mark a test case for a certain test scenario, like: @TestCaseWithScenario(_scenarios1) def test_demo(self): ... @TestCaseWithScenario(_scenarios2) def test_demo2(self): I already implement something that seems to work with subunit: http://paste.openstack.org/show/61175/ I think this would ease up the usage of testscenarios for the negative test framework in tempest and looks quite cool. Feedback is welcome! Regards Marc From: Robert Collins [robe...@robertcollins.net] Sent: Tuesday, January 07, 2014 3:59 AM To: David Kranz Cc: OpenStack Development Mailing List Subject: Re: [openstack-dev] [qa] negative test generation and testscenarios On 7 January 2014 07:41, David Kranz dkr...@redhat.com wrote: Thanks to all who looked at https://review.openstack.org/#/c/64733/. There were a few minor issues I will address but the biggest one was the suggestion to run each variation as a separate test case using testscenarios. After looking into that I see a problem with this use case. Many of these negative tests need to allocate resources such as servers that are then referenced in the test cases. Our tests currently do stuff like that in setUpClass. But testscenarios, unless I misunderstand the code, replicates the whole class which means the resource setup would also be duplicated for each test. I think we need to share allocated resources across the potentially large number of negative variations, and see two options for that: You don't understand the code :). testscenarios doesn't affect the class of tests. As long as the resulting suite is wrapped in a setUpClass compatible TestSuite object, setUpClass should keep working. 1. Some magic similar to testscenarios but that creates test methods with varying parameters instead of test classes with varying attributes. Leave allocation in setUpClass. -1 No different to testscenarios, just different to be different. 2. Using testscenarios as-is but move the allocation/cleanup to the module level (fixtures?) This would be a little complicated because a bunch of the setup stuff is defined as methods on the test base classes. Also, the classes would run in parallel. Long term perhaps, not needed for this work. -Rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa] negative test generation and testscenarios
Thanks to all who looked at https://review.openstack.org/#/c/64733/. There were a few minor issues I will address but the biggest one was the suggestion to run each variation as a separate test case using testscenarios. After looking into that I see a problem with this use case. Many of these negative tests need to allocate resources such as servers that are then referenced in the test cases. Our tests currently do stuff like that in setUpClass. But testscenarios, unless I misunderstand the code, replicates the whole class which means the resource setup would also be duplicated for each test. I think we need to share allocated resources across the potentially large number of negative variations, and see two options for that: 1. Some magic similar to testscenarios but that creates test methods with varying parameters instead of test classes with varying attributes. Leave allocation in setUpClass. 2. Using testscenarios as-is but move the allocation/cleanup to the module level (fixtures?) This would be a little complicated because a bunch of the setup stuff is defined as methods on the test base classes. Also, the classes would run in parallel. Thoughts or opinions? -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] negative test generation and testscenarios
On 7 January 2014 07:41, David Kranz dkr...@redhat.com wrote: Thanks to all who looked at https://review.openstack.org/#/c/64733/. There were a few minor issues I will address but the biggest one was the suggestion to run each variation as a separate test case using testscenarios. After looking into that I see a problem with this use case. Many of these negative tests need to allocate resources such as servers that are then referenced in the test cases. Our tests currently do stuff like that in setUpClass. But testscenarios, unless I misunderstand the code, replicates the whole class which means the resource setup would also be duplicated for each test. I think we need to share allocated resources across the potentially large number of negative variations, and see two options for that: You don't understand the code :). testscenarios doesn't affect the class of tests. As long as the resulting suite is wrapped in a setUpClass compatible TestSuite object, setUpClass should keep working. 1. Some magic similar to testscenarios but that creates test methods with varying parameters instead of test classes with varying attributes. Leave allocation in setUpClass. -1 No different to testscenarios, just different to be different. 2. Using testscenarios as-is but move the allocation/cleanup to the module level (fixtures?) This would be a little complicated because a bunch of the setup stuff is defined as methods on the test base classes. Also, the classes would run in parallel. Long term perhaps, not needed for this work. -Rob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [qa] Negative test generation
I took a little time to think some more about this and pushed a little working prototype of some ideas I had https://review.openstack.org/#/c/64733/1. Comments about the approach are most welcome. Note that I minimized any refactoring of the existing hierarchy for this prorotype. -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
