I do think negative tests should live on, most likely in their own tempest-plugin based repository. However, I'm not sure that I agree with the reason being because they are functional tests rather than integration. If you look at the current tests, one could argue that many non-Nova tests are actually functional tests. The Nova tests have the luck of requiring other services to function at all, but volumes, identity, and other services can be and are being tested in isolation in at least some Tempest tests. If removing functional tests from Tempest is problem that is trying to be solved, then there's a scope much larger than negative tests to address.
I'm also concerned about the idea of moving these tests back to their individual project repos. To run the same tests that I do today, I would need to install each individual project as well as Tempest to get the same coverage that I get today. That feels like a quite heavy burden just to be able to keep the same coverage as there is now. This would also mean that Tempest should likely be gated on the tests that exist in the project repositories, which seems like would add a great deal of complexity and maintenance. While I understand that some of these tests may be a duplicate effort in the context of the gate, they are needed coverage when testing a deployed OpenStack environment. Rather than sending the negative tests and others scattered back to the individual projects, would it make sense to have a separate repository to hold "extended" Tempest tests? This would allow negative tests to stay in a centralized location, as well as provide a home for tests that might not be desirable for the gate (e.x. longer scenarios, the authorization tests, etc). What I would like to avoid is the pruning of tests that are meaningful to simply reduce execution time. If what is desired is a better definition between what is Tempest and what is not, then I think that is a more important conversation to have in depth. Daryl ________________________________________ From: Ken'ichi Ohmichi <[email protected]> Sent: Wednesday, March 16, 2016 8:20 PM To: OpenStack Development Mailing List Subject: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest Hi I have one proposal[1] related to negative tests in Tempest, and hoping opinions before doing that. Now Tempest contains negative tests and sometimes patches are being posted for adding more negative tests, but I'd like to propose removing them from Tempest instead. Negative tests verify surfaces of REST APIs for each component without any integrations between components. That doesn't seem integration tests which are scope of Tempest. In addition, we need to spend the test operating time on different component's gate if adding negative tests into Tempest. For example, we are operating negative tests of Keystone and more components on the gate of Nova. That is meaningless, so we need to avoid more negative tests into Tempest now. If wanting to add negative tests, it is a nice option to implement these tests on each component repo with Tempest plugin interface. We can avoid operating negative tests on different component gates and each component team can decide what negative tests are valuable on the gate. In long term, all negative tests will be migrated into each component repo with Tempest plugin interface. We will be able to operate valuable negative tests only on each gate. Any thoughts? Thanks Ken Ohmichi --- [1]: https://review.openstack.org/#/c/293197/ __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
