On Mon, Mar 30, 2015 at 12:21:18PM +0530, Rohan Kanade wrote: > Since tests can now be removed from Tempest < > https://wiki.openstack.org/wiki/QA/Tempest-test-removal> and migrated to > their specific projects. > > Does Tempest plan to discover/run these tests in tempest gates? If yes, how > is that going to be done? Will there be a discovery mechanism in Tempest > to discover tests from individual projects? >
No, the idea behind that wiki page is to outline the procedure for finding something that is out of scope and doesn't belong in tempest and is also safe to remove from the tempest jobs. The point of going through that entire procedure is that the test being removed should not be run in the tempest gates anymore and will become the domain of the other project. Also, IMO the moved test ideally won't be in the same pattern of a tempest test or have the same constraints of a tempest test and would ideally be more coupled to the project under test's internals. So that wouldn't be appropriate to include in a tempest run either. For example, the first test we removed with that procedure was: https://review.openstack.org/#/c/158852/ which removed the flavor negative tests from tempest. These were just testing operations that would go no deeper than Nova's DB layer. Which was something we couldn't verify in tempest. They also didn't really belong in tempest because they were just implicitly verifying Nova's DB layer through API responses. The replacement tests: http://git.openstack.org/cgit/openstack/nova/tree/nova/tests/functional/wsgi/test_flavor_manage.py were able to verify the state of the DB was correct and ensure the correct behavior both in the api and nova's internals. This kind of testing is something which doesn't belong in tempest or any other external test suite. It is also what I feel we should be targeting for with project specific in-tree functional testing and the kind of thing we should be using the removal process on that wiki page for. -Matt Treinish
pgpHrbiGjLdjV.pgp
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev