Re: [openstack-dev] [QA] How will Tempest discover/run tests migrated to specific projects?
Rohan, In Rally we are going to automate this work extending rally verify command. You can read here in more details: http://boris-42.me/rally-verfiy-as-the-control-plane-for-gabbi-tempest-in-tree-functional-tests/ As well, you are welcome to take a part in spec discussion here: https://review.openstack.org/#/c/166487/ Best regards, Boris Pavlovic On Mon, Mar 30, 2015 at 9:51 AM, Rohan Kanade openst...@rohankanade.com wrote: Since tests can now be removed from Tempest https://wiki.openstack.org/wiki/QA/Tempest-test-removal and migrated to their specific projects. Does Tempest plan to discover/run these tests in tempest gates? If yes, how is that going to be done? Will there be a discovery mechanism in Tempest to discover tests from individual projects? Regards, Rohan Kanade __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA] How will Tempest discover/run tests migrated to specific projects?
On Mon, Mar 30, 2015 at 12:21:18PM +0530, Rohan Kanade wrote: Since tests can now be removed from Tempest https://wiki.openstack.org/wiki/QA/Tempest-test-removal and migrated to their specific projects. Does Tempest plan to discover/run these tests in tempest gates? If yes, how is that going to be done? Will there be a discovery mechanism in Tempest to discover tests from individual projects? No, the idea behind that wiki page is to outline the procedure for finding something that is out of scope and doesn't belong in tempest and is also safe to remove from the tempest jobs. The point of going through that entire procedure is that the test being removed should not be run in the tempest gates anymore and will become the domain of the other project. Also, IMO the moved test ideally won't be in the same pattern of a tempest test or have the same constraints of a tempest test and would ideally be more coupled to the project under test's internals. So that wouldn't be appropriate to include in a tempest run either. For example, the first test we removed with that procedure was: https://review.openstack.org/#/c/158852/ which removed the flavor negative tests from tempest. These were just testing operations that would go no deeper than Nova's DB layer. Which was something we couldn't verify in tempest. They also didn't really belong in tempest because they were just implicitly verifying Nova's DB layer through API responses. The replacement tests: http://git.openstack.org/cgit/openstack/nova/tree/nova/tests/functional/wsgi/test_flavor_manage.py were able to verify the state of the DB was correct and ensure the correct behavior both in the api and nova's internals. This kind of testing is something which doesn't belong in tempest or any other external test suite. It is also what I feel we should be targeting for with project specific in-tree functional testing and the kind of thing we should be using the removal process on that wiki page for. -Matt Treinish pgpHrbiGjLdjV.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
