Hi, On Thu, Mar 17, 2016 at 2:20 AM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com> wrote:
> Hi > > I have one proposal[1] related to negative tests in Tempest, and > hoping opinions before doing that. > > Now Tempest contains negative tests and sometimes patches are being > posted for adding more negative tests, but I'd like to propose > removing them from Tempest instead. > > Negative tests verify surfaces of REST APIs for each component without > any integrations between components. That doesn't seem integration > tests which are scope of Tempest. > Tempest is not only about integration tests. I mean, we have hundreds of tests that are not integration tests. > In addition, we need to spend the test operating time on different > component's gate if adding negative tests into Tempest. For example, > we are operating negative tests of Keystone and more > components on the gate of Nova. That is meaningless, so we need to > avoid more negative tests into Tempest now. > You have a good point here. But this problem (running tests for project X on project Y's gate) should be addressed more generally not only for negative tests. > > If wanting to add negative tests, it is a nice option to implement > these tests on each component repo with Tempest plugin interface. We > can avoid operating negative tests on different component gates and > each component team can decide what negative tests are valuable on the > gate. > > In long term, all negative tests will be migrated into each component > repo with Tempest plugin interface. We will be able to operate > valuable negative tests only on each gate. > > Any thoughts? > I am not sure we should remove negative tests from Tempest. Agreed that we should reject most new negative tests, but some negative tests do test useful things imo. Also I ran all the negative tests today: "Ran: 452 tests in 144.0000 sec." They just account for 2 minutes and 20sec in the gate. That's very little, removing them won't bring a lot. And the code for negative tests is quite contain, not a big maintenance burden. Jordan
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev