Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-20 Thread Kairat Kushaev
Also curious about this. It seems weird to separate the 'positive' and the 'negative' ones, assuming those patches are mostly contributed by the same group of developers. Yeah, agree. This approach leads to situation when I need to look at two places to observe test coverage for each component.

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-20 Thread GHANSHYAM MANN
On Thu, Mar 17, 2016 at 3:24 PM, Kairat Kushaev wrote: > > Also curious about this. It seems weird to separate the 'positive' and > the 'negative' ones, assuming those patches are mostly contributed by > the same group of developers. > > > Yeah, agree. This approach leads

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Assaf Muller
On Wed, Mar 16, 2016 at 10:41 PM, Jim Rollenhagen wrote: > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote: >> Hi >> >> I have one proposal[1] related to negative tests in Tempest, and >> hoping opinions before doing that. >> >> Now Tempest contains

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Ken'ichi Ohmichi
2016-03-16 20:27 GMT-07:00 Assaf Muller : > On Wed, Mar 16, 2016 at 10:41 PM, Jim Rollenhagen > wrote: >> On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote: >>> Hi >>> >>> I have one proposal[1] related to negative tests in Tempest, and

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Andrea Frittoli
On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi wrote: > 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen : > > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote: > >> Hi > >> > >> I have one proposal[1] related to negative tests in

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Rodrigo Duarte
Totally agree here, also, having positive/negative API tests in Tempest helps in the API stability effort. Although the API is owned by the service in question, it interacts with other services and making sure the API is stable is valuable for the communication between them. We know a recent

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Jordan Pittier
Hi, On Thu, Mar 17, 2016 at 2:20 AM, Ken'ichi Ohmichi wrote: > Hi > > I have one proposal[1] related to negative tests in Tempest, and > hoping opinions before doing that. > > Now Tempest contains negative tests and sometimes patches are being > posted for adding more

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Ken'ichi Ohmichi
2016-03-17 5:32 GMT-07:00 Adam Young : > On 03/16/2016 11:01 PM, Ken'ichi Ohmichi wrote: >> >> 2016-03-16 19:29 GMT-07:00 Adam Young : >>> >>> On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote: Hi I have one proposal[1] related to negative

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread GHANSHYAM MANN
On Fri, Mar 18, 2016 at 9:06 AM, Ken'ichi Ohmichi wrote: > 2016-03-17 4:05 GMT-07:00 Andrea Frittoli : >> On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi >> wrote: >>> >>> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Daniel Mellado
El 17/03/16 a las 04:27, Assaf Muller escribió: > On Wed, Mar 16, 2016 at 10:41 PM, Jim Rollenhagen > wrote: >> On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote: >>> Hi >>> >>> I have one proposal[1] related to negative tests in Tempest, and >>> hoping

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Ken'ichi Ohmichi
2016-03-17 3:22 GMT-07:00 Jordan Pittier : >> >> If wanting to add negative tests, it is a nice option to implement >> these tests on each component repo with Tempest plugin interface. We >> can avoid operating negative tests on different component gates and >> each

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Ken'ichi Ohmichi
2016-03-17 15:05 GMT-07:00 Rochelle Grober : > (Sorry for the top post. It was this or bottom post because of company > choice of email systems) (no problem :) > Integration tests, corner cases, negative tests. Lots of names that don't > have clear definitions in

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Adam Young
On 03/16/2016 11:01 PM, Ken'ichi Ohmichi wrote: 2016-03-16 19:29 GMT-07:00 Adam Young : On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote: Hi I have one proposal[1] related to negative tests in Tempest, and hoping opinions before doing that. Now Tempest contains negative

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Masayuki Igawa
From: GHANSHYAM MANN <ghanshyamm...@gmail.com> Subject: Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest Date: Fri, 18 Mar 2016 10:05:39 +0900 > On Fri, Mar 18, 2016 at 9:06 AM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com> > wrote: >> 2016-03-

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Jim Rollenhagen
On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote: > Hi > > I have one proposal[1] related to negative tests in Tempest, and > hoping opinions before doing that. > > Now Tempest contains negative tests and sometimes patches are being > posted for adding more negative tests, but

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Ken'ichi Ohmichi
2016-03-17 4:05 GMT-07:00 Andrea Frittoli : > On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi > wrote: >> >> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen : >> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Rochelle Grober
(Sorry for the top post. It was this or bottom post because of company choice of email systems) Integration tests, corner cases, negative tests. Lots of names that don't have clear definitions in this discussion. But, it seems like the collection of "negative tests" include both functional

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Ken'ichi Ohmichi
2016-03-18 1:50 GMT-07:00 Masayuki Igawa <masayuki.ig...@gmail.com>: > From: GHANSHYAM MANN <ghanshyamm...@gmail.com> > Subject: Re: [openstack-dev] [QA][all] Propose to remove negative tests from > Tempest > Date: Fri, 18 Mar 2016 10:05:39 +0900 > >> On Fri,

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Adam Young
On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote: Hi I have one proposal[1] related to negative tests in Tempest, and hoping opinions before doing that. Now Tempest contains negative tests and sometimes patches are being posted for adding more negative tests, but I'd like to propose removing

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Ken'ichi Ohmichi
2016-03-16 19:29 GMT-07:00 Adam Young : > On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote: >> >> Hi >> >> I have one proposal[1] related to negative tests in Tempest, and >> hoping opinions before doing that. >> >> Now Tempest contains negative tests and sometimes patches are

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Attila Fazekas
Ohmichi" <ken1ohmi...@gmail.com> > To: "OpenStack Development Mailing List" <openstack-dev@lists.openstack.org> > Sent: Thursday, March 17, 2016 2:20:11 AM > Subject: [openstack-dev] [QA][all] Propose to remove negative tests from > Tempest > > Hi

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Ken'ichi Ohmichi
2016-03-16 19:41 GMT-07:00 Jim Rollenhagen : > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote: >> Hi >> >> I have one proposal[1] related to negative tests in Tempest, and >> hoping opinions before doing that. >> >> Now Tempest contains negative tests and

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Jim Rollenhagen
On Wed, Mar 16, 2016 at 10:29:48PM -0400, Adam Young wrote: > On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote: > >Hi > > > >I have one proposal[1] related to negative tests in Tempest, and > >hoping opinions before doing that. > > > >Now Tempest contains negative tests and sometimes patches are

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-19 Thread Qiming Teng
> > I'd love to see this idea explored further. What happens if Tempest > ends up without tests, as a library for shared code as well as a > centralized place to run tests from via plugins? > Also curious about this. It seems weird to separate the 'positive' and the 'negative' ones, assuming

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-18 Thread Daryl Walleck
<ken1ohmi...@gmail.com> Sent: Wednesday, March 16, 2016 8:20 PM To: OpenStack Development Mailing List Subject: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest Hi I have one proposal[1] related to negative tests in Tempest, and hoping opinions before doing th

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-18 Thread Valeriy Ponomaryov
Main reason to split negative tests out, that I see, is attempt to remove "corner, not integrational" tests out of Tempest tree. If so, then "negative" tests is not proper choice, I think. Tempest has lots of positive "corner" test cases. So, if move something, then we need to move exactly

[openstack-dev] [QA][all] Propose to remove negative tests from Tempest

2016-03-18 Thread Ken'ichi Ohmichi
Hi I have one proposal[1] related to negative tests in Tempest, and hoping opinions before doing that. Now Tempest contains negative tests and sometimes patches are being posted for adding more negative tests, but I'd like to propose removing them from Tempest instead. Negative tests verify