Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Kairat Kushaev]

Also curious about this. It seems weird to separate the 'positive' and
the 'negative' ones, assuming those patches are mostly contributed by
the same group of developers.


Yeah, agree. This approach leads to situation when I need to look at two
places to observe test coverage for each component.
Also when I would like to add some tests I need to contribute to two places
which is not convenient for reviewers for contributors IMO.


Best regards,
Kairat Kushaev

On Thu, Mar 17, 2016 at 8:31 AM, Qiming Teng 
wrote:

> >
> > I'd love to see this idea explored further. What happens if Tempest
> > ends up without tests, as a library for shared code as well as a
> > centralized place to run tests from via plugins?
> >
>
> Also curious about this. It seems weird to separate the 'positive' and
> the 'negative' ones, assuming those patches are mostly contributed by
> the same group of developers.
>
> Qiming
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[GHANSHYAM MANN]

On Thu, Mar 17, 2016 at 3:24 PM, Kairat Kushaev  wrote:
> 
> Also curious about this. It seems weird to separate the 'positive' and
> the 'negative' ones, assuming those patches are mostly contributed by
> the same group of developers.
> 
>
> Yeah, agree. This approach leads to situation when I need to look at two
> places to observe test coverage for each component.
> Also when I would like to add some tests I need to contribute to two places
> which is not convenient for reviewers for contributors IMO.

But currently also it is same situation where some negative tests are
in tempest and some are in project repo either as functional tests or
unit  [1].
Whenever we review new negative test patch, we have to check whether
that already exist in respective project or not if not then we can
consider it to add in tempest.

This could have valid if projects does not have any negative testing
in their tree and all reply on tempest tests for that.

Note- I am considering the surface level negative tests here, but yea
if that negative tests covers larger scope than just negative input
testing then, yes it should be in one place always.

[1]: 
https://github.com/openstack/keystone/blob/master/keystone/tests/unit/test_v3_identity.py

>
>
> Best regards,
> Kairat Kushaev
>
> On Thu, Mar 17, 2016 at 8:31 AM, Qiming Teng 
> wrote:
>>
>> >
>> > I'd love to see this idea explored further. What happens if Tempest
>> > ends up without tests, as a library for shared code as well as a
>> > centralized place to run tests from via plugins?
>> >
>>
>> Also curious about this. It seems weird to separate the 'positive' and
>> the 'negative' ones, assuming those patches are mostly contributed by
>> the same group of developers.
>>
>> Qiming
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Assaf Muller]

On Wed, Mar 16, 2016 at 10:41 PM, Jim Rollenhagen
 wrote:
> On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>> Hi
>>
>> I have one proposal[1] related to negative tests in Tempest, and
>> hoping opinions before doing that.
>>
>> Now Tempest contains negative tests and sometimes patches are being
>> posted for adding more negative tests, but I'd like to propose
>> removing them from Tempest instead.
>>
>> Negative tests verify surfaces of REST APIs for each component without
>> any integrations between components. That doesn't seem integration
>> tests which are scope of Tempest.
>> In addition, we need to spend the test operating time on different
>> component's gate if adding negative tests into Tempest. For example,
>> we are operating negative tests of Keystone and more
>> components on the gate of Nova. That is meaningless, so we need to
>> avoid more negative tests into Tempest now.
>>
>> If wanting to add negative tests, it is a nice option to implement
>> these tests on each component repo with Tempest plugin interface. We
>> can avoid operating negative tests on different component gates and
>> each component team can decide what negative tests are valuable on the
>> gate.
>>
>> In long term, all negative tests will be migrated into each component
>> repo with Tempest plugin interface. We will be able to operate
>> valuable negative tests only on each gate.
>
> So, positive tests in tempest, negative tests as a plugin.
>
> Is there any longer term goal to have all tests for all projects in a
> plugin for that project? Seems odd to separate them.

I'd love to see this idea explored further. What happens if Tempest
ends up without tests, as a library for shared code as well as a
centralized place to run tests from via plugins?

>
> // jim
>
>>
>> Any thoughts?
>>
>> Thanks
>> Ken Ohmichi
>>
>> ---
>> [1]: https://review.openstack.org/#/c/293197/
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Ken'ichi Ohmichi]

2016-03-16 20:27 GMT-07:00 Assaf Muller :
> On Wed, Mar 16, 2016 at 10:41 PM, Jim Rollenhagen
>  wrote:
>> On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>>> Hi
>>>
>>> I have one proposal[1] related to negative tests in Tempest, and
>>> hoping opinions before doing that.
>>>
>>> Now Tempest contains negative tests and sometimes patches are being
>>> posted for adding more negative tests, but I'd like to propose
>>> removing them from Tempest instead.
>>>
>>> Negative tests verify surfaces of REST APIs for each component without
>>> any integrations between components. That doesn't seem integration
>>> tests which are scope of Tempest.
>>> In addition, we need to spend the test operating time on different
>>> component's gate if adding negative tests into Tempest. For example,
>>> we are operating negative tests of Keystone and more
>>> components on the gate of Nova. That is meaningless, so we need to
>>> avoid more negative tests into Tempest now.
>>>
>>> If wanting to add negative tests, it is a nice option to implement
>>> these tests on each component repo with Tempest plugin interface. We
>>> can avoid operating negative tests on different component gates and
>>> each component team can decide what negative tests are valuable on the
>>> gate.
>>>
>>> In long term, all negative tests will be migrated into each component
>>> repo with Tempest plugin interface. We will be able to operate
>>> valuable negative tests only on each gate.
>>
>> So, positive tests in tempest, negative tests as a plugin.
>>
>> Is there any longer term goal to have all tests for all projects in a
>> plugin for that project? Seems odd to separate them.
>
> I'd love to see this idea explored further. What happens if Tempest
> ends up without tests, as a library for shared code as well as a
> centralized place to run tests from via plugins?

Now Tempest contains library code and the other projects can use them
as library.
We are trying to increase the library code for more usability.
The qa-spac https://review.openstack.org/#/c/275966/ is nice to be understood.

Thanks
Ken Ohmichi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Andrea Frittoli]

On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi 
wrote:

> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen :
> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
> >> Hi
> >>
> >> I have one proposal[1] related to negative tests in Tempest, and
> >> hoping opinions before doing that.
> >>
> >> Now Tempest contains negative tests and sometimes patches are being
> >> posted for adding more negative tests, but I'd like to propose
> >> removing them from Tempest instead.
> >>
> >> Negative tests verify surfaces of REST APIs for each component without
> >> any integrations between components. That doesn't seem integration
> >> tests which are scope of Tempest.
> >> In addition, we need to spend the test operating time on different
> >> component's gate if adding negative tests into Tempest. For example,
> >> we are operating negative tests of Keystone and more
> >> components on the gate of Nova. That is meaningless, so we need to
> >> avoid more negative tests into Tempest now.
> >>
> >> If wanting to add negative tests, it is a nice option to implement
> >> these tests on each component repo with Tempest plugin interface. We
> >> can avoid operating negative tests on different component gates and
> >> each component team can decide what negative tests are valuable on the
> >> gate.
> >>
> >> In long term, all negative tests will be migrated into each component
> >> repo with Tempest plugin interface. We will be able to operate
> >> valuable negative tests only on each gate.
> >
> > So, positive tests in tempest, negative tests as a plugin.
> >
> > Is there any longer term goal to have all tests for all projects in a
> > plugin for that project? Seems odd to separate them.
>
> Yeah, from implementation viewpoint, that seems a little odd.
> but from the main scope of Tempest and to avoid unnecessary gate
> operation time, that can be acceptable, I feel.
> Negative tests can be corner cases in most cases, they don't seem
> integration tests.
>

I think it's difficult to define a single black and white criteria for
negative tests, as they encompass a wide range of types of tests.

I agree that things that only testing the API level of a service (not even
a DB behind) do not necessarily belong in tempest - i.e. testing of input
validation done by an API.  We could have a guideline for such tests to be
implemented as unit/functional tests in tree of the service.

However Tempest is also interoperability, so we should keep at least a few
negative API checks in tempest (for the core six services) to enforce that
return codes do not change inadvertently in negative cases, which could
break existing clients and applications.

If a project was to move all negative tests out of tempest, than they might
consider have hacking rules to prevent modifying the code and tests at the
same time, and change behaviour inadvertently.

andrea


>
> Thanks
> Ken Ohmichi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Rodrigo Duarte]

Totally agree here, also, having positive/negative API tests in Tempest
helps in the API stability effort. Although the API is owned by the service
in question, it interacts with other services and making sure the API is
stable is valuable for the communication between them.

We know a recent example where a change in keystone API caused a change in
Cinder.

On Thu, Mar 17, 2016 at 8:05 AM, Andrea Frittoli 
wrote:

>
>
> On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi 
> wrote:
>
>> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen :
>> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>> >> Hi
>> >>
>> >> I have one proposal[1] related to negative tests in Tempest, and
>> >> hoping opinions before doing that.
>> >>
>> >> Now Tempest contains negative tests and sometimes patches are being
>> >> posted for adding more negative tests, but I'd like to propose
>> >> removing them from Tempest instead.
>> >>
>> >> Negative tests verify surfaces of REST APIs for each component without
>> >> any integrations between components. That doesn't seem integration
>> >> tests which are scope of Tempest.
>> >> In addition, we need to spend the test operating time on different
>> >> component's gate if adding negative tests into Tempest. For example,
>> >> we are operating negative tests of Keystone and more
>> >> components on the gate of Nova. That is meaningless, so we need to
>> >> avoid more negative tests into Tempest now.
>> >>
>> >> If wanting to add negative tests, it is a nice option to implement
>> >> these tests on each component repo with Tempest plugin interface. We
>> >> can avoid operating negative tests on different component gates and
>> >> each component team can decide what negative tests are valuable on the
>> >> gate.
>> >>
>> >> In long term, all negative tests will be migrated into each component
>> >> repo with Tempest plugin interface. We will be able to operate
>> >> valuable negative tests only on each gate.
>> >
>> > So, positive tests in tempest, negative tests as a plugin.
>> >
>> > Is there any longer term goal to have all tests for all projects in a
>> > plugin for that project? Seems odd to separate them.
>>
>> Yeah, from implementation viewpoint, that seems a little odd.
>> but from the main scope of Tempest and to avoid unnecessary gate
>> operation time, that can be acceptable, I feel.
>> Negative tests can be corner cases in most cases, they don't seem
>> integration tests.
>>
>
> I think it's difficult to define a single black and white criteria for
> negative tests, as they encompass a wide range of types of tests.
>
> I agree that things that only testing the API level of a service (not even
> a DB behind) do not necessarily belong in tempest - i.e. testing of input
> validation done by an API.  We could have a guideline for such tests to be
> implemented as unit/functional tests in tree of the service.
>
> However Tempest is also interoperability, so we should keep at least a few
> negative API checks in tempest (for the core six services) to enforce that
> return codes do not change inadvertently in negative cases, which could
> break existing clients and applications.
>
> If a project was to move all negative tests out of tempest, than they
> might consider have hacking rules to prevent modifying the code and tests
> at the same time, and change behaviour inadvertently.
>
> andrea
>
>
>>
>> Thanks
>> Ken Ohmichi
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Rodrigo Duarte Sousa
Senior Quality Engineer @ Red Hat
MSc in Computer Science
http://rodrigods.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Jordan Pittier]

Hi,

On Thu, Mar 17, 2016 at 2:20 AM, Ken'ichi Ohmichi 
wrote:

> Hi
>
> I have one proposal[1] related to negative tests in Tempest, and
> hoping opinions before doing that.
>
> Now Tempest contains negative tests and sometimes patches are being
> posted for adding more negative tests, but I'd like to propose
> removing them from Tempest instead.
>
> Negative tests verify surfaces of REST APIs for each component without
> any integrations between components. That doesn't seem integration
> tests which are scope of Tempest.
>
Tempest is not only about integration tests. I mean, we have hundreds of
tests that are not integration tests.


> In addition, we need to spend the test operating time on different
> component's gate if adding negative tests into Tempest. For example,
> we are operating negative tests of Keystone and more
> components on the gate of Nova. That is meaningless, so we need to
> avoid more negative tests into Tempest now.
>
You have a good point here. But this problem (running tests for project X
on project Y's gate) should be addressed more generally not only for
negative tests.


>
> If wanting to add negative tests, it is a nice option to implement
> these tests on each component repo with Tempest plugin interface. We
> can avoid operating negative tests on different component gates and
> each component team can decide what negative tests are valuable on the
> gate.
>
> In long term, all negative tests will be migrated into each component
> repo with Tempest plugin interface. We will be able to operate
> valuable negative tests only on each gate.
>
> Any thoughts?
>

I am not sure we should remove negative tests from Tempest. Agreed that we
should reject most new negative tests, but some negative
tests do test useful things imo. Also I ran all the negative tests today:
"Ran: 452 tests in 144. sec." They just account for 2 minutes and 20sec
in the gate. That's very little, removing them won't bring a lot. And the
code for negative tests is quite contain, not a big maintenance burden.

Jordan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Ken'ichi Ohmichi]

2016-03-17 5:32 GMT-07:00 Adam Young :
> On 03/16/2016 11:01 PM, Ken'ichi Ohmichi wrote:
>>
>> 2016-03-16 19:29 GMT-07:00 Adam Young :
>>>
>>> On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote:

 Hi

 I have one proposal[1] related to negative tests in Tempest, and
 hoping opinions before doing that.

 Now Tempest contains negative tests and sometimes patches are being
 posted for adding more negative tests, but I'd like to propose
 removing them from Tempest instead.

 Negative tests verify surfaces of REST APIs for each component without
 any integrations between components. That doesn't seem integration
 tests which are scope of Tempest.
 In addition, we need to spend the test operating time on different
 component's gate if adding negative tests into Tempest. For example,
 we are operating negative tests of Keystone and more
 components on the gate of Nova. That is meaningless, so we need to
 avoid more negative tests into Tempest now.

 If wanting to add negative tests, it is a nice option to implement
 these tests on each component repo with Tempest plugin interface. We
 can avoid operating negative tests on different component gates and
 each component team can decide what negative tests are valuable on the
 gate.
>>>
>>>
>>> Hear hear!  For Keystone, please put them in the Keystone Client
>>> Functional
>>> tests.
>>
>> They are negative tests of Keystone API, not keystoneclient tests.
>> These tests are implemented with Tempest original rest clients without
>> official clients because it is nice to test variable cases in Tempest.
>> So it is nice to keep them in Keystone repo, I feel.
>
>
> That works, too, and there is a Functional test section in Keystone Proper.
>
> Most of the Keystone APIs are fairly well covered by the Unit tests, which
> make a full HTTP call through a liteish server, so the functional tests are
> really to cover the live database (soon to include LDAP, hopefully)
> integrations.

Nice point.
Yeah, most these cases(without DB) are covered by unit tests of each
project, I guess.
At least, Nova also covers these cases with unit tests.

Thanks

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[GHANSHYAM MANN]

On Fri, Mar 18, 2016 at 9:06 AM, Ken'ichi Ohmichi  wrote:
> 2016-03-17 4:05 GMT-07:00 Andrea Frittoli :
>> On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi 
>> wrote:
>>>
>>> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen :
>>> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>>> >> Hi
>>> >>
>>> >> I have one proposal[1] related to negative tests in Tempest, and
>>> >> hoping opinions before doing that.
>>> >>
>>> >> Now Tempest contains negative tests and sometimes patches are being
>>> >> posted for adding more negative tests, but I'd like to propose
>>> >> removing them from Tempest instead.
>>> >>
>>> >> Negative tests verify surfaces of REST APIs for each component without
>>> >> any integrations between components. That doesn't seem integration
>>> >> tests which are scope of Tempest.
>>> >> In addition, we need to spend the test operating time on different
>>> >> component's gate if adding negative tests into Tempest. For example,
>>> >> we are operating negative tests of Keystone and more
>>> >> components on the gate of Nova. That is meaningless, so we need to
>>> >> avoid more negative tests into Tempest now.
>>> >>
>>> >> If wanting to add negative tests, it is a nice option to implement
>>> >> these tests on each component repo with Tempest plugin interface. We
>>> >> can avoid operating negative tests on different component gates and
>>> >> each component team can decide what negative tests are valuable on the
>>> >> gate.
>>> >>
>>> >> In long term, all negative tests will be migrated into each component
>>> >> repo with Tempest plugin interface. We will be able to operate
>>> >> valuable negative tests only on each gate.
>>> >
>>> > So, positive tests in tempest, negative tests as a plugin.
>>> >
>>> > Is there any longer term goal to have all tests for all projects in a
>>> > plugin for that project? Seems odd to separate them.
>>>
>>> Yeah, from implementation viewpoint, that seems a little odd.
>>> but from the main scope of Tempest and to avoid unnecessary gate
>>> operation time, that can be acceptable, I feel.
>>> Negative tests can be corner cases in most cases, they don't seem
>>> integration tests.
>>
>> I think it's difficult to define a single black and white criteria for
>> negative tests, as they encompass a wide range of types of tests.
>>
>> I agree that things that only testing the API level of a service (not even a
>> DB behind) do not necessarily belong in tempest - i.e. testing of input
>> validation done by an API.  We could have a guideline for such tests to be
>> implemented as unit/functional tests in tree of the service.

Yes, this is key point here. If we see ~70% of the negative tests are
just checking API surface level (wrong input validation), which
defiantly
not belong to Tempest scope. Those should be in respective projects
repo either by functional/unit/plugin.
But in that case we have to define a very clear criteria about what
level of negative testing should be in scope of Tempest.

Also another key point is that, as we have lot of surface level
negative testing in Tempest, should we reject the new one?
For me sometime it makes difficult to reject those as we already have
some in tempest.

My vote here is we reject the new surface level negative tests and try
to move all existing negative tests(surface level) out of Tempest
ASAP.
And those can be just moved to projects functional/unit tests.

>
> Yeah, it is difficult to distinguish corner cases or not on negative
> tests as the criteria.
> If necessary to do that, we(QA team) need to read the implementation
> code of the core six services deeply during Tempest reviews. Then I
> rushed to remove all of them. My first proposal is not good according
> to feedback, but I'm happy to get feedback to see our direction :-)
>
> The guideline is a nice idea.
> If necessary to add more negative tests into Tempest, how about
> requiring to write the reason which explains new tests are not corner
> cases in the commit message?
> We can know the merit of new negative ones when reviewing.
>
>> However Tempest is also interoperability, so we should keep at least a few
>> negative API checks in tempest (for the core six services) to enforce that
>> return codes do not change inadvertently in negative cases, which could
>> break existing clients and applications.
>
> This also is a nice point.
> How to change error return codes is unclear to me at this time.
> In Nova, there are some exceptions for changing error return code
> without microversion bumping as [1]. This kind of guideline will be
> discussed later.

This makes Tempest scope little bit unclear again. If we want to
verify all error codes in Tempest then it leads to have all surface
level negative testing also in Tempest. There are lot of scenario
where error codes can be verified and will be difficult to cover all
in Tempest.

Current negative tests 

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Daniel Mellado]

El 17/03/16 a las 04:27, Assaf Muller escribió:
> On Wed, Mar 16, 2016 at 10:41 PM, Jim Rollenhagen
>  wrote:
>> On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>>> Hi
>>>
>>> I have one proposal[1] related to negative tests in Tempest, and
>>> hoping opinions before doing that.
>>>
>>> Now Tempest contains negative tests and sometimes patches are being
>>> posted for adding more negative tests, but I'd like to propose
>>> removing them from Tempest instead.
>>>
>>> Negative tests verify surfaces of REST APIs for each component without
>>> any integrations between components. That doesn't seem integration
>>> tests which are scope of Tempest.
>>> In addition, we need to spend the test operating time on different
>>> component's gate if adding negative tests into Tempest. For example,
>>> we are operating negative tests of Keystone and more
>>> components on the gate of Nova. That is meaningless, so we need to
>>> avoid more negative tests into Tempest now.
>>>
>>> If wanting to add negative tests, it is a nice option to implement
>>> these tests on each component repo with Tempest plugin interface. We
>>> can avoid operating negative tests on different component gates and
>>> each component team can decide what negative tests are valuable on the
>>> gate.
>>>
>>> In long term, all negative tests will be migrated into each component
>>> repo with Tempest plugin interface. We will be able to operate
>>> valuable negative tests only on each gate.
>> So, positive tests in tempest, negative tests as a plugin.
>>
>> Is there any longer term goal to have all tests for all projects in a
>> plugin for that project? Seems odd to separate them.
> I'd love to see this idea explored further. What happens if Tempest
> ends up without tests, as a library for shared code as well as a
> centralized place to run tests from via plugins?
I think this should be further discussed as some tests, such as the
scenario ones, make use of several projects. So for scenario tests at
least I think that we should keep them inside the core tempest repo.
Besides that such change would also affect different projects, such as
Defcore/Refstack, where the plugin usage would make complex to keep a
list of non-tree tests.
>
>> // jim
>>
>>> Any thoughts?
>>>
>>> Thanks
>>> Ken Ohmichi
>>>
>>> ---
>>> [1]: https://review.openstack.org/#/c/293197/
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Ken'ichi Ohmichi]

2016-03-17 3:22 GMT-07:00 Jordan Pittier :
>>
>> If wanting to add negative tests, it is a nice option to implement
>> these tests on each component repo with Tempest plugin interface. We
>> can avoid operating negative tests on different component gates and
>> each component team can decide what negative tests are valuable on the
>> gate.
>>
>> In long term, all negative tests will be migrated into each component
>> repo with Tempest plugin interface. We will be able to operate
>> valuable negative tests only on each gate.
>>
>> Any thoughts?
>
>
> I am not sure we should remove negative tests from Tempest. Agreed that we
> should reject most new negative tests, but some negative
> tests do test useful things imo. Also I ran all the negative tests today:
> "Ran: 452 tests in 144. sec." They just account for 2 minutes and 20sec
> in the gate. That's very little, removing them won't bring a lot. And the
> code for negative tests is quite contain, not a big maintenance burden.

This is a nice point. The merit of removal is less from the operation time.

My concern was the existing negative tests attract new ones like "ah,
current Tempest doesn't cover these (corner) cases. We should add
them".
I agree with keeping the existing negative tests if we reject new
corner negative tests.

Thanks
Ken Ohmichi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Ken'ichi Ohmichi]

2016-03-17 15:05 GMT-07:00 Rochelle Grober :
> (Sorry for the top post.  It was this or bottom post because of company
> choice of email systems)

(no problem :)

> Integration tests, corner cases, negative tests.  Lots of names that don't
> have clear definitions in this discussion.  But, it seems like the
> collection of "negative tests" include both functional and integrated tests.
> Maybe we should analyze the tests to see where they belong. So, perhaps we
> can ask an important question here:
>
> · Does the test provoke a change of behavior outside the designated
> project, depending on the output(s) from the test action?  In other words,
> if the test is attempting to generate a 4xx response, does it cause another
> project or client to act differently than if it returns a 2xx or different
> 4xx response?  An example of this would be the project the response is
> returned to attempts a retry.
>
> This should qualify as a tempest test because it drives actions from
> multiple projects.  Yes, it is a "negative" api test, but run in an
> integrated system, it is also a normal integration test that demonstrates
> the integrated system behaves appropriately on expected error conditions
> from other projects.  Or it throws an exception or an error message and we
> know we have a problem with either the api or the misbehaving project.
>
> Andrea aptly pointed out that these tests may very well be interoperability
> tests that should be in a gate somewhere.  After all, what are APIs, but
> defined interfaces between two components.  Sometimes it's between human and
> software, but most often in OpenStack, it's between two pieces of software
> written by different groups of developers.  Hence, the API *is* the
> integration point and ideally, all request/response options should be
> validated, whether happy path or not so happy.
>
> And, oh by the way, "negative" responses are crucial interoperability points
> for DefCore.

Thanks for your comment from the DefCore viewpoint.
That is very appreciated for me.

I understand the *success* status code should be stable for OpenStack
interoperability, but I don't yet for the *error* status code for
that.

I saw some *error* status code changes with valid implementations.
For example, if a client passes invalid format id as the URI to the
corresponding server, we can imagine two error status codes:
HTTP400(BadRequest) or HTTP404(NotFound).
From the implementation viewpoint, HTTP404 can be acceptable because
such resource must not exist in its database and we can keep the
implementation simple without a format check code. However, sometimes
we want to add the format check to avoid querying its database. In
this case, the error status code is changed from HTTP404 to HTTP400.
Now Tempest is checking such error status codes as negative tests, and
the above changes would be blocked.

Maybe the above cases would be corner, but I'd like to know what is
actual merit of error status code check from the interoperability
thing.

Thanks

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Adam Young]

On 03/16/2016 11:01 PM, Ken'ichi Ohmichi wrote:

2016-03-16 19:29 GMT-07:00 Adam Young :

On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote:

Hi

I have one proposal[1] related to negative tests in Tempest, and
hoping opinions before doing that.

Now Tempest contains negative tests and sometimes patches are being
posted for adding more negative tests, but I'd like to propose
removing them from Tempest instead.

Negative tests verify surfaces of REST APIs for each component without
any integrations between components. That doesn't seem integration
tests which are scope of Tempest.
In addition, we need to spend the test operating time on different
component's gate if adding negative tests into Tempest. For example,
we are operating negative tests of Keystone and more
components on the gate of Nova. That is meaningless, so we need to
avoid more negative tests into Tempest now.

If wanting to add negative tests, it is a nice option to implement
these tests on each component repo with Tempest plugin interface. We
can avoid operating negative tests on different component gates and
each component team can decide what negative tests are valuable on the
gate.


Hear hear!  For Keystone, please put them in the Keystone Client Functional
tests.

They are negative tests of Keystone API, not keystoneclient tests.
These tests are implemented with Tempest original rest clients without
official clients because it is nice to test variable cases in Tempest.
So it is nice to keep them in Keystone repo, I feel.


That works, too, and there is a Functional test section in Keystone Proper.

Most of the Keystone APIs are fairly well covered by the Unit tests, 
which make a full HTTP call through a liteish server, so the functional 
tests are really to cover the live database (soon to include LDAP, 
hopefully) integrations.


Thanks
Ken Ohmichi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Masayuki Igawa]

From: GHANSHYAM MANN <ghanshyamm...@gmail.com>
Subject: Re: [openstack-dev] [QA][all] Propose to remove negative tests from 
Tempest
Date: Fri, 18 Mar 2016 10:05:39 +0900

> On Fri, Mar 18, 2016 at 9:06 AM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com> 
> wrote:
>> 2016-03-17 4:05 GMT-07:00 Andrea Frittoli <andrea.fritt...@gmail.com>:
>>> On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi <ken1ohmi...@gmail.com>
>>> wrote:
>>>>
>>>> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen <j...@jimrollenhagen.com>:
>>>> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>>>> >> Hi
>>>> >>
>>>> >> I have one proposal[1] related to negative tests in Tempest, and
>>>> >> hoping opinions before doing that.
>>>> >>
>>>> >> Now Tempest contains negative tests and sometimes patches are being
>>>> >> posted for adding more negative tests, but I'd like to propose
>>>> >> removing them from Tempest instead.
>>>> >>
>>>> >> Negative tests verify surfaces of REST APIs for each component without
>>>> >> any integrations between components. That doesn't seem integration
>>>> >> tests which are scope of Tempest.
>>>> >> In addition, we need to spend the test operating time on different
>>>> >> component's gate if adding negative tests into Tempest. For example,
>>>> >> we are operating negative tests of Keystone and more
>>>> >> components on the gate of Nova. That is meaningless, so we need to
>>>> >> avoid more negative tests into Tempest now.
>>>> >>
>>>> >> If wanting to add negative tests, it is a nice option to implement
>>>> >> these tests on each component repo with Tempest plugin interface. We
>>>> >> can avoid operating negative tests on different component gates and
>>>> >> each component team can decide what negative tests are valuable on the
>>>> >> gate.
>>>> >>
>>>> >> In long term, all negative tests will be migrated into each component
>>>> >> repo with Tempest plugin interface. We will be able to operate
>>>> >> valuable negative tests only on each gate.
>>>> >
>>>> > So, positive tests in tempest, negative tests as a plugin.
>>>> >
>>>> > Is there any longer term goal to have all tests for all projects in a
>>>> > plugin for that project? Seems odd to separate them.
>>>>
>>>> Yeah, from implementation viewpoint, that seems a little odd.
>>>> but from the main scope of Tempest and to avoid unnecessary gate
>>>> operation time, that can be acceptable, I feel.
>>>> Negative tests can be corner cases in most cases, they don't seem
>>>> integration tests.
>>>
>>> I think it's difficult to define a single black and white criteria for
>>> negative tests, as they encompass a wide range of types of tests.
>>>
>>> I agree that things that only testing the API level of a service (not even a
>>> DB behind) do not necessarily belong in tempest - i.e. testing of input
>>> validation done by an API.  We could have a guideline for such tests to be
>>> implemented as unit/functional tests in tree of the service.
> 
> Yes, this is key point here. If we see ~70% of the negative tests are
> just checking API surface level (wrong input validation), which
> defiantly
> not belong to Tempest scope. Those should be in respective projects
> repo either by functional/unit/plugin.
> But in that case we have to define a very clear criteria about what
> level of negative testing should be in scope of Tempest.
> 
> Also another key point is that, as we have lot of surface level
> negative testing in Tempest, should we reject the new one?
> For me sometime it makes difficult to reject those as we already have
> some in tempest.
> 
> My vote here is we reject the new surface level negative tests and try
> to move all existing negative tests(surface level) out of Tempest
> ASAP.
> And those can be just moved to projects functional/unit tests.
> 
>>
>> Yeah, it is difficult to distinguish corner cases or not on negative
>> tests as the criteria.
>> If necessary to do that, we(QA team) need to read the implementation
>> code of the core six services deeply during Tempest reviews. Then I
>> rushed to remove all of them. My first proposal is not good according
&

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Jim Rollenhagen]

On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
> Hi
> 
> I have one proposal[1] related to negative tests in Tempest, and
> hoping opinions before doing that.
> 
> Now Tempest contains negative tests and sometimes patches are being
> posted for adding more negative tests, but I'd like to propose
> removing them from Tempest instead.
> 
> Negative tests verify surfaces of REST APIs for each component without
> any integrations between components. That doesn't seem integration
> tests which are scope of Tempest.
> In addition, we need to spend the test operating time on different
> component's gate if adding negative tests into Tempest. For example,
> we are operating negative tests of Keystone and more
> components on the gate of Nova. That is meaningless, so we need to
> avoid more negative tests into Tempest now.
> 
> If wanting to add negative tests, it is a nice option to implement
> these tests on each component repo with Tempest plugin interface. We
> can avoid operating negative tests on different component gates and
> each component team can decide what negative tests are valuable on the
> gate.
> 
> In long term, all negative tests will be migrated into each component
> repo with Tempest plugin interface. We will be able to operate
> valuable negative tests only on each gate.

So, positive tests in tempest, negative tests as a plugin.

Is there any longer term goal to have all tests for all projects in a
plugin for that project? Seems odd to separate them.

// jim

> 
> Any thoughts?
> 
> Thanks
> Ken Ohmichi
> 
> ---
> [1]: https://review.openstack.org/#/c/293197/
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Ken'ichi Ohmichi]

2016-03-17 4:05 GMT-07:00 Andrea Frittoli :
> On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi 
> wrote:
>>
>> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen :
>> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>> >> Hi
>> >>
>> >> I have one proposal[1] related to negative tests in Tempest, and
>> >> hoping opinions before doing that.
>> >>
>> >> Now Tempest contains negative tests and sometimes patches are being
>> >> posted for adding more negative tests, but I'd like to propose
>> >> removing them from Tempest instead.
>> >>
>> >> Negative tests verify surfaces of REST APIs for each component without
>> >> any integrations between components. That doesn't seem integration
>> >> tests which are scope of Tempest.
>> >> In addition, we need to spend the test operating time on different
>> >> component's gate if adding negative tests into Tempest. For example,
>> >> we are operating negative tests of Keystone and more
>> >> components on the gate of Nova. That is meaningless, so we need to
>> >> avoid more negative tests into Tempest now.
>> >>
>> >> If wanting to add negative tests, it is a nice option to implement
>> >> these tests on each component repo with Tempest plugin interface. We
>> >> can avoid operating negative tests on different component gates and
>> >> each component team can decide what negative tests are valuable on the
>> >> gate.
>> >>
>> >> In long term, all negative tests will be migrated into each component
>> >> repo with Tempest plugin interface. We will be able to operate
>> >> valuable negative tests only on each gate.
>> >
>> > So, positive tests in tempest, negative tests as a plugin.
>> >
>> > Is there any longer term goal to have all tests for all projects in a
>> > plugin for that project? Seems odd to separate them.
>>
>> Yeah, from implementation viewpoint, that seems a little odd.
>> but from the main scope of Tempest and to avoid unnecessary gate
>> operation time, that can be acceptable, I feel.
>> Negative tests can be corner cases in most cases, they don't seem
>> integration tests.
>
> I think it's difficult to define a single black and white criteria for
> negative tests, as they encompass a wide range of types of tests.
>
> I agree that things that only testing the API level of a service (not even a
> DB behind) do not necessarily belong in tempest - i.e. testing of input
> validation done by an API.  We could have a guideline for such tests to be
> implemented as unit/functional tests in tree of the service.

Yeah, it is difficult to distinguish corner cases or not on negative
tests as the criteria.
If necessary to do that, we(QA team) need to read the implementation
code of the core six services deeply during Tempest reviews. Then I
rushed to remove all of them. My first proposal is not good according
to feedback, but I'm happy to get feedback to see our direction :-)

The guideline is a nice idea.
If necessary to add more negative tests into Tempest, how about
requiring to write the reason which explains new tests are not corner
cases in the commit message?
We can know the merit of new negative ones when reviewing.

> However Tempest is also interoperability, so we should keep at least a few
> negative API checks in tempest (for the core six services) to enforce that
> return codes do not change inadvertently in negative cases, which could
> break existing clients and applications.

This also is a nice point.
How to change error return codes is unclear to me at this time.
In Nova, there are some exceptions for changing error return code
without microversion bumping as [1]. This kind of guideline will be
discussed later.

Now I am fine to keep the existing negative tests in Tempest, anyway.

Thanks
Ken Ohmichi

---
[1]: 
https://github.com/openstack/nova/blob/master/doc/source/api_microversion_dev.rst#f2

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Rochelle Grober]

(Sorry for the top post.  It was this or bottom post because of company choice 
of email systems)

Integration tests, corner cases, negative tests.  Lots of names that don't have 
clear definitions in this discussion.  But, it seems like the collection of 
"negative tests" include both functional and integrated tests.  Maybe we should 
analyze the tests to see where they belong. So, perhaps we can ask an important 
question here:

· Does the test provoke a change of behavior outside the designated 
project, depending on the output(s) from the test action?  In other words, if 
the test is attempting to generate a 4xx response, does it cause another 
project or client to act differently than if it returns a 2xx or different 4xx 
response?  An example of this would be the project the response is returned to 
attempts a retry.
This should qualify as a tempest test because it drives actions from multiple 
projects.  Yes, it is a "negative" api test, but run in an integrated system, 
it is also a normal integration test that demonstrates the integrated system 
behaves appropriately on expected error conditions from other projects.  Or it 
throws an exception or an error message and we know we have a problem with 
either the api or the misbehaving project.

Andrea aptly pointed out that these tests may very well be interoperability 
tests that should be in a gate somewhere.  After all, what are APIs, but 
defined interfaces between two components.  Sometimes it's between human and 
software, but most often in OpenStack, it's between two pieces of software 
written by different groups of developers.  Hence, the API *is* the integration 
point and ideally, all request/response options should be validated, whether 
happy path or not so happy.

And, oh by the way, "negative" responses are crucial interoperability points 
for DefCore.

--Rocky

From: Valeriy Ponomaryov March 16, 2016 11:25 PM
Main reason to split negative tests out, that I see, is attempt to remove 
"corner, not integrational" tests out of Tempest tree. If so, then "negative" 
tests is not proper choice, I think.
Tempest has lots of positive "corner" test cases. So, if move something, then 
we need to move exactly "corner" tests and leave only "integrational" tests of 
any kind - either it is negative or positive.
It is said, that doing so, all single-project related tests will be in its own 
repo as a plugin and only integrational tests will be in Tempest.

Valeriy

On Thu, Mar 17, 2016 at 7:31 AM, Qiming Teng 
> wrote:
>
> I'd love to see this idea explored further. What happens if Tempest
> ends up without tests, as a library for shared code as well as a
> centralized place to run tests from via plugins?
>

Also curious about this. It seems weird to separate the 'positive' and
the 'negative' ones, assuming those patches are mostly contributed by
the same group of developers.

Qiming


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Kind Regards
Valeriy Ponomaryov
www.mirantis.com
vponomar...@mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Ken'ichi Ohmichi]

2016-03-18 1:50 GMT-07:00 Masayuki Igawa <masayuki.ig...@gmail.com>:
> From: GHANSHYAM MANN <ghanshyamm...@gmail.com>
> Subject: Re: [openstack-dev] [QA][all] Propose to remove negative tests from 
> Tempest
> Date: Fri, 18 Mar 2016 10:05:39 +0900
>
>> On Fri, Mar 18, 2016 at 9:06 AM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com> 
>> wrote:
>>> 2016-03-17 4:05 GMT-07:00 Andrea Frittoli <andrea.fritt...@gmail.com>:
>>>> On Thu, Mar 17, 2016 at 2:57 AM Ken'ichi Ohmichi <ken1ohmi...@gmail.com>
>>>> wrote:
>>>>>
>>>>> 2016-03-16 19:41 GMT-07:00 Jim Rollenhagen <j...@jimrollenhagen.com>:
>>>>> > On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>>>>> >> Hi
>>>>> >>
>>>>> >> I have one proposal[1] related to negative tests in Tempest, and
>>>>> >> hoping opinions before doing that.
>>>>> >>
>>>>> >> Now Tempest contains negative tests and sometimes patches are being
>>>>> >> posted for adding more negative tests, but I'd like to propose
>>>>> >> removing them from Tempest instead.
>>>>> >>
>>>>> >> Negative tests verify surfaces of REST APIs for each component without
>>>>> >> any integrations between components. That doesn't seem integration
>>>>> >> tests which are scope of Tempest.
>>>>> >> In addition, we need to spend the test operating time on different
>>>>> >> component's gate if adding negative tests into Tempest. For example,
>>>>> >> we are operating negative tests of Keystone and more
>>>>> >> components on the gate of Nova. That is meaningless, so we need to
>>>>> >> avoid more negative tests into Tempest now.
>>>>> >>
>>>>> >> If wanting to add negative tests, it is a nice option to implement
>>>>> >> these tests on each component repo with Tempest plugin interface. We
>>>>> >> can avoid operating negative tests on different component gates and
>>>>> >> each component team can decide what negative tests are valuable on the
>>>>> >> gate.
>>>>> >>
>>>>> >> In long term, all negative tests will be migrated into each component
>>>>> >> repo with Tempest plugin interface. We will be able to operate
>>>>> >> valuable negative tests only on each gate.
>>>>> >
>>>>> > So, positive tests in tempest, negative tests as a plugin.
>>>>> >
>>>>> > Is there any longer term goal to have all tests for all projects in a
>>>>> > plugin for that project? Seems odd to separate them.
>>>>>
>>>>> Yeah, from implementation viewpoint, that seems a little odd.
>>>>> but from the main scope of Tempest and to avoid unnecessary gate
>>>>> operation time, that can be acceptable, I feel.
>>>>> Negative tests can be corner cases in most cases, they don't seem
>>>>> integration tests.
>>>>
>>>> I think it's difficult to define a single black and white criteria for
>>>> negative tests, as they encompass a wide range of types of tests.
>>>>
>>>> I agree that things that only testing the API level of a service (not even 
>>>> a
>>>> DB behind) do not necessarily belong in tempest - i.e. testing of input
>>>> validation done by an API.  We could have a guideline for such tests to be
>>>> implemented as unit/functional tests in tree of the service.
>>
>> Yes, this is key point here. If we see ~70% of the negative tests are
>> just checking API surface level (wrong input validation), which
>> defiantly
>> not belong to Tempest scope. Those should be in respective projects
>> repo either by functional/unit/plugin.
>> But in that case we have to define a very clear criteria about what
>> level of negative testing should be in scope of Tempest.
>>
>> Also another key point is that, as we have lot of surface level
>> negative testing in Tempest, should we reject the new one?
>> For me sometime it makes difficult to reject those as we already have
>> some in tempest.
>>
>> My vote here is we reject the new surface level negative tests and try
>> to move all existing negative tests(surface level) out of Tempest
>> ASAP.
>> And those can be j

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Adam Young]

On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote:

Hi

I have one proposal[1] related to negative tests in Tempest, and
hoping opinions before doing that.

Now Tempest contains negative tests and sometimes patches are being
posted for adding more negative tests, but I'd like to propose
removing them from Tempest instead.

Negative tests verify surfaces of REST APIs for each component without
any integrations between components. That doesn't seem integration
tests which are scope of Tempest.
In addition, we need to spend the test operating time on different
component's gate if adding negative tests into Tempest. For example,
we are operating negative tests of Keystone and more
components on the gate of Nova. That is meaningless, so we need to
avoid more negative tests into Tempest now.

If wanting to add negative tests, it is a nice option to implement
these tests on each component repo with Tempest plugin interface. We
can avoid operating negative tests on different component gates and
each component team can decide what negative tests are valuable on the
gate.


Hear hear!  For Keystone, please put them in the Keystone Client 
Functional tests.




In long term, all negative tests will be migrated into each component
repo with Tempest plugin interface. We will be able to operate
valuable negative tests only on each gate.

Any thoughts?

Thanks
Ken Ohmichi

---
[1]: https://review.openstack.org/#/c/293197/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Ken'ichi Ohmichi]

2016-03-16 19:29 GMT-07:00 Adam Young :
> On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote:
>>
>> Hi
>>
>> I have one proposal[1] related to negative tests in Tempest, and
>> hoping opinions before doing that.
>>
>> Now Tempest contains negative tests and sometimes patches are being
>> posted for adding more negative tests, but I'd like to propose
>> removing them from Tempest instead.
>>
>> Negative tests verify surfaces of REST APIs for each component without
>> any integrations between components. That doesn't seem integration
>> tests which are scope of Tempest.
>> In addition, we need to spend the test operating time on different
>> component's gate if adding negative tests into Tempest. For example,
>> we are operating negative tests of Keystone and more
>> components on the gate of Nova. That is meaningless, so we need to
>> avoid more negative tests into Tempest now.
>>
>> If wanting to add negative tests, it is a nice option to implement
>> these tests on each component repo with Tempest plugin interface. We
>> can avoid operating negative tests on different component gates and
>> each component team can decide what negative tests are valuable on the
>> gate.
>
>
> Hear hear!  For Keystone, please put them in the Keystone Client Functional
> tests.

They are negative tests of Keystone API, not keystoneclient tests.
These tests are implemented with Tempest original rest clients without
official clients because it is nice to test variable cases in Tempest.
So it is nice to keep them in Keystone repo, I feel.

Thanks
Ken Ohmichi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Attila Fazekas]

Most negative test supposed to be very simple and we should not spend
too much time in them.

The right question:
Are we able to run 100 negative test/sec ?
Where is the time spent ?

If we are able to solve the main issue,
probably we do not need to worry about how many negative test we have.

Not all negative test a simple dumb thing.
If we would do smarter test selection, likely
we would need to keep only the slower ones.
So at the and almost nothing gained.


BTW, we can increase the number of tempest workers.

Best Regards,
Attila

- Original Message -
> From: "Ken'ichi Ohmichi" <ken1ohmi...@gmail.com>
> To: "OpenStack Development Mailing List" <openstack-dev@lists.openstack.org>
> Sent: Thursday, March 17, 2016 2:20:11 AM
> Subject: [openstack-dev] [QA][all] Propose to remove negative tests from  
> Tempest
> 
> Hi
> 
> I have one proposal[1] related to negative tests in Tempest, and
> hoping opinions before doing that.
> 
> Now Tempest contains negative tests and sometimes patches are being
> posted for adding more negative tests, but I'd like to propose
> removing them from Tempest instead.
> 
> Negative tests verify surfaces of REST APIs for each component without
> any integrations between components. That doesn't seem integration
> tests which are scope of Tempest.
> In addition, we need to spend the test operating time on different
> component's gate if adding negative tests into Tempest. For example,
> we are operating negative tests of Keystone and more
> components on the gate of Nova. That is meaningless, so we need to
> avoid more negative tests into Tempest now.
> 
> If wanting to add negative tests, it is a nice option to implement
> these tests on each component repo with Tempest plugin interface. We
> can avoid operating negative tests on different component gates and
> each component team can decide what negative tests are valuable on the
> gate.
> 
> In long term, all negative tests will be migrated into each component
> repo with Tempest plugin interface. We will be able to operate
> valuable negative tests only on each gate.
> 
> Any thoughts?
> 
> Thanks
> Ken Ohmichi
> 
> ---
> [1]: https://review.openstack.org/#/c/293197/
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Ken'ichi Ohmichi]

2016-03-16 19:41 GMT-07:00 Jim Rollenhagen :
> On Wed, Mar 16, 2016 at 06:20:11PM -0700, Ken'ichi Ohmichi wrote:
>> Hi
>>
>> I have one proposal[1] related to negative tests in Tempest, and
>> hoping opinions before doing that.
>>
>> Now Tempest contains negative tests and sometimes patches are being
>> posted for adding more negative tests, but I'd like to propose
>> removing them from Tempest instead.
>>
>> Negative tests verify surfaces of REST APIs for each component without
>> any integrations between components. That doesn't seem integration
>> tests which are scope of Tempest.
>> In addition, we need to spend the test operating time on different
>> component's gate if adding negative tests into Tempest. For example,
>> we are operating negative tests of Keystone and more
>> components on the gate of Nova. That is meaningless, so we need to
>> avoid more negative tests into Tempest now.
>>
>> If wanting to add negative tests, it is a nice option to implement
>> these tests on each component repo with Tempest plugin interface. We
>> can avoid operating negative tests on different component gates and
>> each component team can decide what negative tests are valuable on the
>> gate.
>>
>> In long term, all negative tests will be migrated into each component
>> repo with Tempest plugin interface. We will be able to operate
>> valuable negative tests only on each gate.
>
> So, positive tests in tempest, negative tests as a plugin.
>
> Is there any longer term goal to have all tests for all projects in a
> plugin for that project? Seems odd to separate them.

Yeah, from implementation viewpoint, that seems a little odd.
but from the main scope of Tempest and to avoid unnecessary gate
operation time, that can be acceptable, I feel.
Negative tests can be corner cases in most cases, they don't seem
integration tests.

Thanks
Ken Ohmichi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Jim Rollenhagen]

On Wed, Mar 16, 2016 at 10:29:48PM -0400, Adam Young wrote:
> On 03/16/2016 09:20 PM, Ken'ichi Ohmichi wrote:
> >Hi
> >
> >I have one proposal[1] related to negative tests in Tempest, and
> >hoping opinions before doing that.
> >
> >Now Tempest contains negative tests and sometimes patches are being
> >posted for adding more negative tests, but I'd like to propose
> >removing them from Tempest instead.
> >
> >Negative tests verify surfaces of REST APIs for each component without
> >any integrations between components. That doesn't seem integration
> >tests which are scope of Tempest.
> >In addition, we need to spend the test operating time on different
> >component's gate if adding negative tests into Tempest. For example,
> >we are operating negative tests of Keystone and more
> >components on the gate of Nova. That is meaningless, so we need to
> >avoid more negative tests into Tempest now.
> >
> >If wanting to add negative tests, it is a nice option to implement
> >these tests on each component repo with Tempest plugin interface. We
> >can avoid operating negative tests on different component gates and
> >each component team can decide what negative tests are valuable on the
> >gate.
> 
> Hear hear!  For Keystone, please put them in the Keystone Client Functional
> tests.

I'd recommend putting the negative tests in a plugin in the keystone
tree itself. Otherwise, how do you ensure that a change to keystone
doesn't suddenly make a bad API call go from 400 to 200, for instance?

(this assumes that keystoneclient functional tests don't run against
keystone changes, of course)

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Qiming Teng]

> 
> I'd love to see this idea explored further. What happens if Tempest
> ends up without tests, as a library for shared code as well as a
> centralized place to run tests from via plugins?
> 

Also curious about this. It seems weird to separate the 'positive' and
the 'negative' ones, assuming those patches are mostly contributed by
the same group of developers.

Qiming


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Daryl Walleck]

I do think negative tests should live on, most likely in their own 
tempest-plugin based repository. However, I'm not sure that I agree with the 
reason being because they are functional tests rather than integration. If you 
look at the current tests, one could argue that many non-Nova tests are 
actually functional tests. The Nova tests have the luck of requiring other 
services to function at all, but volumes, identity, and other services can be 
and are being tested in isolation in at least some Tempest tests. If removing 
functional tests from Tempest is problem that is trying to be solved, then 
there's a scope much larger than negative tests to address.

I'm also concerned about the idea of moving these tests back to their 
individual project repos. To run the same tests that I do today, I would need 
to install each individual project as well as Tempest to get the same coverage 
that I get today. That feels like a quite heavy burden just to be able to keep 
the same coverage as there is now. This would also mean that Tempest should 
likely be gated on the tests that exist in the project repositories, which 
seems like would add a great deal of complexity and maintenance.

While I understand that some of these tests may be a duplicate effort in the 
context of the gate, they are needed coverage when testing a deployed OpenStack 
environment. Rather than sending the negative tests and others scattered back 
to the individual projects, would it make sense to have a separate repository 
to hold "extended" Tempest tests? This would allow negative tests to stay in a 
centralized location, as well as provide a home for tests that might not be 
desirable for the gate (e.x. longer scenarios, the authorization tests, etc). 
What I would like to avoid is the pruning of tests that are meaningful to 
simply reduce execution time. If what is desired is a better definition between 
what is Tempest and what is not, then I think that is a more important 
conversation to have in depth.

Daryl

From: Ken'ichi Ohmichi <ken1ohmi...@gmail.com>
Sent: Wednesday, March 16, 2016 8:20 PM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [QA][all] Propose to remove negative tests from
Tempest

Hi

I have one proposal[1] related to negative tests in Tempest, and
hoping opinions before doing that.

Now Tempest contains negative tests and sometimes patches are being
posted for adding more negative tests, but I'd like to propose
removing them from Tempest instead.

Negative tests verify surfaces of REST APIs for each component without
any integrations between components. That doesn't seem integration
tests which are scope of Tempest.
In addition, we need to spend the test operating time on different
component's gate if adding negative tests into Tempest. For example,
we are operating negative tests of Keystone and more
components on the gate of Nova. That is meaningless, so we need to
avoid more negative tests into Tempest now.

If wanting to add negative tests, it is a nice option to implement
these tests on each component repo with Tempest plugin interface. We
can avoid operating negative tests on different component gates and
each component team can decide what negative tests are valuable on the
gate.

In long term, all negative tests will be migrated into each component
repo with Tempest plugin interface. We will be able to operate
valuable negative tests only on each gate.

Any thoughts?

Thanks
Ken Ohmichi

---
[1]: https://review.openstack.org/#/c/293197/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Valeriy Ponomaryov]

Main reason to split negative tests out, that I see, is attempt to remove
"corner, not integrational" tests out of Tempest tree. If so, then
"negative" tests is not proper choice, I think.
Tempest has lots of positive "corner" test cases. So, if move something,
then we need to move exactly "corner" tests and leave only "integrational"
tests of any kind - either it is negative or positive.
It is said, that doing so, all single-project related tests will be in its
own repo as a plugin and only integrational tests will be in Tempest.

Valeriy

On Thu, Mar 17, 2016 at 7:31 AM, Qiming Teng 
wrote:

> >
> > I'd love to see this idea explored further. What happens if Tempest
> > ends up without tests, as a library for shared code as well as a
> > centralized place to run tests from via plugins?
> >
>
> Also curious about this. It seems weird to separate the 'positive' and
> the 'negative' ones, assuming those patches are mostly contributed by
> the same group of developers.
>
> Qiming
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kind Regards
Valeriy Ponomaryov
www.mirantis.com
vponomar...@mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [QA][all] Propose to remove negative tests from Tempest

[Ken'ichi Ohmichi]

Hi

I have one proposal[1] related to negative tests in Tempest, and
hoping opinions before doing that.

Now Tempest contains negative tests and sometimes patches are being
posted for adding more negative tests, but I'd like to propose
removing them from Tempest instead.

Negative tests verify surfaces of REST APIs for each component without
any integrations between components. That doesn't seem integration
tests which are scope of Tempest.
In addition, we need to spend the test operating time on different
component's gate if adding negative tests into Tempest. For example,
we are operating negative tests of Keystone and more
components on the gate of Nova. That is meaningless, so we need to
avoid more negative tests into Tempest now.

If wanting to add negative tests, it is a nice option to implement
these tests on each component repo with Tempest plugin interface. We
can avoid operating negative tests on different component gates and
each component team can decide what negative tests are valuable on the
gate.

In long term, all negative tests will be migrated into each component
repo with Tempest plugin interface. We will be able to operate
valuable negative tests only on each gate.

Any thoughts?

Thanks
Ken Ohmichi

---
[1]: https://review.openstack.org/#/c/293197/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev