Re: [openstack-dev] [Neutron][QA] Call to action - Neutron/Tempest API tests dedup

[Assaf Muller]

On Sat, May 14, 2016 at 8:03 AM, zhi  wrote:
> hi, Muller.
>
> As you mentioned, is there will have a individual Tempest plugin named
> "Neutron Tempest plugin" in future?

It already exists :)

http://docs.openstack.org/developer/neutron/devref/development.environment.html#id3

The plugin was contributed by Daniel Mellado and we use it at the gate:
https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/neutron.yaml#L34

>
>
> Thanks
> Zhi Chang
>
> 2016-05-14 6:53 GMT+08:00 Assaf Muller :
>>
>> TL;DR: I'm looking for volunteers for tasks 1, 2 and 3 listed below.
>> Help would be hugely appreciated and %(local_drink)s will be bought in
>> Barcelona. I've posted example patches that demonstrate the idea.
>> Needless to say I'm here to provide reviews and to answer questions.
>> Additionally, most of the discussions have been within the Neutron
>> community and I'm looking for feedback from Tempest folks.
>>
>>
>> The context:
>> The Neutron community has been engaged in a long running effort to
>> move some of the networking API tests to the Neutron tree. We started
>> by copying the api/network directory tree, later keeping only the
>> tests, importing the test infrastructure itself from Tempest. We
>> continued by minimizing the imports we do from Tempest (Excluding
>> tempest.lib), and introduced a Neutron Tempest plugin.
>>
>> One issue that remains is that some of the tests are still found in
>> both repositories. This confuses contributors and wastes compute
>> resources. Since the tests run against stable/{liberty|mitaka} and
>> master, it should be safe to dedup. I proposed a line in the sand so
>> that 'core resources' remain to be tested in Tempest and more
>> 'advanced' APIs are tested in Neutron. The concept was agreed upon by
>> the Neutron and (Then) Tempest PTLs, and the specifics were discussed
>> and a consensus was found in patch [2]. Here is the resulting doc for
>> your viewing pleasure [5].
>>
>>
>> The work:
>> After I removed the API tests for core resources from the Neutron
>> tree, there remain three tasks to finish the de-dup:
>>
>> 1*) Remove tests for advanced APIs from Tempest. The full list of
>> tests that I propose be removed from Tempest is tracked here [1] (With
>> the rationale found at [2]), and an example patch may be found here
>> [4].
>> 2) Push tests for Neutron core resources that were added after the
>> fork from Tempest, then delete these from Neutron. This is also
>> tracked in [1], with example patches found here [6]. This is not a
>> strict cut/paste as the way Tempest and Neutron interact with clients
>> is slightly different. Fun!
>> 3) Sync tests for Neutron core resources that were updated after the
>> fork from Tempest. Test modifications include: Bug fixes for raceful
>> tests, py3 fixes, doc string typos and more. This is also tracked in
>> [1], with example patches found here [3].
>>
>> * I believe that as far as the Tempest test removal criteria found at
>> [7], this case falls under the first exception: 'The class of testing
>> has been decided to be outside the scope of tempest' and we may skip
>> the three prong rule for removal. Input welcome.
>>
>> [1] https://etherpad.openstack.org/p/neutron-tempest-defork
>> [2] https://review.openstack.org/#/c/280427/
>> [3] https://review.openstack.org/#/c/316280/ +
>> https://review.openstack.org/#/c/316283/
>> [4] https://review.openstack.org/#/c/316183/
>> [5]
>> docs.openstack.org/developer/neutron/devref/development.environment.html#api-tests
>> [6] https://review.openstack.org/#/c/316265/ +
>> https://review.openstack.org/#/c/316269/
>> [7] https://wiki.openstack.org/wiki/QA/Tempest-test-removal
>>
>> The work is tracked via:
>> * https://review.openstack.org/#/q/topic:bug/1552960
>> * https://bugs.launchpad.net/neutron/+bug/1552960
>> * https://etherpad.openstack.org/p/neutron-tempest-defork
>> * My head
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][QA] Call to action - Neutron/Tempest API tests dedup

[zhi]

hi, Muller.

As you mentioned, is there will have a individual Tempest plugin
named "Neutron
Tempest plugin" in future?

Thanks
Zhi Chang

2016-05-14 6:53 GMT+08:00 Assaf Muller :

> TL;DR: I'm looking for volunteers for tasks 1, 2 and 3 listed below.
> Help would be hugely appreciated and %(local_drink)s will be bought in
> Barcelona. I've posted example patches that demonstrate the idea.
> Needless to say I'm here to provide reviews and to answer questions.
> Additionally, most of the discussions have been within the Neutron
> community and I'm looking for feedback from Tempest folks.
>
>
> The context:
> The Neutron community has been engaged in a long running effort to
> move some of the networking API tests to the Neutron tree. We started
> by copying the api/network directory tree, later keeping only the
> tests, importing the test infrastructure itself from Tempest. We
> continued by minimizing the imports we do from Tempest (Excluding
> tempest.lib), and introduced a Neutron Tempest plugin.
>
> One issue that remains is that some of the tests are still found in
> both repositories. This confuses contributors and wastes compute
> resources. Since the tests run against stable/{liberty|mitaka} and
> master, it should be safe to dedup. I proposed a line in the sand so
> that 'core resources' remain to be tested in Tempest and more
> 'advanced' APIs are tested in Neutron. The concept was agreed upon by
> the Neutron and (Then) Tempest PTLs, and the specifics were discussed
> and a consensus was found in patch [2]. Here is the resulting doc for
> your viewing pleasure [5].
>
>
> The work:
> After I removed the API tests for core resources from the Neutron
> tree, there remain three tasks to finish the de-dup:
>
> 1*) Remove tests for advanced APIs from Tempest. The full list of
> tests that I propose be removed from Tempest is tracked here [1] (With
> the rationale found at [2]), and an example patch may be found here
> [4].
> 2) Push tests for Neutron core resources that were added after the
> fork from Tempest, then delete these from Neutron. This is also
> tracked in [1], with example patches found here [6]. This is not a
> strict cut/paste as the way Tempest and Neutron interact with clients
> is slightly different. Fun!
> 3) Sync tests for Neutron core resources that were updated after the
> fork from Tempest. Test modifications include: Bug fixes for raceful
> tests, py3 fixes, doc string typos and more. This is also tracked in
> [1], with example patches found here [3].
>
> * I believe that as far as the Tempest test removal criteria found at
> [7], this case falls under the first exception: 'The class of testing
> has been decided to be outside the scope of tempest' and we may skip
> the three prong rule for removal. Input welcome.
>
> [1] https://etherpad.openstack.org/p/neutron-tempest-defork
> [2] https://review.openstack.org/#/c/280427/
> [3] https://review.openstack.org/#/c/316280/ +
> https://review.openstack.org/#/c/316283/
> [4] https://review.openstack.org/#/c/316183/
> [5]
> docs.openstack.org/developer/neutron/devref/development.environment.html#api-tests
> [6] https://review.openstack.org/#/c/316265/ +
> https://review.openstack.org/#/c/316269/
> [7] https://wiki.openstack.org/wiki/QA/Tempest-test-removal
>
> The work is tracked via:
> * https://review.openstack.org/#/q/topic:bug/1552960
> * https://bugs.launchpad.net/neutron/+bug/1552960
> * https://etherpad.openstack.org/p/neutron-tempest-defork
> * My head
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][QA] Call to action - Neutron/Tempest API tests dedup

[Assaf Muller]

TL;DR: I'm looking for volunteers for tasks 1, 2 and 3 listed below.
Help would be hugely appreciated and %(local_drink)s will be bought in
Barcelona. I've posted example patches that demonstrate the idea.
Needless to say I'm here to provide reviews and to answer questions.
Additionally, most of the discussions have been within the Neutron
community and I'm looking for feedback from Tempest folks.


The context:
The Neutron community has been engaged in a long running effort to
move some of the networking API tests to the Neutron tree. We started
by copying the api/network directory tree, later keeping only the
tests, importing the test infrastructure itself from Tempest. We
continued by minimizing the imports we do from Tempest (Excluding
tempest.lib), and introduced a Neutron Tempest plugin.

One issue that remains is that some of the tests are still found in
both repositories. This confuses contributors and wastes compute
resources. Since the tests run against stable/{liberty|mitaka} and
master, it should be safe to dedup. I proposed a line in the sand so
that 'core resources' remain to be tested in Tempest and more
'advanced' APIs are tested in Neutron. The concept was agreed upon by
the Neutron and (Then) Tempest PTLs, and the specifics were discussed
and a consensus was found in patch [2]. Here is the resulting doc for
your viewing pleasure [5].


The work:
After I removed the API tests for core resources from the Neutron
tree, there remain three tasks to finish the de-dup:

1*) Remove tests for advanced APIs from Tempest. The full list of
tests that I propose be removed from Tempest is tracked here [1] (With
the rationale found at [2]), and an example patch may be found here
[4].
2) Push tests for Neutron core resources that were added after the
fork from Tempest, then delete these from Neutron. This is also
tracked in [1], with example patches found here [6]. This is not a
strict cut/paste as the way Tempest and Neutron interact with clients
is slightly different. Fun!
3) Sync tests for Neutron core resources that were updated after the
fork from Tempest. Test modifications include: Bug fixes for raceful
tests, py3 fixes, doc string typos and more. This is also tracked in
[1], with example patches found here [3].

* I believe that as far as the Tempest test removal criteria found at
[7], this case falls under the first exception: 'The class of testing
has been decided to be outside the scope of tempest' and we may skip
the three prong rule for removal. Input welcome.

[1] https://etherpad.openstack.org/p/neutron-tempest-defork
[2] https://review.openstack.org/#/c/280427/
[3] https://review.openstack.org/#/c/316280/ +
https://review.openstack.org/#/c/316283/
[4] https://review.openstack.org/#/c/316183/
[5] 
docs.openstack.org/developer/neutron/devref/development.environment.html#api-tests
[6] https://review.openstack.org/#/c/316265/ +
https://review.openstack.org/#/c/316269/
[7] https://wiki.openstack.org/wiki/QA/Tempest-test-removal

The work is tracked via:
* https://review.openstack.org/#/q/topic:bug/1552960
* https://bugs.launchpad.net/neutron/+bug/1552960
* https://etherpad.openstack.org/p/neutron-tempest-defork
* My head

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev