Armando M. <[email protected]> wrote:
Hi Neutrinos,
I would like to share a proposal with you on how we could scale our
ever-growing testing needs, and at the same time, provide guidance to the
developers who care about the quality of the work they produce, and how
they can protect it from the dreadful regressions!
In [1] you will see a Decision Tree: the objective is to guide us in
deciding what is the best testing strategy for a feature, and what type
of support to expect from our CI engine. In a nutshell we have the
following basic options we can choose from:
• Unit testing
• Functional testing
• API testing (*)
• Fullstack testing
• Scenario testing (**)
The bottom line is: we should tap into our existing infrastructure as
much as we can to try to minimize the amount of CI moving parts that
allow us to test the numerous features that Neutron has. Some work has to
happen (as highlighted below) before we can make this decision process as
smooth as possible. In particular:
(*) The API testing framework allows us to execute tempest-style API
tests against a Neutron live server. Assaf and I are working to address
the overlap between Tempest and this framework, and to ensure that
there's a clear demarcation to what test belongs to Tempest and what test
belongs to Neutron codebases respectively. More will follow in another
thread, so stay tuned. On the other hand, there's some work that needs to
go into this testing framework to make the server load up non-default
extensions.
I think we already load up "non-default" extensions as needed (think QoS or
LBaaS):
https://github.com/openstack/neutron/blob/master/neutron/tests/contrib/gate_hook.sh#L25
(**) This is something that Ihar started in [2]. I am still unclear on
how we intend to leverage this job and yet keep scenario-like tests for
hardcore Neutron features in the Neutron tree. Ihar is thinking QoS, I
may be thinking dpdk, someone else might be thinking to something yet
again different (don't get bogged down on the actual examples). We'll
iterate on [2], but I see it as an integral part of our end-to-end
testing strategy. More to follow.
Side note: I don’t believe dpdk job would need tempest in the first place
since it does not touch API in any way, so could and should be covered with
functional/fullstack just fine. So let’s better talk e.g. ‘flavors’ here if
we need another example in addition to QoS.
As for new additional jobs that may come and go: I believe that we have
to revisit our approach to introduce experimental and non-voting jobs,
and how we make them stable and ultimately running as gate jobs and, long
term, having the features they tests supplant the ones they are meant to
supersede (think pecan, or dvr). I have an idea on how to address the
non-voting neglect conundrum, but I'll tackle it in another thread.
Finally, a word about projects that need testing with Neutron, advanced
services and beyond. The same way the neutron-lib initiative is tackling
the python side of the Neutron codebase as of now, we'll have to work to
identify the common pieces of functionality that will allow the reuse of
the testing machinery across multiple project, in a modular and decouple
fashion, so that when Neutron related projects want to integrate with the
Neutron CI engine, they will do so using good patterns rather than bad
ones; I believe that [3] is one of them, and that's why I have been
pushing back.
Apart from in-neutron work on common testing infrastructure, we need to
think of how we manage gate jobs for all stadium projects. F.e. most of
them are forced into adopting a huge hack-around called
tools/tox_install.sh that would install proper neutron for them. Instead of
reimplementing the script for every subproject, we better work with infra
to define common job macros that would manage neutron installation for us.
It would simplify our lives a lot if only we would not need to maintain the
script in each and every stadium repo.
Feedback welcome, as I am sure I may overlooked something.
Cheers,
Armando
[1] https://wiki.openstack.org/wiki/Network/Testing
[2] https://review.openstack.org/#/c/247697/
[3] https://review.openstack.org/#/c/242925/
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
