On 10/30/2014 09:33 AM, David Kranz wrote:
> On 10/30/2014 07:49 AM, Sean Dague wrote:
>> On 10/29/2014 12:30 PM, Matthew Treinish wrote:
>>> Hi everyone,
>>> Before we start the larger discussion at summit next week about the future 
>>> of
>>> testing in OpenStack - specifically about spinning up functional testing 
>>> and how
>>> it relates to tempest - I would like to share some of my thoughts on how we 
>>> can
>>> get things started and how I think they'll eventually come together.
>>> Currently in tempest we have a large number of tests (mostly api-focused)
>>> which are probably a better fit for a project's functional test suite. The 
>>> best
>>> example I can think of is the nova flavors tests. Validation of flavor
>>> manipulation doesn't need to run in the integrated test suite on every 
>>> commit to
>>> every project because it only requires Nova. A simple win for initiating 
>>> in-tree
>>> functional testing would be to move these kinds of tests into the projects 
>>> and
>>> run the tests from the project repos instead of from tempest.
>> I think a lot of the negative API testing is also a great thing to be
>> done back at the project level. All of that testing should be able to
>> work without a full OpenStack, as it should be caught and managed by
>> the API service and never get any further than that.
>>> This would have the advantage of making tempest slimmer for every project
>>> and begin the process of getting projects to take responsibility for their
>>> functional testing rather than relying on tempest. As tests are moved 
>>> tempest
>>> can start to become the integration test suite it was meant to be. It would
>>> retain only tests that involve multiple projects and stop being the 
>>> OpenStack
>>> black box testing suite. I think that this is the right direction for 
>>> tempest
>>> moving forward, especially as we move to having project-specific functional
>>> testing.
>>> Doing this migration is dependent on some refactors in tempest and moving
>>> the required bits to tempest-lib so they can be easily consumed by the
>>> other projects. This will be discussed at summit, is being planned
>>> for implementation this cycle, and is similar to what is currently in 
>>> progress
>>> for the cli tests.
>>> The only reason this testing existed in tempest in the first place was as
>>> mechanism to block and then add friction against breaking api changes. 
>>> Tempest's
>>> api testing has been been pretty successful at achieving these goals. We'll 
>>> want
>>> to ensure that migrated tests retain these characteristics. If we are using
>>> clients from tempest-lib we should get this automatically since to break
>>> the api you'd have to change the api client. Another option proposed was to
>>> introduce a hacking rule that would block changes to api tests at the same 
>>> time
>>> other code was being changed.
>>> There is also a concern for external consumers of tempest if we move the 
>>> tests
>>> out of the tempest tree (I'm thinking refstack). I think the solution is
>>> to maintain a load_tests discovery method inside of tempest or elsewhere 
>>> that
>>> will run the appropriate tests from the other repos for something like 
>>> refstack.
>>> Assuming that things are built in a compatible way using the same framework 
>>> then
>>> running the tests from separate repos should be a simple matter of pointing 
>>> the
>>> test runner in the right direction.
>> I think we can see where this takes us. I'm still skeptical of cross
>> project loading of tests because it's often quite fragile. However,
>> if you look at what refstack did they had a giant evaluation of all
>> of tempest and pruned a bunch of stuff out. I would imagine maybe
>> there is a conversation there about tests that refstack feels are
>> important to stay in Tempest for their validation reasons. I think
>> having a few paths that are tested both in Tempest and in project
>> functional tests is not a bad thing.
> Refstack is not the only thing that cares about validation of real
> clouds. As we move forward with this, it would be good to separate the
> issues of "in which repo does a functional test live" and "can a
> functional test be run against a real cloud". IMO, over use of mocking
> (broadly defined) in functional tests should be avoided unless it is
> configurable to also work in an unmocked fashion. Whether the way to
> combine all of the functional tests is by cross project loading of
> tests or by some other means is more of an implementation detail.
Part of the perspective I'm bringing in is actually knowing what to do
when your tests fail. Using Tempest against real clouds is great, people
should keep doing that. But if you are rolling out a real cloud
yourself, in the future you should be running the functional tests in
staging to ensure you are functioning. Those will also provide you,
hopefully, with a better path to understand what's wrong.

This will mean that as an arbitrary 3rd party accessing a public cloud,
you don't have a test suite that pushes every button of the cloud. But I
think that's an acceptable trade off. Because pushing on odd edge cases
might produce a fail, but if there is no clear path from the fail to a
fix, it doesn't help anyone.

Finding failures in OpenStack is not hard. Finding failures in OpenStack
in a way that points towards resolution is what we need to be striving for.

>> But I think that's an end of cycle at best discussion.
>> Also, there probably need to be a few discussions anyway of
>> refstack/tempest/defcore. The fact that Keystone was dropped from
>> defcore because there were no non admin Keystone tests explicitly in
>> Tempest (even though we make over 5000 keystone non admin API calls
>> over a tempest run) was very odd. That is something that could have
>> been fixed in a day.
>>> I also want to comment on the role of functional testing. What I've proposed
>>> here is only one piece of what project specific functional testing should be
>>> and just what I feel is a good/easy start. I don't feel that this should be
>>> the only testing done in the projects.  I'm suggesting this as a first
>>> step because the tests already exist and it should be a relatively simple 
>>> task.
>>> I also feel that using tempest-lib like this shouldn't be a hard 
>>> requirement.
>>> Ideally the client definitions shouldn't have to live externally, or if 
>>> they did
>>> they would be the official clients, but I am suggesting this as a first 
>>> step to
>>> start a migration out of tempest.
>>> I don't want anyone to feel that they need block their functional testing
>>> efforts until tempest-lib becomes more useable. The larger value from 
>>> functional
>>> testing is actually in enabling testing more tightly coupled to the projects
>>> (e.g. whitebox testing). I feel that any work necessary to enable functional
>>> testing should be prioritized.
>> Agreed. From a Nova perspective I'm going to be layering up some of
>> this functionality with existing in tree tools. We also have a bunch
>> of potentially needed functional tests based on what I found in bug
>> triage this fall. So the priority is going to be getting something
>> working right to nail the bugs in Nova first, then align where it
>> makes sense.
> I'm not sure enabling white-box testing is the whole story.  Going
> forward, with project gates spending more time on relevant tests and
> more tests running post-merge, there are plenty of valuable black-box
> tests that we could have. Whitebox testing would be a valuable
> addition of course.
>> I actually think the negative test generator code in tempest would be
>> a good candidate for tempest lib, and would actually work better with
>> those negative definitions in the project as functional tests.
> ++
> I think a main reason more negative tests have not been created using
> the generator is that it requires deep knowledge of the schemas.
> Ideally, these schemas would be the output of some generator in each
> project, or provided by the project manually.
>  -David
>>     -Sean
>>> -Matt Treinish
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> -- 
>> Sean Dague
>> http://dague.net
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Sean Dague

Attachment: signature.asc
Description: OpenPGP digital signature

OpenStack-dev mailing list

Reply via email to