On 10/19/2013 04:46 AM, Steven Hardy wrote:
On Fri, Oct 18, 2013 at 01:48:08PM -0400, Sean Dague wrote:
So v3 keystone API is one thing, but I'm a little concerned with
moving the client testing to Tempest haphazardly. If we are testing
the API surface on the servers, the clients should be able to
correctly test all of this via a mock of those API returns, which
would let us separate concerns here and keep the client tests close
to their code as unit tests.
IME testing the API and clients separately via mocked interfaces is not
enough, you still end up with several categories of bugs which can (and do,
regularly) sneak through:
- The API implementation is wrong, and there is a corresponding mistake in
the unit tests. This is obviously a common problem with any unit test
containing mock responses, here's a recent example of such an issue,
discovered while writing these tests..
https://review.openstack.org/#/c/51448/
Ok, fair, but was that API being tested in Tempest? (grep -r
impersonation in tempest tree suggests not).
To be fair I wasn't suggesting mocking was good enough by itself, it was
the combination of:
* API coverage in Tempest to ensure API contract
* mocking in the project based on the fact that API was solidified by
Tempest real testing
- The API implementation is correct, but there are undocumented or
non-obvious restrictions on combinations of arguments which work, or are
allowed. This is a common issue with keystone IME, where some
combinations work, and others give a 403 response with no information
about why. So you end up reading the API code or trying stuff at random,
whereas with a set of client tests, we can demonstrate and verify that
all required/expected usage patterns work and stay working.
We're actually actively trying to figure out what can migrate out of
tempest back to the integrated projects, so that we get our biggest
bang for our buck.
In that case, one could argue that the best thing is to only test via the
clients, and move the API-surface tests back into the projects as
integration tests.
Here I think we'll have to disagree.
The OpenStack API is the contract. While other OpenStack components are
using the python clients for convenience, the API should be equally
accessible on any language. And having the API testing in tempest has
prevented some big compatibility breaks in our API, so that's going to
remain important.
And I think this is going to be an important part of this discussion
because of the role that the clients have in smoothing over potential
issues in the API.
So it will be important that client library testing, if/when we add it,
comes after the equivalent API testing being put in place. Ensuring API
solidity is first order priority, and today there a huge holes in that
for most projects in tempest (keystone definitely in that list).
IMHO it makes sense to make the gate tests the most end-to-end tests
possible, using the primary integration point for each project (which in
most cases is the client not direct API interaction.)
Definitely agree we should have plenty of end-to-end tests in the gate,
it's the reason we've got the scenario tests, to do exactly this kind of
through testing.
Thanks for all the discussion so far, I'll make sure I attend the summit
session so we can figure out the next steps.
Also I apologize if my patches caught you off-guard, I (mis)interpreted my
initial chat with dkranz as blessing to proceed, and with Havana deadlines
looming I just hacked out the tests in an effort to get my patch into
keystoneclient.
No worries, it was just new to most of us, and we were really deliberate
about the test directory structure and scope out of last summit, so it's
just something I don't want to change lightly. But the discussion is
flowing now, which is all good... and let's us figure out the best way
forward.
-Sean
--
Sean Dague
http://dague.net
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
