On Fri, Oct 18, 2013 at 01:48:08PM -0400, Sean Dague wrote: > So v3 keystone API is one thing, but I'm a little concerned with > moving the client testing to Tempest haphazardly. If we are testing > the API surface on the servers, the clients should be able to > correctly test all of this via a mock of those API returns, which > would let us separate concerns here and keep the client tests close > to their code as unit tests.
IME testing the API and clients separately via mocked interfaces is not enough, you still end up with several categories of bugs which can (and do, regularly) sneak through: - The API implementation is wrong, and there is a corresponding mistake in the unit tests. This is obviously a common problem with any unit test containing mock responses, here's a recent example of such an issue, discovered while writing these tests.. https://review.openstack.org/#/c/51448/ - The API implementation is correct, but there are undocumented or non-obvious restrictions on combinations of arguments which work, or are allowed. This is a common issue with keystone IME, where some combinations work, and others give a 403 response with no information about why. So you end up reading the API code or trying stuff at random, whereas with a set of client tests, we can demonstrate and verify that all required/expected usage patterns work and stay working. > We're actually actively trying to figure out what can migrate out of > tempest back to the integrated projects, so that we get our biggest > bang for our buck. In that case, one could argue that the best thing is to only test via the clients, and move the API-surface tests back into the projects as integration tests. IMHO it makes sense to make the gate tests the most end-to-end tests possible, using the primary integration point for each project (which in most cases is the client not direct API interaction.) Thanks for all the discussion so far, I'll make sure I attend the summit session so we can figure out the next steps. Also I apologize if my patches caught you off-guard, I (mis)interpreted my initial chat with dkranz as blessing to proceed, and with Havana deadlines looming I just hacked out the tests in an effort to get my patch into keystoneclient. Thanks, Steve _______________________________________________ OpenStack-dev mailing list OpenStackemail@example.com http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev