On 10/18/2013 01:54 PM, Sean Dague wrote:
On 10/18/2013 11:59 AM, Dolph Mathews wrote:
On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <sha...@redhat.com
<mailto:sha...@redhat.com>> wrote:
Hi all,
Starting a thread to discuss $subject, as requested in:
https://review.openstack.org/#/c/51558/
First a bit of background. I wrote a keystoneclient patch, and
ayoung
stated he'd like it tested via tempest before he'd ack it:
https://review.openstack.org/#/c/48462/
So I spoke to ayoung and dkranz on IRC, showing them my local tests
for the
patch. dkranz suggested creating a "client_lib" directory, where we
could
build out a more comprehensive set of tests over time, adding to the
inital
tests related to keystone trusts client additions.
A couple of things to note:
- These are end-to-end tests, designed to test not only the
client, but
also the API and keystone backend functionality. So arguably
this could
just be a scenario test, e.g scenario/keystone/test_v3_auth.py
I'd love to be able to run these tests against a wider variety of
service configurations (e.g. LDAP!), which tempest is obviously more
suitable for.
Realize that today, all the gate is a very simplistic keystone setup.
If there had been work to bring up different keystone backends with
the tests we currently have, I think I'd have a different take on
these tests.
My main focus is how we get the biggest bang for our buck, and up
until this point we've left direct client testing largely off the
table because we had API testing (so the API surface should be a known
quantity) and cli testing, because the cli turned out to be massively
full of exceptions. But client lib testing feels like it should be
able to be accomplished without redoing this all over again by
assuming the contract, mocking to it, and testing in unit tests.
I really don't understand why cli (shell programming language) and the
python clients should be treated differently. The exact same argument
could be made for cli.
Is there a reason we don't think that's viable?
Also, this is probably a good summit session, so if someone wants to
submit it, I'll land it on the QA track. Realize that if we do expand
tempest scope to include client libs, we really need to do so somewhat
systematically to cover all the clients, so we don't end up with just
a few keystone client tests and that's it.
I put this in the etherpad a few weeks ago "Strategy for avoiding
duplication with unit tests." We need a real strategy for where
different kinds of tests should go. And all this makes it even more
clear that we need a way to separate the functional description of a
test from the environment in which it can run. The decision of whether a
test should run in a real env, or mocked in various ways, should be more
abstracted from the actual test code where possible IMO.
-David
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
