On 10/18/2013 01:54 PM, Sean Dague wrote:
On 10/18/2013 11:59 AM, Dolph Mathews wrote:

On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <sha...@redhat.com
<mailto:sha...@redhat.com>> wrote:

    Hi all,

    Starting a thread to discuss $subject, as requested in:


First a bit of background. I wrote a keystoneclient patch, and ayoung
    stated he'd like it tested via tempest before he'd ack it:


    So I spoke to ayoung and dkranz on IRC, showing them my local tests
    for the
    patch.  dkranz suggested creating a "client_lib" directory, where we
    build out a more comprehensive set of tests over time, adding to the
    tests related to keystone trusts client additions.

    A couple of things to note:
- These are end-to-end tests, designed to test not only the client, but
       also the API and keystone backend functionality.  So arguably
    this could
       just be a scenario test, e.g scenario/keystone/test_v3_auth.py

I'd love to be able to run these tests against a wider variety of
service configurations (e.g. LDAP!), which tempest is obviously more
suitable for.

Realize that today, all the gate is a very simplistic keystone setup. If there had been work to bring up different keystone backends with the tests we currently have, I think I'd have a different take on these tests.

My main focus is how we get the biggest bang for our buck, and up until this point we've left direct client testing largely off the table because we had API testing (so the API surface should be a known quantity) and cli testing, because the cli turned out to be massively full of exceptions. But client lib testing feels like it should be able to be accomplished without redoing this all over again by assuming the contract, mocking to it, and testing in unit tests.
I really don't understand why cli (shell programming language) and the python clients should be treated differently. The exact same argument could be made for cli.

Is there a reason we don't think that's viable?

Also, this is probably a good summit session, so if someone wants to submit it, I'll land it on the QA track. Realize that if we do expand tempest scope to include client libs, we really need to do so somewhat systematically to cover all the clients, so we don't end up with just a few keystone client tests and that's it.
I put this in the etherpad a few weeks ago "Strategy for avoiding duplication with unit tests." We need a real strategy for where different kinds of tests should go. And all this makes it even more clear that we need a way to separate the functional description of a test from the environment in which it can run. The decision of whether a test should run in a real env, or mocked in various ways, should be more abstracted from the actual test code where possible IMO.


OpenStack-dev mailing list

Reply via email to