On 10/18/2013 01:54 PM, Sean Dague wrote:
On 10/18/2013 11:59 AM, Dolph Mathews wrote:
On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy <sha...@redhat.com
<mailto:sha...@redhat.com>> wrote:
Hi all,
Starting a thread to discuss $subject, as requested in:
https://review.openstack.org/#/c/51558/
First a bit of background. I wrote a keystoneclient patch, and
ayoung
stated he'd like it tested via tempest before he'd ack it:
https://review.openstack.org/#/c/48462/
So I spoke to ayoung and dkranz on IRC, showing them my local tests
for the
patch. dkranz suggested creating a "client_lib" directory, where we
could
build out a more comprehensive set of tests over time, adding to the
inital
tests related to keystone trusts client additions.
A couple of things to note:
- These are end-to-end tests, designed to test not only the
client, but
also the API and keystone backend functionality. So arguably
this could
just be a scenario test, e.g scenario/keystone/test_v3_auth.py
I'd love to be able to run these tests against a wider variety of
service configurations (e.g. LDAP!), which tempest is obviously more
suitable for.
Realize that today, all the gate is a very simplistic keystone setup.
If there had been work to bring up different keystone backends with
the tests we currently have, I think I'd have a different take on
these tests.
My main focus is how we get the biggest bang for our buck, and up
until this point we've left direct client testing largely off the
table because we had API testing (so the API surface should be a known
quantity) and cli testing, because the cli turned out to be massively
full of exceptions. But client lib testing feels like it should be
able to be accomplished without redoing this all over again by
assuming the contract, mocking to it, and testing in unit tests.
Is there a reason we don't think that's viable?
Also, this is probably a good summit session, so if someone wants to
submit it, I'll land it on the QA track. Realize that if we do expand
tempest scope to include client libs, we really need to do so somewhat
systematically to cover all the clients, so we don't end up with just
a few keystone client tests and that's it.
Client work needs to be in a project external to both the client and the
server. Hence our push toward Tempest. I think that all of the
projects have this same limitation: In order to get changes tested in
their client, they need a live server, but the client tests don't run a
server. We can't put the tests in the server, because the corresponding
code changes have not been yet checked in to the client. Chicken and Egg.
Client code is public code. In many cases, it is the primary way that
people integrate with Keystone, Nova, and other long established
projects. We cannot afford to break the contract in the Clients as that
breaks things for lots of consumers. Tempest is the system of record.
Having Tempest devs review tests keep Keystone (and other) devs
honest. No more sneaking through a unit test change with code changes
that secretly break things. Now, if you want to change a public API,
you need to address that in a Tempest test before ever making a change
to either Keystone or the Client. This is like double book accounting,
and it is a good-thing (tm).
I don't know, however, if we need to push all of the projects to do
this, or if, intstead, we should just let Keystone show the way. Once
the pattern is established, and we have worked out the kinks between
Tempest and Keystone, the Keystone devs can act as amabassadors to other
projects in order to pass on the accumulated wisdom.
There are a lot of short comings to the current testing. Keystone runs
SQL migration tests only against SQLite. This is a waste of time. In
addition, the gate does not run them against MySQL or PostGRES, which
means that bugs get through. The API tests are not run against multiple
Backends. At a minimum, we need to ensure that both SQL and LDAP are
tested for Identity, and SQL and Memcached are tested for Tokens.
Again, ideally against a real database, not just SQLite.
While we are submitting these tests to Tempest, they don't need to be
run for commits on other projects. Only for commits to Keystone and
Keystone Client. So the impact on the Gate jobs should be minimal.
Keystone patches are not so proligfic that they alone are killing the
gate, are they?
I think we can work toward a setup like this:
Devstack + mysql as the default.
Deploy a non-default domain via LDAP
Same thing for Postgtesql
The non-default domain in LDAP will require us sorting out some issues
we've identified, so I would not expect that to be ready until around
Icehouse 1 timeframe.
-Sean
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
