Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Steven Hardy]

On Mon, Oct 21, 2013 at 08:14:03AM -0400, Sean Dague wrote:
> On 10/21/2013 05:19 AM, Steven Hardy wrote:
> 
> >>
> >>Definitely agree we should have plenty of end-to-end tests in the
> >>gate, it's the reason we've got the scenario tests, to do exactly
> >>this kind of through testing.
> >
> >Ok, it seems like a potential solution which may keep all involved happy
> >would be:
> >- Add new API tests which provide full coverage of the documented
> >   interfaces to trusts
> >- Add a few scenario tests which provide end-to-end testing of the most
> >   important interfaces (these will use the client API)
> >
> >The scenario tests could just be those in my patches, moved from client_lib
> >to scenario/identity?
> 
> If there is a rush on a short term landing of code, making it a
> scenario test is a fine approach. And API tests for trust would be
> *highly* appreciated.

Ok, if we can land API tests then I guess the scenario/client tests can
wait until after the summit discussions (ayoung has indicated he's OK with
that plan in https://review.openstack.org/#/c/51558/)

I've raised this BP to track adding the API tests, I'll work on getting
those together:

https://blueprints.launchpad.net/tempest/+spec/keystone-trust-api

Thanks,

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Sean Dague]

On 10/21/2013 05:19 AM, Steven Hardy wrote:



Definitely agree we should have plenty of end-to-end tests in the
gate, it's the reason we've got the scenario tests, to do exactly
this kind of through testing.


Ok, it seems like a potential solution which may keep all involved happy
would be:
- Add new API tests which provide full coverage of the documented
   interfaces to trusts
- Add a few scenario tests which provide end-to-end testing of the most
   important interfaces (these will use the client API)

The scenario tests could just be those in my patches, moved from client_lib
to scenario/identity?


If there is a rush on a short term landing of code, making it a scenario 
test is a fine approach. And API tests for trust would be *highly* 
appreciated.


I still think the conversation about what's needed for Keystone in 
Icehouse is worth the summit conversation. I think I'd rather back it up 
from just the client_lib conversation, and actually talk about Keystone 
special needs out of the gate pipeline, and what kind of split of 
functionality on the tempest/devstack side vs. the unit test side makes 
sense.


So if there isn't a rush, I'd take it to summit, and lets come up with a 
more holistic plan there.


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Steven Hardy]

On Sat, Oct 19, 2013 at 07:45:33AM -0400, Sean Dague wrote:
> On 10/19/2013 04:46 AM, Steven Hardy wrote:
> >On Fri, Oct 18, 2013 at 01:48:08PM -0400, Sean Dague wrote:
> >>So v3 keystone API is one thing, but I'm a little concerned with
> >>moving the client testing to Tempest haphazardly.  If we are testing
> >>the API surface on the servers, the clients should be able to
> >>correctly test all of this via a mock of those API returns, which
> >>would let us separate concerns here and keep the client tests close
> >>to their code as unit tests.
> >
> >IME testing the API and clients separately via mocked interfaces is not
> >enough, you still end up with several categories of bugs which can (and do,
> >regularly) sneak through:
> >
> >- The API implementation is wrong, and there is a corresponding mistake in
> >   the unit tests.  This is obviously a common problem with any unit test
> >   containing mock responses, here's a recent example of such an issue,
> >   discovered while writing these tests..
> >
> >   https://review.openstack.org/#/c/51448/
> 
> Ok, fair, but was that API being tested in Tempest? (grep -r
> impersonation in tempest tree suggests not).

No, you're right, it looks like there's no API coverage of trusts at all.

> To be fair I wasn't suggesting mocking was good enough by itself, it
> was the combination of:
>  * API coverage in Tempest to ensure API contract
>  * mocking in the project based on the fact that API was solidified
> by Tempest real testing
> 
> >- The API implementation is correct, but there are undocumented or
> >   non-obvious restrictions on combinations of arguments which work, or are
> >   allowed.  This is a common issue with keystone IME, where some
> >   combinations work, and others give a 403 response with no information
> >   about why.  So you end up reading the API code or trying stuff at random,
> >   whereas with a set of client tests, we can demonstrate and verify that
> >   all required/expected usage patterns work and stay working.
> >
> >>We're actually actively trying to figure out what can migrate out of
> >>tempest back to the integrated projects, so that we get our biggest
> >>bang for our buck.
> >
> >In that case, one could argue that the best thing is to only test via the
> >clients, and move the API-surface tests back into the projects as
> >integration tests.
> 
> Here I think we'll have to disagree.
> 
> The OpenStack API is the contract. While other OpenStack components
> are using the python clients for convenience, the API should be
> equally accessible on any language. And having the API testing in
> tempest has prevented some big compatibility breaks in our API, so
> that's going to remain important.
> 
> And I think this is going to be an important part of this discussion
> because of the role that the clients have in smoothing over
> potential issues in the API.

Ok, so I'm not saying don't test the API, just that given your stated goal
of migrating stuff out of tempest, it is something which could potentially
be tested via unit/integration tests.

The problem is that none of the OpenStack projects use the API directly to
interact with keystone (or any other service for that matter), every single
one of them uses keystoneclient, so IMHO test coverage of the clients as
well as the API is really important.

> So it will be important that client library testing, if/when we add
> it, comes after the equivalent API testing being put in place.
> Ensuring API solidity is first order priority, and today there a
> huge holes in that for most projects in tempest (keystone definitely
> in that list).
> 
> >IMHO it makes sense to make the gate tests the most end-to-end tests
> >possible, using the primary integration point for each project (which in
> >most cases is the client not direct API interaction.)
> 
> Definitely agree we should have plenty of end-to-end tests in the
> gate, it's the reason we've got the scenario tests, to do exactly
> this kind of through testing.

Ok, it seems like a potential solution which may keep all involved happy
would be:
- Add new API tests which provide full coverage of the documented
  interfaces to trusts
- Add a few scenario tests which provide end-to-end testing of the most
  important interfaces (these will use the client API)

The scenario tests could just be those in my patches, moved from client_lib
to scenario/identity?

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Sean Dague]

On 10/19/2013 04:46 AM, Steven Hardy wrote:

On Fri, Oct 18, 2013 at 01:48:08PM -0400, Sean Dague wrote:

So v3 keystone API is one thing, but I'm a little concerned with
moving the client testing to Tempest haphazardly.  If we are testing
the API surface on the servers, the clients should be able to
correctly test all of this via a mock of those API returns, which
would let us separate concerns here and keep the client tests close
to their code as unit tests.


IME testing the API and clients separately via mocked interfaces is not
enough, you still end up with several categories of bugs which can (and do,
regularly) sneak through:

- The API implementation is wrong, and there is a corresponding mistake in
   the unit tests.  This is obviously a common problem with any unit test
   containing mock responses, here's a recent example of such an issue,
   discovered while writing these tests..

   https://review.openstack.org/#/c/51448/


Ok, fair, but was that API being tested in Tempest? (grep -r 
impersonation in tempest tree suggests not).


To be fair I wasn't suggesting mocking was good enough by itself, it was 
the combination of:

 * API coverage in Tempest to ensure API contract
 * mocking in the project based on the fact that API was solidified by 
Tempest real testing



- The API implementation is correct, but there are undocumented or
   non-obvious restrictions on combinations of arguments which work, or are
   allowed.  This is a common issue with keystone IME, where some
   combinations work, and others give a 403 response with no information
   about why.  So you end up reading the API code or trying stuff at random,
   whereas with a set of client tests, we can demonstrate and verify that
   all required/expected usage patterns work and stay working.


We're actually actively trying to figure out what can migrate out of
tempest back to the integrated projects, so that we get our biggest
bang for our buck.


In that case, one could argue that the best thing is to only test via the
clients, and move the API-surface tests back into the projects as
integration tests.


Here I think we'll have to disagree.

The OpenStack API is the contract. While other OpenStack components are 
using the python clients for convenience, the API should be equally 
accessible on any language. And having the API testing in tempest has 
prevented some big compatibility breaks in our API, so that's going to 
remain important.


And I think this is going to be an important part of this discussion 
because of the role that the clients have in smoothing over potential 
issues in the API.


So it will be important that client library testing, if/when we add it, 
comes after the equivalent API testing being put in place. Ensuring API 
solidity is first order priority, and today there a huge holes in that 
for most projects in tempest (keystone definitely in that list).



IMHO it makes sense to make the gate tests the most end-to-end tests
possible, using the primary integration point for each project (which in
most cases is the client not direct API interaction.)


Definitely agree we should have plenty of end-to-end tests in the gate, 
it's the reason we've got the scenario tests, to do exactly this kind of 
through testing.



Thanks for all the discussion so far, I'll make sure I attend the summit
session so we can figure out the next steps.

Also I apologize if my patches caught you off-guard, I (mis)interpreted my
initial chat with dkranz as blessing to proceed, and with Havana deadlines
looming I just hacked out the tests in an effort to get my patch into
keystoneclient.


No worries, it was just new to most of us, and we were really deliberate 
about the test directory structure and scope out of last summit, so it's 
just something I don't want to change lightly. But the discussion is 
flowing now, which is all good... and let's us figure out the best way 
forward.


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Sean Dague]

On 10/18/2013 09:10 PM, Adam Young wrote:

On 10/18/2013 07:21 PM, Sean Dague wrote:

On 10/18/2013 05:09 PM, Dolph Mathews wrote:


On Fri, Oct 18, 2013 at 3:19 PM, David Stanek mailto:dsta...@dstanek.com>> wrote:


On Fri, Oct 18, 2013 at 1:48 PM, Sean Dague mailto:s...@dague.net>> wrote:

On 10/18/2013 12:04 PM, Brant Knudson wrote:


2) "git clone"ing the keystoneclient doesn't work well with
parallel
testing (we have a similar problem in our tests with our
"pristine"
database backup)


Can you go into the specifics of why?


We use unsafe paths for the test SQLite database and test config
files.  Instead of using something like tempfile we are using
hardcoded paths.  When the setUp method is run in parallel it will
stomp on other tests.  I believe the 'git clone' is the same way.
  The clone happens in the setUp so if you have 2 test methods in
that test class one of the cloning operations will break.

I have a bug filed for the DB/config file issue already. The
cloning issue may solved by putting it into the setupClass instead
of setUp.  I'd have to try it.


test_keystoneclient is really an integration test between the client &
server, but expecting internet access to run the tests in keystone's own
repo has been a long-standing complaint (although this bug was only
recently filed):
https://bugs.launchpad.net/keystone/+bug/1191999


Ok, cool. It sounds like we actually should probably talk through
what's all needed to do this right for keystone at summit, especially
as we're talking about creating a new class of tests for it. Can
someone propose a session for it in the QA track?

I think many of us were caught off guard by the patch as typically
we're pretty deliberate about structure in the tree, and it hadn't yet
come up at a QA meeting. So a summit session probably could get us all
agreed on a plan moving forward and make sure we figure out all the
needs of keystone here. I do think it might warrant uniqueness given
that *all* the other services need it.

-Sean


Done
http://summit.openstack.org/cfp/details/313


Accepted. See you there.

-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Steven Hardy]

On Fri, Oct 18, 2013 at 01:48:08PM -0400, Sean Dague wrote:
> So v3 keystone API is one thing, but I'm a little concerned with
> moving the client testing to Tempest haphazardly.  If we are testing
> the API surface on the servers, the clients should be able to
> correctly test all of this via a mock of those API returns, which
> would let us separate concerns here and keep the client tests close
> to their code as unit tests.

IME testing the API and clients separately via mocked interfaces is not
enough, you still end up with several categories of bugs which can (and do,
regularly) sneak through:

- The API implementation is wrong, and there is a corresponding mistake in
  the unit tests.  This is obviously a common problem with any unit test
  containing mock responses, here's a recent example of such an issue,
  discovered while writing these tests..

  https://review.openstack.org/#/c/51448/

- The API implementation is correct, but there are undocumented or
  non-obvious restrictions on combinations of arguments which work, or are
  allowed.  This is a common issue with keystone IME, where some
  combinations work, and others give a 403 response with no information
  about why.  So you end up reading the API code or trying stuff at random,
  whereas with a set of client tests, we can demonstrate and verify that
  all required/expected usage patterns work and stay working.

> We're actually actively trying to figure out what can migrate out of
> tempest back to the integrated projects, so that we get our biggest
> bang for our buck.

In that case, one could argue that the best thing is to only test via the
clients, and move the API-surface tests back into the projects as
integration tests.

IMHO it makes sense to make the gate tests the most end-to-end tests
possible, using the primary integration point for each project (which in
most cases is the client not direct API interaction.)

Thanks for all the discussion so far, I'll make sure I attend the summit
session so we can figure out the next steps.

Also I apologize if my patches caught you off-guard, I (mis)interpreted my
initial chat with dkranz as blessing to proceed, and with Havana deadlines
looming I just hacked out the tests in an effort to get my patch into
keystoneclient.

Thanks,

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Adam Young]

On 10/18/2013 07:21 PM, Sean Dague wrote:

On 10/18/2013 05:09 PM, Dolph Mathews wrote:


On Fri, Oct 18, 2013 at 3:19 PM, David Stanek mailto:dsta...@dstanek.com>> wrote:


On Fri, Oct 18, 2013 at 1:48 PM, Sean Dague mailto:s...@dague.net>> wrote:

On 10/18/2013 12:04 PM, Brant Knudson wrote:


2) "git clone"ing the keystoneclient doesn't work well with
parallel
testing (we have a similar problem in our tests with our
"pristine"
database backup)


Can you go into the specifics of why?


We use unsafe paths for the test SQLite database and test config
files.  Instead of using something like tempfile we are using
hardcoded paths.  When the setUp method is run in parallel it will
stomp on other tests.  I believe the 'git clone' is the same way.
  The clone happens in the setUp so if you have 2 test methods in
that test class one of the cloning operations will break.

I have a bug filed for the DB/config file issue already. The
cloning issue may solved by putting it into the setupClass instead
of setUp.  I'd have to try it.


test_keystoneclient is really an integration test between the client &
server, but expecting internet access to run the tests in keystone's own
repo has been a long-standing complaint (although this bug was only
recently filed):
https://bugs.launchpad.net/keystone/+bug/1191999


Ok, cool. It sounds like we actually should probably talk through 
what's all needed to do this right for keystone at summit, especially 
as we're talking about creating a new class of tests for it. Can 
someone propose a session for it in the QA track?


I think many of us were caught off guard by the patch as typically 
we're pretty deliberate about structure in the tree, and it hadn't yet 
come up at a QA meeting. So a summit session probably could get us all 
agreed on a plan moving forward and make sure we figure out all the 
needs of keystone here. I do think it might warrant uniqueness given 
that *all* the other services need it.


-Sean


Done
http://summit.openstack.org/cfp/details/313

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Sean Dague]

On 10/18/2013 05:09 PM, Dolph Mathews wrote:


On Fri, Oct 18, 2013 at 3:19 PM, David Stanek mailto:dsta...@dstanek.com>> wrote:


On Fri, Oct 18, 2013 at 1:48 PM, Sean Dague mailto:s...@dague.net>> wrote:

On 10/18/2013 12:04 PM, Brant Knudson wrote:


2) "git clone"ing the keystoneclient doesn't work well with
parallel
testing (we have a similar problem in our tests with our
"pristine"
database backup)


Can you go into the specifics of why?


We use unsafe paths for the test SQLite database and test config
files.  Instead of using something like tempfile we are using
hardcoded paths.  When the setUp method is run in parallel it will
stomp on other tests.  I believe the 'git clone' is the same way.
  The clone happens in the setUp so if you have 2 test methods in
that test class one of the cloning operations will break.

I have a bug filed for the DB/config file issue already.  The
cloning issue may solved by putting it into the setupClass instead
of setUp.  I'd have to try it.


test_keystoneclient is really an integration test between the client &
server, but expecting internet access to run the tests in keystone's own
repo has been a long-standing complaint (although this bug was only
recently filed):
https://bugs.launchpad.net/keystone/+bug/1191999


Ok, cool. It sounds like we actually should probably talk through what's 
all needed to do this right for keystone at summit, especially as we're 
talking about creating a new class of tests for it. Can someone propose 
a session for it in the QA track?


I think many of us were caught off guard by the patch as typically we're 
pretty deliberate about structure in the tree, and it hadn't yet come up 
at a QA meeting. So a summit session probably could get us all agreed on 
a plan moving forward and make sure we figure out all the needs of 
keystone here. I do think it might warrant uniqueness given that *all* 
the other services need it.


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Dolph Mathews]

On Fri, Oct 18, 2013 at 3:19 PM, David Stanek  wrote:

>
> On Fri, Oct 18, 2013 at 1:48 PM, Sean Dague  wrote:
>
>> On 10/18/2013 12:04 PM, Brant Knudson wrote:
>>
>>>
>>> 2) "git clone"ing the keystoneclient doesn't work well with parallel
>>> testing (we have a similar problem in our tests with our "pristine"
>>> database backup)
>>>
>>
>> Can you go into the specifics of why?
>>
>
> We use unsafe paths for the test SQLite database and test config files.
>  Instead of using something like tempfile we are using hardcoded paths.
>  When the setUp method is run in parallel it will stomp on other tests.  I
> believe the 'git clone' is the same way.  The clone happens in the setUp so
> if you have 2 test methods in that test class one of the cloning operations
> will break.
>
> I have a bug filed for the DB/config file issue already.  The cloning
> issue may solved by putting it into the setupClass instead of setUp.  I'd
> have to try it.
>

test_keystoneclient is really an integration test between the client &
server, but expecting internet access to run the tests in keystone's own
repo has been a long-standing complaint (although this bug was only
recently filed):
https://bugs.launchpad.net/keystone/+bug/1191999


>
>
> --
> David
> blog: http://www.traceback.org
> twitter: http://twitter.com/dstanek
> www: http://dstanek.com
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

-Dolph
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[David Stanek]

On Fri, Oct 18, 2013 at 1:48 PM, Sean Dague  wrote:

> On 10/18/2013 12:04 PM, Brant Knudson wrote:
>
>>
>> 2) "git clone"ing the keystoneclient doesn't work well with parallel
>> testing (we have a similar problem in our tests with our "pristine"
>> database backup)
>>
>
> Can you go into the specifics of why?


We use unsafe paths for the test SQLite database and test config files.
 Instead of using something like tempfile we are using hardcoded paths.
 When the setUp method is run in parallel it will stomp on other tests.  I
believe the 'git clone' is the same way.  The clone happens in the setUp so
if you have 2 test methods in that test class one of the cloning operations
will break.

I have a bug filed for the DB/config file issue already.  The cloning issue
may solved by putting it into the setupClass instead of setUp.  I'd have to
try it.

-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
www: http://dstanek.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Adam Young]

On 10/18/2013 01:54 PM, Sean Dague wrote:

On 10/18/2013 11:59 AM, Dolph Mathews wrote:


On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy mailto:sha...@redhat.com>> wrote:

Hi all,

Starting a thread to discuss $subject, as requested in:

https://review.openstack.org/#/c/51558/

First a bit of background.  I wrote a keystoneclient patch, and 
ayoung

stated he'd like it tested via tempest before he'd ack it:

https://review.openstack.org/#/c/48462/

So I spoke to ayoung and dkranz on IRC, showing them my local tests
for the
patch.  dkranz suggested creating a "client_lib" directory, where we
could
build out a more comprehensive set of tests over time, adding to the
inital
tests related to keystone trusts client additions.

A couple of things to note:
- These are end-to-end tests, designed to test not only the 
client, but

   also the API and keystone backend functionality.  So arguably
this could
   just be a scenario test, e.g scenario/keystone/test_v3_auth.py


I'd love to be able to run these tests against a wider variety of
service configurations (e.g. LDAP!), which tempest is obviously more
suitable for.


Realize that today, all the gate is a very simplistic keystone setup. 
If there had been work to bring up different keystone backends with 
the tests we currently have, I think I'd have a different take on 
these tests.


My main focus is how we get the biggest bang for our buck, and up 
until this point we've left direct client testing largely off the 
table because we had API testing (so the API surface should be a known 
quantity) and cli testing, because the cli turned out to be massively 
full of exceptions. But client lib testing feels like it should be 
able to be accomplished without redoing this all over again by 
assuming the contract, mocking to it, and testing in unit tests.


Is there a reason we don't think that's viable?

Also, this is probably a good summit session, so if someone wants to 
submit it, I'll land it on the QA track. Realize that if we do expand 
tempest scope to include client libs, we really need to do so somewhat 
systematically to cover all the clients, so we don't end up with just 
a few keystone client tests and that's it.


Client work needs to be in a project external to both the client and the 
server.  Hence our push toward Tempest.  I think that all of the 
projects have this same limitation:  In order to get changes tested in 
their client, they need a live server, but the client tests don't run a 
server.  We can't put the tests in the server, because the corresponding 
code changes have not been yet checked in to the client.  Chicken and Egg.


Client code is public code.  In many cases, it is the primary way that 
people integrate with Keystone, Nova, and other long established 
projects.  We cannot afford to break the contract in the Clients as that 
breaks things for lots of consumers.  Tempest is the system of record.  
Having Tempest devs review tests keep Keystone (and other) devs 
honest.   No more sneaking through a unit test change with code changes 
that secretly break things.  Now, if you want to change a public API,  
you need to address that in a Tempest test before ever making a change 
to either Keystone or the Client. This is like double book accounting, 
and it is a good-thing (tm).


I don't know, however, if we need to push all of the projects to do 
this, or if, intstead, we should just let Keystone show the way. Once 
the pattern is established, and we have worked out the kinks between 
Tempest and Keystone, the Keystone devs can act as amabassadors to other 
projects in order to pass on the accumulated wisdom.


There are a lot of short comings to the current testing.  Keystone runs 
SQL migration tests only against SQLite.  This is a waste of time.  In 
addition, the gate does not run them against MySQL or PostGRES,  which 
means that bugs get through.  The API tests are not run against multiple 
Backends.  At a minimum, we need to ensure that both SQL and LDAP are 
tested for Identity, and SQL and Memcached are tested for Tokens.  
Again, ideally against a real database, not just SQLite.


While we are submitting these tests to Tempest, they don't need to be 
run for commits on other projects.  Only for commits to Keystone and 
Keystone Client.  So the impact on the Gate jobs should be minimal.  
Keystone patches are not so proligfic that they alone are killing the 
gate, are they?


I think we can work toward a setup like this:

Devstack + mysql as the default.
Deploy a non-default domain via LDAP

Same thing for Postgtesql
The non-default domain in LDAP will require us sorting out some issues 
we've identified, so I would not expect that to be ready until around 
Icehouse 1 timeframe.






-Sean




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listin

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[David Kranz]

On 10/18/2013 01:54 PM, Sean Dague wrote:

On 10/18/2013 11:59 AM, Dolph Mathews wrote:


On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy mailto:sha...@redhat.com>> wrote:

Hi all,

Starting a thread to discuss $subject, as requested in:

https://review.openstack.org/#/c/51558/

First a bit of background.  I wrote a keystoneclient patch, and 
ayoung

stated he'd like it tested via tempest before he'd ack it:

https://review.openstack.org/#/c/48462/

So I spoke to ayoung and dkranz on IRC, showing them my local tests
for the
patch.  dkranz suggested creating a "client_lib" directory, where we
could
build out a more comprehensive set of tests over time, adding to the
inital
tests related to keystone trusts client additions.

A couple of things to note:
- These are end-to-end tests, designed to test not only the 
client, but

   also the API and keystone backend functionality.  So arguably
this could
   just be a scenario test, e.g scenario/keystone/test_v3_auth.py


I'd love to be able to run these tests against a wider variety of
service configurations (e.g. LDAP!), which tempest is obviously more
suitable for.


Realize that today, all the gate is a very simplistic keystone setup. 
If there had been work to bring up different keystone backends with 
the tests we currently have, I think I'd have a different take on 
these tests.


My main focus is how we get the biggest bang for our buck, and up 
until this point we've left direct client testing largely off the 
table because we had API testing (so the API surface should be a known 
quantity) and cli testing, because the cli turned out to be massively 
full of exceptions. But client lib testing feels like it should be 
able to be accomplished without redoing this all over again by 
assuming the contract, mocking to it, and testing in unit tests.
I really don't understand why cli (shell programming language) and the 
python clients should be treated differently. The exact same argument 
could be made for cli.


Is there a reason we don't think that's viable?

Also, this is probably a good summit session, so if someone wants to 
submit it, I'll land it on the QA track. Realize that if we do expand 
tempest scope to include client libs, we really need to do so somewhat 
systematically to cover all the clients, so we don't end up with just 
a few keystone client tests and that's it.
I put this in the etherpad a few weeks ago "Strategy for avoiding 
duplication with unit tests." We need a real strategy for where 
different kinds of tests should go. And all this makes it even more 
clear that we need a way to separate the functional description of a 
test from the environment in which it can run. The decision of whether a 
test should run in a real env, or mocked in various ways, should be more 
abstracted from the actual test code where possible IMO.


 -David



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Sean Dague]

On 10/18/2013 11:59 AM, Dolph Mathews wrote:


On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy mailto:sha...@redhat.com>> wrote:

Hi all,

Starting a thread to discuss $subject, as requested in:

https://review.openstack.org/#/c/51558/

First a bit of background.  I wrote a keystoneclient patch, and ayoung
stated he'd like it tested via tempest before he'd ack it:

https://review.openstack.org/#/c/48462/

So I spoke to ayoung and dkranz on IRC, showing them my local tests
for the
patch.  dkranz suggested creating a "client_lib" directory, where we
could
build out a more comprehensive set of tests over time, adding to the
inital
tests related to keystone trusts client additions.

A couple of things to note:
- These are end-to-end tests, designed to test not only the client, but
   also the API and keystone backend functionality.  So arguably
this could
   just be a scenario test, e.g scenario/keystone/test_v3_auth.py


I'd love to be able to run these tests against a wider variety of
service configurations (e.g. LDAP!), which tempest is obviously more
suitable for.


Realize that today, all the gate is a very simplistic keystone setup. If 
there had been work to bring up different keystone backends with the 
tests we currently have, I think I'd have a different take on these tests.


My main focus is how we get the biggest bang for our buck, and up until 
this point we've left direct client testing largely off the table 
because we had API testing (so the API surface should be a known 
quantity) and cli testing, because the cli turned out to be massively 
full of exceptions. But client lib testing feels like it should be able 
to be accomplished without redoing this all over again by assuming the 
contract, mocking to it, and testing in unit tests.


Is there a reason we don't think that's viable?

Also, this is probably a good summit session, so if someone wants to 
submit it, I'll land it on the QA track. Realize that if we do expand 
tempest scope to include client libs, we really need to do so somewhat 
systematically to cover all the clients, so we don't end up with just a 
few keystone client tests and that's it.


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Adam Young]

On 10/18/2013 12:04 PM, Brant Knudson wrote:
To provide a bit more background... Keystone has a bunch of 
keystoneclient tests for the v2 API. These tests actually "git clone" 
a version of keystoneclient (master, essex-3, and 0.1.1)[0], to use 
for testing.  Maybe at some point the tests were just for 
client-server compatibility, but now they're used for more than that; 
for example, they're used for tests that require going through the 
paste pipeline and v2 controller. It's just the quickest way to get 
some tests written. In addition, there's versions of the client tests 
for both the kvs and sql backends.


This causes several problems,
1) It looks like we're not keeping the versions to test up-to-date -- 
should we checkout supported releases instead?
2) "git clone"ing the keystoneclient doesn't work well with parallel 
testing (we have a similar problem in our tests with our "pristine" 
database backup)

3) These tests eat up lots of memory which we've gotten complaints about.

Getting v3 API keystoneclient/keystone testing in tempest is going to 
hopefully lead to getting the v2 tests out of Keystone. Thanks to 
Steve for taking this first step!
I was going to try an approach where we used tempest to just call the 
code in Keystone as a first step.  That was one of the reasons that I 
was in favor of moving the Keystone tests into the keystone namespace.


We need to skip the git clone, step, obviously, and I am not certain 
about our use of fixtures:  we might need to redo these so the sample 
data doesn't conflict with what Tempest expects.






For the v3 API, the tests don't use the keystoneclient but instead use 
webtest [1] and the REST API.


[0] 
https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070
[1] 
https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69


We'll need V3 client support eventually, and we should use Tempest as 
the primary test environement  for that.




- Brant



On Fri, Oct 18, 2013 at 10:59 AM, Dolph Mathews 
mailto:dolph.math...@gmail.com>> wrote:



On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy mailto:sha...@redhat.com>> wrote:

Hi all,

Starting a thread to discuss $subject, as requested in:

https://review.openstack.org/#/c/51558/

First a bit of background.  I wrote a keystoneclient patch,
and ayoung
stated he'd like it tested via tempest before he'd ack it:

https://review.openstack.org/#/c/48462/

So I spoke to ayoung and dkranz on IRC, showing them my local
tests for the
patch.  dkranz suggested creating a "client_lib" directory,
where we could
build out a more comprehensive set of tests over time, adding
to the inital
tests related to keystone trusts client additions.

A couple of things to note:
- These are end-to-end tests, designed to test not only the
client, but
  also the API and keystone backend functionality.  So
arguably this could
  just be a scenario test, e.g scenario/keystone/test_v3_auth.py


I'd love to be able to run these tests against a wider variety of
service configurations (e.g. LDAP!), which tempest is obviously
more suitable for.


- The intention is to excercise logic which is hard to fully
test with
  unit or integration tests, and to catch issues like
incompatibility
  between client and API - e.g keystoneclient tests may pass,
but we need
  to make sure the client actually works against the real
keystone API.


All of our tests under keystone.tests.test_keystoneclient fall
into this category as well:


https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py


https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py


Working on Heat has given me a pretty good insight into the
python-*client
API's, as we use them to orchestrate actions with every
openstack service;
IMO anything we can do to make these interfaces more robust
(and catch
bugs, several of which I found already while writing these
tests) is a
good-thing (tm).


++


I'd welcome feedback on the patch above, and what will be the most
acceptable approach to the tempest team for adding these tests.

More links:

https://review.openstack.org/#/c/51559/
https://review.openstack.org/#/c/51560/
https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api

Thanks!

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.op

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Sean Dague]

On 10/18/2013 12:04 PM, Brant Knudson wrote:

To provide a bit more background... Keystone has a bunch of
keystoneclient tests for the v2 API. These tests actually "git clone" a
version of keystoneclient (master, essex-3, and 0.1.1)[0], to use for
testing.  Maybe at some point the tests were just for client-server
compatibility, but now they're used for more than that; for example,
they're used for tests that require going through the paste pipeline and
v2 controller. It's just the quickest way to get some tests written. In
addition, there's versions of the client tests for both the kvs and sql
backends.

This causes several problems,
1) It looks like we're not keeping the versions to test up-to-date --
should we checkout supported releases instead?
2) "git clone"ing the keystoneclient doesn't work well with parallel
testing (we have a similar problem in our tests with our "pristine"
database backup)


Can you go into the specifics of why?


3) These tests eat up lots of memory which we've gotten complaints about.


Again, can you go into the specifics of why?


Getting v3 API keystoneclient/keystone testing in tempest is going to
hopefully lead to getting the v2 tests out of Keystone. Thanks to Steve
for taking this first step!

For the v3 API, the tests don't use the keystoneclient but instead use
webtest [1] and the REST API.

[0]
https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070
[1]
https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69


So v3 keystone API is one thing, but I'm a little concerned with moving 
the client testing to Tempest haphazardly.  If we are testing the API 
surface on the servers, the clients should be able to correctly test all 
of this via a mock of those API returns, which would let us separate 
concerns here and keep the client tests close to their code as unit tests.


We're actually actively trying to figure out what can migrate out of 
tempest back to the integrated projects, so that we get our biggest bang 
for our buck.


Also, realize in a tempest environment there is only going to be the 
latest version of the clients, so this is going to massively reduce your 
test environment from what you have today.


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[David Kranz]

Thanks, Steve. I suggested a new directory because we really need to 
have more complete tests of the client libs since they are not tied to 
particular OpenStack releases and we claim the current libs should work 
with older releases. That said, I did not realize the intent was to do 
more than test the client libraries. If the intent of the follow-ons to 
this patch is to be more scenario-like rather than just increasing 
coverage for the keystoneclient api then it makes sense to go in 
scenario/keystone.


I still think we should have more client lib coverage than the haphazard 
usage we will get from scenario but a new directory could wait until 
such code starts to exist.


 -David

On 10/18/2013 11:34 AM, Steven Hardy wrote:

Hi all,

Starting a thread to discuss $subject, as requested in:

https://review.openstack.org/#/c/51558/

First a bit of background.  I wrote a keystoneclient patch, and ayoung
stated he'd like it tested via tempest before he'd ack it:

https://review.openstack.org/#/c/48462/

So I spoke to ayoung and dkranz on IRC, showing them my local tests for the
patch.  dkranz suggested creating a "client_lib" directory, where we could
build out a more comprehensive set of tests over time, adding to the inital
tests related to keystone trusts client additions.

A couple of things to note:
- These are end-to-end tests, designed to test not only the client, but
   also the API and keystone backend functionality.  So arguably this could
   just be a scenario test, e.g scenario/keystone/test_v3_auth.py

- The intention is to excercise logic which is hard to fully test with
   unit or integration tests, and to catch issues like incompatibility
   between client and API - e.g keystoneclient tests may pass, but we need
   to make sure the client actually works against the real keystone API.

Working on Heat has given me a pretty good insight into the python-*client
API's, as we use them to orchestrate actions with every openstack service;
IMO anything we can do to make these interfaces more robust (and catch
bugs, several of which I found already while writing these tests) is a
good-thing (tm).

I'd welcome feedback on the patch above, and what will be the most
acceptable approach to the tempest team for adding these tests.

More links:

https://review.openstack.org/#/c/51559/
https://review.openstack.org/#/c/51560/
https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api

Thanks!

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Brant Knudson]

To provide a bit more background... Keystone has a bunch of keystoneclient
tests for the v2 API. These tests actually "git clone" a version of
keystoneclient (master, essex-3, and 0.1.1)[0], to use for testing.  Maybe
at some point the tests were just for client-server compatibility, but now
they're used for more than that; for example, they're used for tests that
require going through the paste pipeline and v2 controller. It's just the
quickest way to get some tests written. In addition, there's versions of
the client tests for both the kvs and sql backends.

This causes several problems,
1) It looks like we're not keeping the versions to test up-to-date --
should we checkout supported releases instead?
2) "git clone"ing the keystoneclient doesn't work well with parallel
testing (we have a similar problem in our tests with our "pristine"
database backup)
3) These tests eat up lots of memory which we've gotten complaints about.

Getting v3 API keystoneclient/keystone testing in tempest is going to
hopefully lead to getting the v2 tests out of Keystone. Thanks to Steve for
taking this first step!

For the v3 API, the tests don't use the keystoneclient but instead use
webtest [1] and the REST API.

[0]
https://github.com/openstack/keystone/blob/master/keystone/tests/test_keystoneclient.py#L1070
[1]
https://github.com/openstack/keystone/blob/master/keystone/tests/test_content_types.py#L69

- Brant



On Fri, Oct 18, 2013 at 10:59 AM, Dolph Mathews wrote:

>
> On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy  wrote:
>
>> Hi all,
>>
>> Starting a thread to discuss $subject, as requested in:
>>
>> https://review.openstack.org/#/c/51558/
>>
>> First a bit of background.  I wrote a keystoneclient patch, and ayoung
>> stated he'd like it tested via tempest before he'd ack it:
>>
>> https://review.openstack.org/#/c/48462/
>>
>> So I spoke to ayoung and dkranz on IRC, showing them my local tests for
>> the
>> patch.  dkranz suggested creating a "client_lib" directory, where we could
>> build out a more comprehensive set of tests over time, adding to the
>> inital
>> tests related to keystone trusts client additions.
>>
>> A couple of things to note:
>> - These are end-to-end tests, designed to test not only the client, but
>>   also the API and keystone backend functionality.  So arguably this could
>>   just be a scenario test, e.g scenario/keystone/test_v3_auth.py
>>
>
> I'd love to be able to run these tests against a wider variety of service
> configurations (e.g. LDAP!), which tempest is obviously more suitable for.
>
>
>>
>> - The intention is to excercise logic which is hard to fully test with
>>   unit or integration tests, and to catch issues like incompatibility
>>   between client and API - e.g keystoneclient tests may pass, but we need
>>   to make sure the client actually works against the real keystone API.
>>
>
> All of our tests under keystone.tests.test_keystoneclient fall into this
> category as well:
>
>
> https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py
>
>
> https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py
>
>
>>
>> Working on Heat has given me a pretty good insight into the python-*client
>> API's, as we use them to orchestrate actions with every openstack service;
>> IMO anything we can do to make these interfaces more robust (and catch
>> bugs, several of which I found already while writing these tests) is a
>> good-thing (tm).
>>
>
> ++
>
>
>>
>> I'd welcome feedback on the patch above, and what will be the most
>> acceptable approach to the tempest team for adding these tests.
>>
>> More links:
>>
>> https://review.openstack.org/#/c/51559/
>> https://review.openstack.org/#/c/51560/
>> https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api
>>
>> Thanks!
>>
>> Steve
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
>
> -Dolph
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Dolph Mathews]

On Fri, Oct 18, 2013 at 10:34 AM, Steven Hardy  wrote:

> Hi all,
>
> Starting a thread to discuss $subject, as requested in:
>
> https://review.openstack.org/#/c/51558/
>
> First a bit of background.  I wrote a keystoneclient patch, and ayoung
> stated he'd like it tested via tempest before he'd ack it:
>
> https://review.openstack.org/#/c/48462/
>
> So I spoke to ayoung and dkranz on IRC, showing them my local tests for the
> patch.  dkranz suggested creating a "client_lib" directory, where we could
> build out a more comprehensive set of tests over time, adding to the inital
> tests related to keystone trusts client additions.
>
> A couple of things to note:
> - These are end-to-end tests, designed to test not only the client, but
>   also the API and keystone backend functionality.  So arguably this could
>   just be a scenario test, e.g scenario/keystone/test_v3_auth.py
>

I'd love to be able to run these tests against a wider variety of service
configurations (e.g. LDAP!), which tempest is obviously more suitable for.


>
> - The intention is to excercise logic which is hard to fully test with
>   unit or integration tests, and to catch issues like incompatibility
>   between client and API - e.g keystoneclient tests may pass, but we need
>   to make sure the client actually works against the real keystone API.
>

All of our tests under keystone.tests.test_keystoneclient fall into this
category as well:


https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient.py


https://github.com/openstack/keystone/blob/a0e26c1882d83989bee3726a5ae08cbe3f32a2b5/keystone/tests/test_keystoneclient_sql.py


>
> Working on Heat has given me a pretty good insight into the python-*client
> API's, as we use them to orchestrate actions with every openstack service;
> IMO anything we can do to make these interfaces more robust (and catch
> bugs, several of which I found already while writing these tests) is a
> good-thing (tm).
>

++


>
> I'd welcome feedback on the patch above, and what will be the most
> acceptable approach to the tempest team for adding these tests.
>
> More links:
>
> https://review.openstack.org/#/c/51559/
> https://review.openstack.org/#/c/51560/
> https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api
>
> Thanks!
>
> Steve
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 

-Dolph
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][keystone] Adding client library related tests to tempest

[Steven Hardy]

Hi all,

Starting a thread to discuss $subject, as requested in:

https://review.openstack.org/#/c/51558/

First a bit of background.  I wrote a keystoneclient patch, and ayoung
stated he'd like it tested via tempest before he'd ack it:

https://review.openstack.org/#/c/48462/

So I spoke to ayoung and dkranz on IRC, showing them my local tests for the
patch.  dkranz suggested creating a "client_lib" directory, where we could
build out a more comprehensive set of tests over time, adding to the inital
tests related to keystone trusts client additions.

A couple of things to note:
- These are end-to-end tests, designed to test not only the client, but
  also the API and keystone backend functionality.  So arguably this could
  just be a scenario test, e.g scenario/keystone/test_v3_auth.py

- The intention is to excercise logic which is hard to fully test with
  unit or integration tests, and to catch issues like incompatibility
  between client and API - e.g keystoneclient tests may pass, but we need
  to make sure the client actually works against the real keystone API.

Working on Heat has given me a pretty good insight into the python-*client
API's, as we use them to orchestrate actions with every openstack service;
IMO anything we can do to make these interfaces more robust (and catch
bugs, several of which I found already while writing these tests) is a
good-thing (tm).

I'd welcome feedback on the patch above, and what will be the most
acceptable approach to the tempest team for adding these tests.

More links:

https://review.openstack.org/#/c/51559/
https://review.openstack.org/#/c/51560/
https://blueprints.launchpad.net/tempest/+spec/keystoneclient-api

Thanks!

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev