Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread Eoghan Glynn


 Hi everyone,
 
 Before we start the larger discussion at summit next week about the future of
 testing in OpenStack - specifically about spinning up functional testing and
 how
 it relates to tempest - I would like to share some of my thoughts on how we
 can
 get things started and how I think they'll eventually come together.
 
 Currently in tempest we have a large number of tests (mostly api-focused)
 which are probably a better fit for a project's functional test suite. The
 best
 example I can think of is the nova flavors tests. Validation of flavor
 manipulation doesn't need to run in the integrated test suite on every commit
 to
 every project because it only requires Nova. A simple win for initiating
 in-tree
 functional testing would be to move these kinds of tests into the projects
 and
 run the tests from the project repos instead of from tempest.
 
 This would have the advantage of making tempest slimmer for every project
 and begin the process of getting projects to take responsibility for their
 functional testing rather than relying on tempest. As tests are moved tempest
 can start to become the integration test suite it was meant to be. It would
 retain only tests that involve multiple projects and stop being the OpenStack
 black box testing suite. I think that this is the right direction for tempest
 moving forward, especially as we move to having project-specific functional
 testing.
 
 Doing this migration is dependent on some refactors in tempest and moving
 the required bits to tempest-lib so they can be easily consumed by the
 other projects. This will be discussed at summit, is being planned
 for implementation this cycle, and is similar to what is currently in
 progress
 for the cli tests.
 
 The only reason this testing existed in tempest in the first place was as
 mechanism to block and then add friction against breaking api changes.
 Tempest's
 api testing has been been pretty successful at achieving these goals. We'll
 want
 to ensure that migrated tests retain these characteristics. If we are using
 clients from tempest-lib we should get this automatically since to break
 the api you'd have to change the api client. Another option proposed was to
 introduce a hacking rule that would block changes to api tests at the same
 time
 other code was being changed.
 
 There is also a concern for external consumers of tempest if we move the
 tests
 out of the tempest tree (I'm thinking refstack). I think the solution is
 to maintain a load_tests discovery method inside of tempest or elsewhere that
 will run the appropriate tests from the other repos for something like
 refstack.
 Assuming that things are built in a compatible way using the same framework
 then
 running the tests from separate repos should be a simple matter of pointing
 the
 test runner in the right direction.
 
 I also want to comment on the role of functional testing. What I've proposed
 here is only one piece of what project specific functional testing should be
 and just what I feel is a good/easy start. I don't feel that this should be
 the only testing done in the projects.  I'm suggesting this as a first
 step because the tests already exist and it should be a relatively simple
 task.
 I also feel that using tempest-lib like this shouldn't be a hard requirement.
 Ideally the client definitions shouldn't have to live externally, or if they
 did
 they would be the official clients, but I am suggesting this as a first step
 to
 start a migration out of tempest.
 
 I don't want anyone to feel that they need block their functional testing
 efforts until tempest-lib becomes more useable. The larger value from
 functional
 testing is actually in enabling testing more tightly coupled to the projects
 (e.g. whitebox testing). I feel that any work necessary to enable functional
 testing should be prioritized.

Thanks Matt for getting the ball rolling on this conversation in advance
of summit.

Probably stating the obvious here, but I feel we should make a concious
effort to keep the approaches to in-tree functional testing as consistent
as possible across projects.

Towards that end, it would be good for folks with an interest in this area
to attend each other's sessions where possible:
  
 Cross-project: Tue, 12:05 [1]
 Heat:  Wed, 13:50 [2]
 Nova:  Wed, 16:30 [3]
 Ceilometer:Wed, 17:20 [4]
 QA:Wed, 17:20 [5]

Unfortunately there's a clash there between the QA tempest-lib session
and the ceilo session. I'm not sure how reasonable it would be to make
a last-minute schedule change to avoid that clash.

Cheers,
Eoghan

[1] http://kilodesignsummit.sched.org/event/575938e4837e8293615845582d7e3e7f
[2] http://kilodesignsummit.sched.org/event/eb261fb08b18ec1eaa2c67492e7fc385
[3] http://kilodesignsummit.sched.org/event/271a9075c1ced6c1269100ff4b8efc37
[4] http://kilodesignsummit.sched.org/event/daf63526a1883e84cec107c70cc6cad3
[5] 

Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread Sean Dague
On 10/29/2014 12:30 PM, Matthew Treinish wrote:
 Hi everyone,

 Before we start the larger discussion at summit next week about the future of
 testing in OpenStack - specifically about spinning up functional testing and 
 how
 it relates to tempest - I would like to share some of my thoughts on how we 
 can
 get things started and how I think they'll eventually come together.

 Currently in tempest we have a large number of tests (mostly api-focused)
 which are probably a better fit for a project's functional test suite. The 
 best
 example I can think of is the nova flavors tests. Validation of flavor
 manipulation doesn't need to run in the integrated test suite on every commit 
 to
 every project because it only requires Nova. A simple win for initiating 
 in-tree
 functional testing would be to move these kinds of tests into the projects and
 run the tests from the project repos instead of from tempest.
I think a lot of the negative API testing is also a great thing to be
done back at the project level. All of that testing should be able to
work without a full OpenStack, as it should be caught and managed by the
API service and never get any further than that.


 This would have the advantage of making tempest slimmer for every project
 and begin the process of getting projects to take responsibility for their
 functional testing rather than relying on tempest. As tests are moved tempest
 can start to become the integration test suite it was meant to be. It would
 retain only tests that involve multiple projects and stop being the OpenStack
 black box testing suite. I think that this is the right direction for tempest
 moving forward, especially as we move to having project-specific functional
 testing.

 Doing this migration is dependent on some refactors in tempest and moving
 the required bits to tempest-lib so they can be easily consumed by the
 other projects. This will be discussed at summit, is being planned
 for implementation this cycle, and is similar to what is currently in progress
 for the cli tests.

 The only reason this testing existed in tempest in the first place was as
 mechanism to block and then add friction against breaking api changes. 
 Tempest's
 api testing has been been pretty successful at achieving these goals. We'll 
 want
 to ensure that migrated tests retain these characteristics. If we are using
 clients from tempest-lib we should get this automatically since to break
 the api you'd have to change the api client. Another option proposed was to
 introduce a hacking rule that would block changes to api tests at the same 
 time
 other code was being changed.

 There is also a concern for external consumers of tempest if we move the tests
 out of the tempest tree (I'm thinking refstack). I think the solution is
 to maintain a load_tests discovery method inside of tempest or elsewhere that
 will run the appropriate tests from the other repos for something like 
 refstack.
 Assuming that things are built in a compatible way using the same framework 
 then
 running the tests from separate repos should be a simple matter of pointing 
 the
 test runner in the right direction.
I think we can see where this takes us. I'm still skeptical of cross
project loading of tests because it's often quite fragile. However, if
you look at what refstack did they had a giant evaluation of all of
tempest and pruned a bunch of stuff out. I would imagine maybe there is
a conversation there about tests that refstack feels are important to
stay in Tempest for their validation reasons. I think having a few paths
that are tested both in Tempest and in project functional tests is not a
bad thing.

But I think that's an end of cycle at best discussion.

Also, there probably need to be a few discussions anyway of
refstack/tempest/defcore. The fact that Keystone was dropped from
defcore because there were no non admin Keystone tests explicitly in
Tempest (even though we make over 5000 keystone non admin API calls over
a tempest run) was very odd. That is something that could have been
fixed in a day.


 I also want to comment on the role of functional testing. What I've proposed
 here is only one piece of what project specific functional testing should be
 and just what I feel is a good/easy start. I don't feel that this should be
 the only testing done in the projects.  I'm suggesting this as a first
 step because the tests already exist and it should be a relatively simple 
 task.
 I also feel that using tempest-lib like this shouldn't be a hard requirement.
 Ideally the client definitions shouldn't have to live externally, or if they 
 did
 they would be the official clients, but I am suggesting this as a first step 
 to
 start a migration out of tempest.

 I don't want anyone to feel that they need block their functional testing
 efforts until tempest-lib becomes more useable. The larger value from 
 functional
 testing is actually in enabling testing more tightly coupled to the projects
 (e.g. 

Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread David Kranz

On 10/30/2014 07:49 AM, Sean Dague wrote:

On 10/29/2014 12:30 PM, Matthew Treinish wrote:

Hi everyone,

Before we start the larger discussion at summit next week about the future of
testing in OpenStack - specifically about spinning up functional testing and how
it relates to tempest - I would like to share some of my thoughts on how we can
get things started and how I think they'll eventually come together.

Currently in tempest we have a large number of tests (mostly api-focused)
which are probably a better fit for a project's functional test suite. The best
example I can think of is the nova flavors tests. Validation of flavor
manipulation doesn't need to run in the integrated test suite on every commit to
every project because it only requires Nova. A simple win for initiating in-tree
functional testing would be to move these kinds of tests into the projects and
run the tests from the project repos instead of from tempest.
I think a lot of the negative API testing is also a great thing to be 
done back at the project level. All of that testing should be able to 
work without a full OpenStack, as it should be caught and managed by 
the API service and never get any further than that.



This would have the advantage of making tempest slimmer for every project
and begin the process of getting projects to take responsibility for their
functional testing rather than relying on tempest. As tests are moved tempest
can start to become the integration test suite it was meant to be. It would
retain only tests that involve multiple projects and stop being the OpenStack
black box testing suite. I think that this is the right direction for tempest
moving forward, especially as we move to having project-specific functional
testing.

Doing this migration is dependent on some refactors in tempest and moving
the required bits to tempest-lib so they can be easily consumed by the
other projects. This will be discussed at summit, is being planned
for implementation this cycle, and is similar to what is currently in progress
for the cli tests.

The only reason this testing existed in tempest in the first place was as
mechanism to block and then add friction against breaking api changes. Tempest's
api testing has been been pretty successful at achieving these goals. We'll want
to ensure that migrated tests retain these characteristics. If we are using
clients from tempest-lib we should get this automatically since to break
the api you'd have to change the api client. Another option proposed was to
introduce a hacking rule that would block changes to api tests at the same time
other code was being changed.

There is also a concern for external consumers of tempest if we move the tests
out of the tempest tree (I'm thinking refstack). I think the solution is
to maintain a load_tests discovery method inside of tempest or elsewhere that
will run the appropriate tests from the other repos for something like refstack.
Assuming that things are built in a compatible way using the same framework then
running the tests from separate repos should be a simple matter of pointing the
test runner in the right direction.
I think we can see where this takes us. I'm still skeptical of cross 
project loading of tests because it's often quite fragile. However, if 
you look at what refstack did they had a giant evaluation of all of 
tempest and pruned a bunch of stuff out. I would imagine maybe there 
is a conversation there about tests that refstack feels are important 
to stay in Tempest for their validation reasons. I think having a few 
paths that are tested both in Tempest and in project functional tests 
is not a bad thing.
Refstack is not the only thing that cares about validation of real 
clouds. As we move forward with this, it would be good to separate the 
issues of in which repo does a functional test live and can a 
functional test be run against a real cloud. IMO, over use of mocking 
(broadly defined) in functional tests should be avoided unless it is 
configurable to also work in an unmocked fashion. Whether the way to 
combine all of the functional tests is by cross project loading of tests 
or by some other means is more of an implementation detail.


But I think that's an end of cycle at best discussion.

Also, there probably need to be a few discussions anyway of 
refstack/tempest/defcore. The fact that Keystone was dropped from 
defcore because there were no non admin Keystone tests explicitly in 
Tempest (even though we make over 5000 keystone non admin API calls 
over a tempest run) was very odd. That is something that could have 
been fixed in a day.



I also want to comment on the role of functional testing. What I've proposed
here is only one piece of what project specific functional testing should be
and just what I feel is a good/easy start. I don't feel that this should be
the only testing done in the projects.  I'm suggesting this as a first
step because the tests already exist and it should be a relatively 

Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread Sean Dague
On 10/30/2014 09:33 AM, David Kranz wrote:
 On 10/30/2014 07:49 AM, Sean Dague wrote:
 On 10/29/2014 12:30 PM, Matthew Treinish wrote:
 Hi everyone,

 Before we start the larger discussion at summit next week about the future 
 of
 testing in OpenStack - specifically about spinning up functional testing 
 and how
 it relates to tempest - I would like to share some of my thoughts on how we 
 can
 get things started and how I think they'll eventually come together.

 Currently in tempest we have a large number of tests (mostly api-focused)
 which are probably a better fit for a project's functional test suite. The 
 best
 example I can think of is the nova flavors tests. Validation of flavor
 manipulation doesn't need to run in the integrated test suite on every 
 commit to
 every project because it only requires Nova. A simple win for initiating 
 in-tree
 functional testing would be to move these kinds of tests into the projects 
 and
 run the tests from the project repos instead of from tempest.
 I think a lot of the negative API testing is also a great thing to be
 done back at the project level. All of that testing should be able to
 work without a full OpenStack, as it should be caught and managed by
 the API service and never get any further than that.

 This would have the advantage of making tempest slimmer for every project
 and begin the process of getting projects to take responsibility for their
 functional testing rather than relying on tempest. As tests are moved 
 tempest
 can start to become the integration test suite it was meant to be. It would
 retain only tests that involve multiple projects and stop being the 
 OpenStack
 black box testing suite. I think that this is the right direction for 
 tempest
 moving forward, especially as we move to having project-specific functional
 testing.

 Doing this migration is dependent on some refactors in tempest and moving
 the required bits to tempest-lib so they can be easily consumed by the
 other projects. This will be discussed at summit, is being planned
 for implementation this cycle, and is similar to what is currently in 
 progress
 for the cli tests.

 The only reason this testing existed in tempest in the first place was as
 mechanism to block and then add friction against breaking api changes. 
 Tempest's
 api testing has been been pretty successful at achieving these goals. We'll 
 want
 to ensure that migrated tests retain these characteristics. If we are using
 clients from tempest-lib we should get this automatically since to break
 the api you'd have to change the api client. Another option proposed was to
 introduce a hacking rule that would block changes to api tests at the same 
 time
 other code was being changed.

 There is also a concern for external consumers of tempest if we move the 
 tests
 out of the tempest tree (I'm thinking refstack). I think the solution is
 to maintain a load_tests discovery method inside of tempest or elsewhere 
 that
 will run the appropriate tests from the other repos for something like 
 refstack.
 Assuming that things are built in a compatible way using the same framework 
 then
 running the tests from separate repos should be a simple matter of pointing 
 the
 test runner in the right direction.
 I think we can see where this takes us. I'm still skeptical of cross
 project loading of tests because it's often quite fragile. However,
 if you look at what refstack did they had a giant evaluation of all
 of tempest and pruned a bunch of stuff out. I would imagine maybe
 there is a conversation there about tests that refstack feels are
 important to stay in Tempest for their validation reasons. I think
 having a few paths that are tested both in Tempest and in project
 functional tests is not a bad thing.
 Refstack is not the only thing that cares about validation of real
 clouds. As we move forward with this, it would be good to separate the
 issues of in which repo does a functional test live and can a
 functional test be run against a real cloud. IMO, over use of mocking
 (broadly defined) in functional tests should be avoided unless it is
 configurable to also work in an unmocked fashion. Whether the way to
 combine all of the functional tests is by cross project loading of
 tests or by some other means is more of an implementation detail.
Part of the perspective I'm bringing in is actually knowing what to do
when your tests fail. Using Tempest against real clouds is great, people
should keep doing that. But if you are rolling out a real cloud
yourself, in the future you should be running the functional tests in
staging to ensure you are functioning. Those will also provide you,
hopefully, with a better path to understand what's wrong.

This will mean that as an arbitrary 3rd party accessing a public cloud,
you don't have a test suite that pushes every button of the cloud. But I
think that's an acceptable trade off. Because pushing on odd edge cases
might produce a fail, but if there is no 

Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread David Kranz

On 10/30/2014 09:52 AM, Sean Dague wrote:

On 10/30/2014 09:33 AM, David Kranz wrote:

On 10/30/2014 07:49 AM, Sean Dague wrote:

On 10/29/2014 12:30 PM, Matthew Treinish wrote:

Hi everyone,

Before we start the larger discussion at summit next week about the future of
testing in OpenStack - specifically about spinning up functional testing and how
it relates to tempest - I would like to share some of my thoughts on how we can
get things started and how I think they'll eventually come together.

Currently in tempest we have a large number of tests (mostly api-focused)
which are probably a better fit for a project's functional test suite. The best
example I can think of is the nova flavors tests. Validation of flavor
manipulation doesn't need to run in the integrated test suite on every commit to
every project because it only requires Nova. A simple win for initiating in-tree
functional testing would be to move these kinds of tests into the projects and
run the tests from the project repos instead of from tempest.
I think a lot of the negative API testing is also a great thing to 
be done back at the project level. All of that testing should be 
able to work without a full OpenStack, as it should be caught and 
managed by the API service and never get any further than that.



This would have the advantage of making tempest slimmer for every project
and begin the process of getting projects to take responsibility for their
functional testing rather than relying on tempest. As tests are moved tempest
can start to become the integration test suite it was meant to be. It would
retain only tests that involve multiple projects and stop being the OpenStack
black box testing suite. I think that this is the right direction for tempest
moving forward, especially as we move to having project-specific functional
testing.

Doing this migration is dependent on some refactors in tempest and moving
the required bits to tempest-lib so they can be easily consumed by the
other projects. This will be discussed at summit, is being planned
for implementation this cycle, and is similar to what is currently in progress
for the cli tests.

The only reason this testing existed in tempest in the first place was as
mechanism to block and then add friction against breaking api changes. Tempest's
api testing has been been pretty successful at achieving these goals. We'll want
to ensure that migrated tests retain these characteristics. If we are using
clients from tempest-lib we should get this automatically since to break
the api you'd have to change the api client. Another option proposed was to
introduce a hacking rule that would block changes to api tests at the same time
other code was being changed.

There is also a concern for external consumers of tempest if we move the tests
out of the tempest tree (I'm thinking refstack). I think the solution is
to maintain a load_tests discovery method inside of tempest or elsewhere that
will run the appropriate tests from the other repos for something like refstack.
Assuming that things are built in a compatible way using the same framework then
running the tests from separate repos should be a simple matter of pointing the
test runner in the right direction.
I think we can see where this takes us. I'm still skeptical of cross 
project loading of tests because it's often quite fragile. However, 
if you look at what refstack did they had a giant evaluation of all 
of tempest and pruned a bunch of stuff out. I would imagine maybe 
there is a conversation there about tests that refstack feels are 
important to stay in Tempest for their validation reasons. I think 
having a few paths that are tested both in Tempest and in project 
functional tests is not a bad thing.
Refstack is not the only thing that cares about validation of real 
clouds. As we move forward with this, it would be good to separate 
the issues of in which repo does a functional test live and can a 
functional test be run against a real cloud. IMO, over use of 
mocking (broadly defined) in functional tests should be avoided 
unless it is configurable to also work in an unmocked fashion. 
Whether the way to combine all of the functional tests is by cross 
project loading of tests or by some other means is more of an 
implementation detail.
Part of the perspective I'm bringing in is actually knowing what to do 
when your tests fail. Using Tempest against real clouds is great, 
people should keep doing that. But if you are rolling out a real cloud 
yourself, in the future you should be running the functional tests in 
staging to ensure you are functioning. Those will also provide you, 
hopefully, with a better path to understand what's wrong.
Sean, sorry if I was unclear. By real clouds, I just meant the tests 
should be able to use  OpenStack apis with no mocking.


 -David





This will mean that as an arbitrary 3rd party accessing a public 
cloud, you don't have a test suite that pushes every button of the 
cloud. But I 

Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread Sean Dague
On 10/30/2014 10:02 AM, David Kranz wrote:
 On 10/30/2014 09:52 AM, Sean Dague wrote:
 On 10/30/2014 09:33 AM, David Kranz wrote:
 On 10/30/2014 07:49 AM, Sean Dague wrote:
 On 10/29/2014 12:30 PM, Matthew Treinish wrote:
 Hi everyone,

 Before we start the larger discussion at summit next week about the 
 future of
 testing in OpenStack - specifically about spinning up functional testing 
 and how
 it relates to tempest - I would like to share some of my thoughts on how 
 we can
 get things started and how I think they'll eventually come together.

 Currently in tempest we have a large number of tests (mostly api-focused)
 which are probably a better fit for a project's functional test suite. 
 The best
 example I can think of is the nova flavors tests. Validation of flavor
 manipulation doesn't need to run in the integrated test suite on every 
 commit to
 every project because it only requires Nova. A simple win for initiating 
 in-tree
 functional testing would be to move these kinds of tests into the 
 projects and
 run the tests from the project repos instead of from tempest.
 I think a lot of the negative API testing is also a great thing to
 be done back at the project level. All of that testing should be
 able to work without a full OpenStack, as it should be caught and
 managed by the API service and never get any further than that.

 This would have the advantage of making tempest slimmer for every project
 and begin the process of getting projects to take responsibility for their
 functional testing rather than relying on tempest. As tests are moved 
 tempest
 can start to become the integration test suite it was meant to be. It 
 would
 retain only tests that involve multiple projects and stop being the 
 OpenStack
 black box testing suite. I think that this is the right direction for 
 tempest
 moving forward, especially as we move to having project-specific 
 functional
 testing.

 Doing this migration is dependent on some refactors in tempest and moving
 the required bits to tempest-lib so they can be easily consumed by the
 other projects. This will be discussed at summit, is being planned
 for implementation this cycle, and is similar to what is currently in 
 progress
 for the cli tests.

 The only reason this testing existed in tempest in the first place was as
 mechanism to block and then add friction against breaking api changes. 
 Tempest's
 api testing has been been pretty successful at achieving these goals. 
 We'll want
 to ensure that migrated tests retain these characteristics. If we are 
 using
 clients from tempest-lib we should get this automatically since to break
 the api you'd have to change the api client. Another option proposed was 
 to
 introduce a hacking rule that would block changes to api tests at the 
 same time
 other code was being changed.

 There is also a concern for external consumers of tempest if we move the 
 tests
 out of the tempest tree (I'm thinking refstack). I think the solution is
 to maintain a load_tests discovery method inside of tempest or elsewhere 
 that
 will run the appropriate tests from the other repos for something like 
 refstack.
 Assuming that things are built in a compatible way using the same 
 framework then
 running the tests from separate repos should be a simple matter of 
 pointing the
 test runner in the right direction.
 I think we can see where this takes us. I'm still skeptical of
 cross project loading of tests because it's often quite fragile.
 However, if you look at what refstack did they had a giant
 evaluation of all of tempest and pruned a bunch of stuff out. I
 would imagine maybe there is a conversation there about tests that
 refstack feels are important to stay in Tempest for their
 validation reasons. I think having a few paths that are tested both
 in Tempest and in project functional tests is not a bad thing.
 Refstack is not the only thing that cares about validation of real
 clouds. As we move forward with this, it would be good to separate
 the issues of in which repo does a functional test live and can a
 functional test be run against a real cloud. IMO, over use of
 mocking (broadly defined) in functional tests should be avoided
 unless it is configurable to also work in an unmocked fashion.
 Whether the way to combine all of the functional tests is by cross
 project loading of tests or by some other means is more of an
 implementation detail.
 Part of the perspective I'm bringing in is actually knowing what to
 do when your tests fail. Using Tempest against real clouds is great,
 people should keep doing that. But if you are rolling out a real
 cloud yourself, in the future you should be running the functional
 tests in staging to ensure you are functioning. Those will also
 provide you, hopefully, with a better path to understand what's wrong.
 Sean, sorry if I was unclear. By real clouds, I just meant the tests
 should be able to use  OpenStack apis with no mocking.
Ok, so part of this remains to be 

Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread Lance Bragstad
On Thu, Oct 30, 2014 at 6:30 AM, Eoghan Glynn egl...@redhat.com wrote:



  Hi everyone,
 
  Before we start the larger discussion at summit next week about the
 future of
  testing in OpenStack - specifically about spinning up functional testing
 and
  how
  it relates to tempest - I would like to share some of my thoughts on how
 we
  can
  get things started and how I think they'll eventually come together.
 
  Currently in tempest we have a large number of tests (mostly api-focused)
  which are probably a better fit for a project's functional test suite.
 The
  best
  example I can think of is the nova flavors tests. Validation of flavor
  manipulation doesn't need to run in the integrated test suite on every
 commit
  to
  every project because it only requires Nova. A simple win for initiating
  in-tree
  functional testing would be to move these kinds of tests into the
 projects
  and
  run the tests from the project repos instead of from tempest.
 
  This would have the advantage of making tempest slimmer for every project
  and begin the process of getting projects to take responsibility for
 their
  functional testing rather than relying on tempest. As tests are moved
 tempest
  can start to become the integration test suite it was meant to be. It
 would
  retain only tests that involve multiple projects and stop being the
 OpenStack
  black box testing suite. I think that this is the right direction for
 tempest
  moving forward, especially as we move to having project-specific
 functional
  testing.
 
  Doing this migration is dependent on some refactors in tempest and moving
  the required bits to tempest-lib so they can be easily consumed by the
  other projects. This will be discussed at summit, is being planned
  for implementation this cycle, and is similar to what is currently in
  progress
  for the cli tests.
 
  The only reason this testing existed in tempest in the first place was as
  mechanism to block and then add friction against breaking api changes.
  Tempest's
  api testing has been been pretty successful at achieving these goals.
 We'll
  want
  to ensure that migrated tests retain these characteristics. If we are
 using
  clients from tempest-lib we should get this automatically since to break
  the api you'd have to change the api client. Another option proposed was
 to
  introduce a hacking rule that would block changes to api tests at the
 same
  time
  other code was being changed.
 
  There is also a concern for external consumers of tempest if we move the
  tests
  out of the tempest tree (I'm thinking refstack). I think the solution is
  to maintain a load_tests discovery method inside of tempest or elsewhere
 that
  will run the appropriate tests from the other repos for something like
  refstack.
  Assuming that things are built in a compatible way using the same
 framework
  then
  running the tests from separate repos should be a simple matter of
 pointing
  the
  test runner in the right direction.
 
  I also want to comment on the role of functional testing. What I've
 proposed
  here is only one piece of what project specific functional testing
 should be
  and just what I feel is a good/easy start. I don't feel that this should
 be
  the only testing done in the projects.  I'm suggesting this as a first
  step because the tests already exist and it should be a relatively simple
  task.
  I also feel that using tempest-lib like this shouldn't be a hard
 requirement.
  Ideally the client definitions shouldn't have to live externally, or if
 they
  did
  they would be the official clients, but I am suggesting this as a first
 step
  to
  start a migration out of tempest.
 
  I don't want anyone to feel that they need block their functional testing
  efforts until tempest-lib becomes more useable. The larger value from
  functional
  testing is actually in enabling testing more tightly coupled to the
 projects
  (e.g. whitebox testing). I feel that any work necessary to enable
 functional
  testing should be prioritized.

 Thanks Matt for getting the ball rolling on this conversation in advance
 of summit.

 Probably stating the obvious here, but I feel we should make a concious
 effort to keep the approaches to in-tree functional testing as consistent
 as possible across projects.

 Towards that end, it would be good for folks with an interest in this area
 to attend each other's sessions where possible:


+1


  Cross-project: Tue, 12:05 [1]
  Heat:  Wed, 13:50 [2]
  Nova:  Wed, 16:30 [3]
  Ceilometer:Wed, 17:20 [4]
  QA:Wed, 17:20 [5]


I think Keystone was planning on dedicating some time to this on Friday, so
our dev/hackathon day. I'm
interested in the process that will be in place for tracking status of the
functional test migration if there will be one.


 Unfortunately there's a clash there between the QA tempest-lib session
 and the ceilo session. I'm not sure how reasonable it would be to make
 a last-minute schedule 

Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread Eoghan Glynn

 Matthew wrote:

 This would have the advantage of making tempest slimmer for every project
 and begin the process of getting projects to take responsibility for their
 functional testing rather than relying on tempest.

[much snipping]

 Sean wrote:

 Ok, so part of this remains to be seen about what the biggest bang for the
 buck is. The class of bugs I feel like we need to nail in Nova right now are
 going to require tests that bring up pieces of the wsgi stack, but are
 probably not runable on a real deploy. Again, this is about debugability.

So this notion of the biggest bang for our buck is an aspect of the drive
for in-tree functional tests, that's not entirely clear to me as yet.

i.e. whether individual projects should be prioritizing within this effort:

(a) the creation of net-new coverage for scenarios (especially known or
suspected bugs) that were not previously tested, in a non-unit sense

(b) the relocation of existing integration test coverage from Tempest to
the project trees, in order to make the management of Tempest more
tractable

It feels like there may be a tension between (a) and (b) in terms of the
pay-off for this effort. I'd interested in hearing other opinions on this,
on what aspect projects are expecting (and expected) to concentrate on
initially.

Cheers,
Eoghan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread Sean Dague
On 10/30/2014 10:47 AM, Eoghan Glynn wrote:
 
 Matthew wrote:

 This would have the advantage of making tempest slimmer for every project
 and begin the process of getting projects to take responsibility for their
 functional testing rather than relying on tempest.
 
 [much snipping]
 
 Sean wrote:

 Ok, so part of this remains to be seen about what the biggest bang for the
 buck is. The class of bugs I feel like we need to nail in Nova right now are
 going to require tests that bring up pieces of the wsgi stack, but are
 probably not runable on a real deploy. Again, this is about debugability.
 
 So this notion of the biggest bang for our buck is an aspect of the drive
 for in-tree functional tests, that's not entirely clear to me as yet.
 
 i.e. whether individual projects should be prioritizing within this effort:
 
 (a) the creation of net-new coverage for scenarios (especially known or
 suspected bugs) that were not previously tested, in a non-unit sense
 
 (b) the relocation of existing integration test coverage from Tempest to
 the project trees, in order to make the management of Tempest more
 tractable
 
 It feels like there may be a tension between (a) and (b) in terms of the
 pay-off for this effort. I'd interested in hearing other opinions on this,
 on what aspect projects are expecting (and expected) to concentrate on
 initially.

For what it's worth I have a bunch of early targets listed for Nova for
our summit session -
https://etherpad.openstack.org/p/kilo-nova-functional-testing

My focus in kilo is going to be first about A), as that provides value
out of the gate (pun intended). Then peel off some stuff from B as makes
sense.

-Sean

-- 
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-30 Thread David Kranz

On 10/30/2014 11:12 AM, Sean Dague wrote:

On 10/30/2014 10:47 AM, Eoghan Glynn wrote:

Matthew wrote:

This would have the advantage of making tempest slimmer for every project
and begin the process of getting projects to take responsibility for their
functional testing rather than relying on tempest.

[much snipping]


Sean wrote:

Ok, so part of this remains to be seen about what the biggest bang for the
buck is. The class of bugs I feel like we need to nail in Nova right now are
going to require tests that bring up pieces of the wsgi stack, but are
probably not runable on a real deploy. Again, this is about debugability.

So this notion of the biggest bang for our buck is an aspect of the drive
for in-tree functional tests, that's not entirely clear to me as yet.

i.e. whether individual projects should be prioritizing within this effort:

(a) the creation of net-new coverage for scenarios (especially known or
 suspected bugs) that were not previously tested, in a non-unit sense

(b) the relocation of existing integration test coverage from Tempest to
 the project trees, in order to make the management of Tempest more
 tractable

It feels like there may be a tension between (a) and (b) in terms of the
pay-off for this effort. I'd interested in hearing other opinions on this,
on what aspect projects are expecting (and expected) to concentrate on
initially.

For what it's worth I have a bunch of early targets listed for Nova for
our summit session -
https://etherpad.openstack.org/p/kilo-nova-functional-testing

My focus in kilo is going to be first about A), as that provides value
out of the gate (pun intended). Then peel off some stuff from B as makes
sense.

-Sean

That seems sensible from the nova point of view and overall health, and 
not all projects have to pursue the same priorities at the same time. 
But a big part of the benefit of (b) is the impact it has on all the 
other projects in that other projects will stop getting as many gate 
failures, and that benefit could be achieved right now by simply 
changing the set of tempest tests that run against each project.


 -David

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [QA][All] Prelude to functional testing summit discussions

2014-10-29 Thread Ken'ichi Ohmichi
Hi Matt,

Thanks for bringing this up, I am so interested in.

2014-10-30 1:30 GMT+09:00 Matthew Treinish mtrein...@kortar.org:

 Before we start the larger discussion at summit next week about the future of
 testing in OpenStack - specifically about spinning up functional testing and 
 how
 it relates to tempest - I would like to share some of my thoughts on how we 
 can
 get things started and how I think they'll eventually come together.

 Currently in tempest we have a large number of tests (mostly api-focused)
 which are probably a better fit for a project's functional test suite. The 
 best
 example I can think of is the nova flavors tests. Validation of flavor
 manipulation doesn't need to run in the integrated test suite on every commit 
 to
 every project because it only requires Nova. A simple win for initiating 
 in-tree
 functional testing would be to move these kinds of tests into the projects and
 run the tests from the project repos instead of from tempest.

I completely agree with the above comment.
There is a lot of hard-coded negative tests in Tempest and most
projects block these negative cases at their surfaces, these cases
don't seem integrated tests.
In addition, sometimes there are more corner cases in Tempest than each project.
So it would be very nice to move this kind of tests from Tempest to
each project and review them on the project.

 This would have the advantage of making tempest slimmer for every project
 and begin the process of getting projects to take responsibility for their
 functional testing rather than relying on tempest. As tests are moved tempest
 can start to become the integration test suite it was meant to be. It would
 retain only tests that involve multiple projects and stop being the OpenStack
 black box testing suite. I think that this is the right direction for tempest
 moving forward, especially as we move to having project-specific functional
 testing.

+1

 Doing this migration is dependent on some refactors in tempest and moving
 the required bits to tempest-lib so they can be easily consumed by the
 other projects. This will be discussed at summit, is being planned
 for implementation this cycle, and is similar to what is currently in progress
 for the cli tests.

 The only reason this testing existed in tempest in the first place was as
 mechanism to block and then add friction against breaking api changes. 
 Tempest's
 api testing has been been pretty successful at achieving these goals. We'll 
 want
 to ensure that migrated tests retain these characteristics. If we are using
 clients from tempest-lib we should get this automatically since to break
 the api you'd have to change the api client. Another option proposed was to
 introduce a hacking rule that would block changes to api tests at the same 
 time
 other code was being changed.

 There is also a concern for external consumers of tempest if we move the tests
 out of the tempest tree (I'm thinking refstack). I think the solution is
 to maintain a load_tests discovery method inside of tempest or elsewhere that
 will run the appropriate tests from the other repos for something like 
 refstack.
 Assuming that things are built in a compatible way using the same framework 
 then
 running the tests from separate repos should be a simple matter of pointing 
 the
 test runner in the right direction.

 I also want to comment on the role of functional testing. What I've proposed
 here is only one piece of what project specific functional testing should be
 and just what I feel is a good/easy start. I don't feel that this should be
 the only testing done in the projects.  I'm suggesting this as a first
 step because the tests already exist and it should be a relatively simple 
 task.
 I also feel that using tempest-lib like this shouldn't be a hard requirement.
 Ideally the client definitions shouldn't have to live externally, or if they 
 did
 they would be the official clients, but I am suggesting this as a first step 
 to
 start a migration out of tempest.

+1 for moving functional tests which means non-cross projects tests to
each project repo.
I like to jump into this work, and I want to move negative tests to
each project ASAP.

Thanks
Ken Ohmichi

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev