Re: [openstack-dev] [nova] novaclient functional test guidelines

2015-03-06 Thread Joe Gordon
First pass at trying to capture this thread into a README:
https://review.openstack.org/162334

On Tue, Feb 24, 2015 at 2:07 PM, Joe Gordon joe.gord...@gmail.com wrote:



 On Tue, Feb 24, 2015 at 1:18 PM, melanie witt melwi...@gmail.com wrote:

 On Feb 24, 2015, at 9:47, Sean Dague s...@dague.net wrote:

  I'm happy if there are other theories about how we do these things,
  being the first functional test in the python-novaclient tree that
  creates and destroys real resources, there isn't an established pattern
  yet. But I think doing all CLI calls in CLI tests is actually really
  cumbersome, especially in the amount of output parsing code needed if
  you are going to setup any complicated resource structure.

 I think I'm in agreement with the pattern you describe.

 I imagine having a set of functional tests for the API, that don't do any
 CLI calls at all. With that we test that the API works properly. Then have
 a separate set for the CLI, which only calls CLI for the command being
 tested, everything else to set up and tear down the test done by API calls.
 This would be done with the rationale that because the entire API
 functionality is tested separately, we can safely use it for setup/teardown
 with the intent to isolate the CLI test to the command being tested and
 avoid introducing side effects from the CLI commands.

 But I suppose one could make the same argument for using CLI everywhere
 (if they are all tested, they can also be trusted not to introduce side
 effects). I tend to favor using the API because it's the most bare bones
 setup/teardown we could use. At the same time I understand the idea of
 performing an entire test using the CLI, as a way of replicating the
 experience a real user might have using the CLI, from start to end. I don't
 think I feel strongly either way.


  I guess its time to revisit the actual status of novaclient and if we
 want to actively move away from it to openstacksdk/OSC as well. If we are
 actively trying to move away from novaclient, using the python API as much
 as possible makes a lot of sense.




 For the --poll stuff, I agree the API should have it and the CLI uses it.
 And with and without poll functionality should be tested separately, API
 and CLI.

 melanie (melwitt)





 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] novaclient functional test guidelines

2015-02-24 Thread melanie witt
On Feb 24, 2015, at 9:47, Sean Dague s...@dague.net wrote:

 I'm happy if there are other theories about how we do these things,
 being the first functional test in the python-novaclient tree that
 creates and destroys real resources, there isn't an established pattern
 yet. But I think doing all CLI calls in CLI tests is actually really
 cumbersome, especially in the amount of output parsing code needed if
 you are going to setup any complicated resource structure.

I think I'm in agreement with the pattern you describe.

I imagine having a set of functional tests for the API, that don't do any CLI 
calls at all. With that we test that the API works properly. Then have a 
separate set for the CLI, which only calls CLI for the command being tested, 
everything else to set up and tear down the test done by API calls. This would 
be done with the rationale that because the entire API functionality is tested 
separately, we can safely use it for setup/teardown with the intent to isolate 
the CLI test to the command being tested and avoid introducing side effects 
from the CLI commands.

But I suppose one could make the same argument for using CLI everywhere (if 
they are all tested, they can also be trusted not to introduce side effects). I 
tend to favor using the API because it's the most bare bones setup/teardown we 
could use. At the same time I understand the idea of performing an entire test 
using the CLI, as a way of replicating the experience a real user might have 
using the CLI, from start to end. I don't think I feel strongly either way.

For the --poll stuff, I agree the API should have it and the CLI uses it. And 
with and without poll functionality should be tested separately, API and CLI.

melanie (melwitt)






signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] novaclient functional test guidelines

2015-02-24 Thread Ed Leafe
On Feb 24, 2015, at 2:30 PM, Sean Dague s...@dague.net wrote:

 Right, I think to some degree novaclient is legacy code, and we should
 focus on specific regressions and bugs without doing to much code change.
 
 The future should be more focussed on openstacksdk and openstackclient.

IMO, openstackclient has an impossible task: taking the varied (and flawed) CLI 
clients and unite them under a single CLI interface. It is better to create the 
clean separation of the API wrapping into a Python library, and keep the CLI 
completely separate.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] novaclient functional test guidelines

2015-02-24 Thread Sean Dague
On 02/24/2015 03:28 PM, Ed Leafe wrote:
 On Feb 24, 2015, at 1:49 PM, Sean Dague s...@dague.net wrote:
 
 IMHO the CLI should have an option to returned raw JSON back instead of
 pretty tabled results as well.

 Um... isn't that just the API calls?

 I'm not sure creating a 3rd functional surface is really the answer
 here, because we still need to actually test the CLI / pretty table output.
 
 The python-openstacksdk project was originally envisioned to wrap the API 
 calls and return usable Python objects. The nova client CLI (or any other 
 CLI, for that matter) would then just provide the command line input parsing 
 and output presentation. It's been a while since I was involved with that 
 project, but it seems that decoupling the command line interface from the 
 Python API wrapper would make testing much, much easier.

Right, I think to some degree novaclient is legacy code, and we should
focus on specific regressions and bugs without doing to much code change.

The future should be more focussed on openstacksdk and openstackclient.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] novaclient functional test guidelines

2015-02-24 Thread Sean Dague
On 02/24/2015 01:47 PM, Joe Gordon wrote:
 
 
 On Tue, Feb 24, 2015 at 9:47 AM, Sean Dague s...@dague.net
 mailto:s...@dague.net wrote:
 
 Towards the end of merging the regression test for the nova
 volume-attach bug - https://review.openstack.org/#/c/157959/ there was a
 discussion around what style the functional tests should take.
 Especially as that had a mix of CLI and API calls in it.
 
 
 
 Thanks for starting this thread.  Once we reach general agreement lets
 put this in a in tree README for record keeping.

Absolutely.

 Here are my thoughts for why that test ended up that way:
 
 1) All resource setup that is table stakes for the test should be done
 via the API, regardless if it's a CLI or API test.
 
 The reason for this is that structured data is returned, which removes
 one possible error in the tests by parsing incorrectly. The API objects
 returned also include things like .delete methods in most cases, which
 means that addCleanup is a little more clean.
 
 
 IMHO the CLI should have an option to returned raw JSON back instead of
 pretty tabled results as well.

Um... isn't that just the API calls?

I'm not sure creating a 3rd functional surface is really the answer
here, because we still need to actually test the CLI / pretty table output.

 2) Main logic should touch which ever interface you are trying to test.
 This was demonstrating a CLI regression, so the volume-attach call was
 important to be done over the CLI.
 
 
 Now... here's where theory runs into issues.
 
 #1 - nova boot is table stakes. Under the above guidelines it should be
 called via API. However --poll is a CLI construct and so saved a custom
 wait loop here. We should implement that custom wait loop down the road
 and make that an API call
 
 
 https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L116
 
 
 This issue is from a real short coming in the python client. So this
 isn't really an issue with your theory, just an issue with novaclient.

Sure, though, if we support the async form in the API we need to test
it. So while adding poll support is a good UX add, it doesn't actually
mean we get to not test the async versions. It just adds another feature
set we need to test.

 #2 - the volume create command is table stakes. It should be an API
 call. However, it can't be because the service catalog redirection only
 works at the CLI layer. This is actually also the crux of bug #1423695.
 The same reason the completion cache code failed is the reason we can't
 use the API for that.
 
 
 https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L129
 
 
 Issues like this are why I wrote the read only nova CLI tests in the
 first place. Unit testing the python API is doable, but unit testing the
 CLI is a little bit more tricky. So I expect issues like this to come up
 over and over again.

So this is actually an issue with the API code. We have a chunk of code
that's directly callable by consumers, but if they do, it will error.

 #3 - the cleanup of the volume should have been API call. See reason
 for #2.
 
 
 https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L131
 
 #4 - the cleanup of the attachment should be an addCleanup via the API.
 See reason for #2 why it's not.
 
 
 https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L155
 
 
 I'm happy if there are other theories about how we do these things,
 being the first functional test in the python-novaclient tree that
 creates and destroys real resources, there isn't an established pattern
 yet. But I think doing all CLI calls in CLI tests is actually really
 cumbersome, especially in the amount of output parsing code needed if
 you are going to setup any complicated resource structure.
 
 
 Here is an alternate theory:
 
 We should have both python API and CLI functional tests. But they should
 be kept separate.
 
 This is to help us make sure both the CLI and python API are actually
 usable interfaces. As the exercise above has shown, they both have
 really major short comings.  I think having in tree functional testing
 covering both the CLI and python API will make it easier for us to
 review new client features in terms of usability.
 
 Here is a very rough proof of concept patch showing the same
 tests: 
 https://review.openstack.org/#/c/157974/2/novaclient/tests/functional/test_volumes.py,cm
 
 No matter how we define this functional testing model, I think its clear
 novaclient needs a decent amount of work before it can really be usable.

Agreed, but I think we should focus on testing the code we actually have
to prevent regressions. I think functional changes to put polling
throughout are nice UX, but 

Re: [openstack-dev] [nova] novaclient functional test guidelines

2015-02-24 Thread Ed Leafe
On Feb 24, 2015, at 1:49 PM, Sean Dague s...@dague.net wrote:

 IMHO the CLI should have an option to returned raw JSON back instead of
 pretty tabled results as well.
 
 Um... isn't that just the API calls?
 
 I'm not sure creating a 3rd functional surface is really the answer
 here, because we still need to actually test the CLI / pretty table output.

The python-openstacksdk project was originally envisioned to wrap the API calls 
and return usable Python objects. The nova client CLI (or any other CLI, for 
that matter) would then just provide the command line input parsing and output 
presentation. It's been a while since I was involved with that project, but it 
seems that decoupling the command line interface from the Python API wrapper 
would make testing much, much easier.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] novaclient functional test guidelines

2015-02-24 Thread Joe Gordon
On Tue, Feb 24, 2015 at 9:47 AM, Sean Dague s...@dague.net wrote:

 Towards the end of merging the regression test for the nova
 volume-attach bug - https://review.openstack.org/#/c/157959/ there was a
 discussion around what style the functional tests should take.
 Especially as that had a mix of CLI and API calls in it.



Thanks for starting this thread.  Once we reach general agreement lets put
this in a in tree README for record keeping.



 Here are my thoughts for why that test ended up that way:

 1) All resource setup that is table stakes for the test should be done
 via the API, regardless if it's a CLI or API test.

 The reason for this is that structured data is returned, which removes
 one possible error in the tests by parsing incorrectly. The API objects
 returned also include things like .delete methods in most cases, which
 means that addCleanup is a little more clean.


IMHO the CLI should have an option to returned raw JSON back instead of
pretty tabled results as well.



 2) Main logic should touch which ever interface you are trying to test.
 This was demonstrating a CLI regression, so the volume-attach call was
 important to be done over the CLI.


 Now... here's where theory runs into issues.

 #1 - nova boot is table stakes. Under the above guidelines it should be
 called via API. However --poll is a CLI construct and so saved a custom
 wait loop here. We should implement that custom wait loop down the road
 and make that an API call


 https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L116


This issue is from a real short coming in the python client. So this isn't
really an issue with your theory, just an issue with novaclient.




 #2 - the volume create command is table stakes. It should be an API
 call. However, it can't be because the service catalog redirection only
 works at the CLI layer. This is actually also the crux of bug #1423695.
 The same reason the completion cache code failed is the reason we can't
 use the API for that.


 https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L129


Issues like this are why I wrote the read only nova CLI tests in the first
place. Unit testing the python API is doable, but unit testing the CLI is a
little bit more tricky. So I expect issues like this to come up over and
over again.




 #3 - the cleanup of the volume should have been API call. See reason for
 #2.


 https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L131

 #4 - the cleanup of the attachment should be an addCleanup via the API.
 See reason for #2 why it's not.


 https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L155


 I'm happy if there are other theories about how we do these things,
 being the first functional test in the python-novaclient tree that
 creates and destroys real resources, there isn't an established pattern
 yet. But I think doing all CLI calls in CLI tests is actually really
 cumbersome, especially in the amount of output parsing code needed if
 you are going to setup any complicated resource structure.


Here is an alternate theory:

We should have both python API and CLI functional tests. But they should be
kept separate.

This is to help us make sure both the CLI and python API are actually
usable interfaces. As the exercise above has shown, they both have really
major short comings.  I think having in tree functional testing covering
both the CLI and python API will make it easier for us to review new client
features in terms of usability.

Here is a very rough proof of concept patch showing the same tests:
https://review.openstack.org/#/c/157974/2/novaclient/tests/functional/test_volumes.py,cm

No matter how we define this functional testing model, I think its clear
novaclient needs a decent amount of work before it can really be usable.



 -Sean

 --
 Sean Dague
 http://dague.net

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] novaclient functional test guidelines

2015-02-24 Thread Joe Gordon
On Tue, Feb 24, 2015 at 12:30 PM, Sean Dague s...@dague.net wrote:

 On 02/24/2015 03:28 PM, Ed Leafe wrote:
  On Feb 24, 2015, at 1:49 PM, Sean Dague s...@dague.net wrote:
 
  IMHO the CLI should have an option to returned raw JSON back instead of
  pretty tabled results as well.
 
  Um... isn't that just the API calls?
 
  I'm not sure creating a 3rd functional surface is really the answer
  here, because we still need to actually test the CLI / pretty table
 output.
 
  The python-openstacksdk project was originally envisioned to wrap the
 API calls and return usable Python objects. The nova client CLI (or any
 other CLI, for that matter) would then just provide the command line input
 parsing and output presentation. It's been a while since I was involved
 with that project, but it seems that decoupling the command line interface
 from the Python API wrapper would make testing much, much easier.

 Right, I think to some degree novaclient is legacy code, and we should
 focus on specific regressions and bugs without doing to much code change.

 The future should be more focussed on openstacksdk and openstackclient.


While I don't disagree with this, it seems like we have been waiting for
openstackclient and openstacksdk for a while now, do we even have a
timeline for moving off of novaclient?



 -Sean

 --
 Sean Dague
 http://dague.net


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] novaclient functional test guidelines

2015-02-24 Thread Joe Gordon
On Tue, Feb 24, 2015 at 1:18 PM, melanie witt melwi...@gmail.com wrote:

 On Feb 24, 2015, at 9:47, Sean Dague s...@dague.net wrote:

  I'm happy if there are other theories about how we do these things,
  being the first functional test in the python-novaclient tree that
  creates and destroys real resources, there isn't an established pattern
  yet. But I think doing all CLI calls in CLI tests is actually really
  cumbersome, especially in the amount of output parsing code needed if
  you are going to setup any complicated resource structure.

 I think I'm in agreement with the pattern you describe.

 I imagine having a set of functional tests for the API, that don't do any
 CLI calls at all. With that we test that the API works properly. Then have
 a separate set for the CLI, which only calls CLI for the command being
 tested, everything else to set up and tear down the test done by API calls.
 This would be done with the rationale that because the entire API
 functionality is tested separately, we can safely use it for setup/teardown
 with the intent to isolate the CLI test to the command being tested and
 avoid introducing side effects from the CLI commands.

 But I suppose one could make the same argument for using CLI everywhere
 (if they are all tested, they can also be trusted not to introduce side
 effects). I tend to favor using the API because it's the most bare bones
 setup/teardown we could use. At the same time I understand the idea of
 performing an entire test using the CLI, as a way of replicating the
 experience a real user might have using the CLI, from start to end. I don't
 think I feel strongly either way.


 I guess its time to revisit the actual status of novaclient and if we want
to actively move away from it to openstacksdk/OSC as well. If we are
actively trying to move away from novaclient, using the python API as much
as possible makes a lot of sense.




 For the --poll stuff, I agree the API should have it and the CLI uses it.
 And with and without poll functionality should be tested separately, API
 and CLI.

 melanie (melwitt)





 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev