Re: [openstack-dev] [nova] novaclient functional test guidelines

[Joe Gordon]

First pass at trying to capture this thread into a README:
https://review.openstack.org/162334

On Tue, Feb 24, 2015 at 2:07 PM, Joe Gordon  wrote:

>
>
> On Tue, Feb 24, 2015 at 1:18 PM, melanie witt  wrote:
>
>> On Feb 24, 2015, at 9:47, Sean Dague  wrote:
>>
>> > I'm happy if there are other theories about how we do these things,
>> > being the first functional test in the python-novaclient tree that
>> > creates and destroys real resources, there isn't an established pattern
>> > yet. But I think doing all CLI calls in CLI tests is actually really
>> > cumbersome, especially in the amount of output parsing code needed if
>> > you are going to setup any complicated resource structure.
>>
>> I think I'm in agreement with the pattern you describe.
>>
>> I imagine having a set of functional tests for the API, that don't do any
>> CLI calls at all. With that we test that the API works properly. Then have
>> a separate set for the CLI, which only calls CLI for the command being
>> tested, everything else to set up and tear down the test done by API calls.
>> This would be done with the rationale that because the entire API
>> functionality is tested separately, we can safely use it for setup/teardown
>> with the intent to isolate the CLI test to the command being tested and
>> avoid introducing side effects from the CLI commands.
>>
>> But I suppose one could make the same argument for using CLI everywhere
>> (if they are all tested, they can also be trusted not to introduce side
>> effects). I tend to favor using the API because it's the most bare bones
>> setup/teardown we could use. At the same time I understand the idea of
>> performing an entire test using the CLI, as a way of replicating the
>> experience a real user might have using the CLI, from start to end. I don't
>> think I feel strongly either way.
>
>
>  I guess its time to revisit the actual status of novaclient and if we
> want to actively move away from it to openstacksdk/OSC as well. If we are
> actively trying to move away from novaclient, using the python API as much
> as possible makes a lot of sense.
>
>
>
>>
>> For the --poll stuff, I agree the API should have it and the CLI uses it.
>> And with and without "poll" functionality should be tested separately, API
>> and CLI.
>>
>> melanie (melwitt)
>>
>>
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] novaclient functional test guidelines

[Joe Gordon]

On Tue, Feb 24, 2015 at 1:18 PM, melanie witt  wrote:

> On Feb 24, 2015, at 9:47, Sean Dague  wrote:
>
> > I'm happy if there are other theories about how we do these things,
> > being the first functional test in the python-novaclient tree that
> > creates and destroys real resources, there isn't an established pattern
> > yet. But I think doing all CLI calls in CLI tests is actually really
> > cumbersome, especially in the amount of output parsing code needed if
> > you are going to setup any complicated resource structure.
>
> I think I'm in agreement with the pattern you describe.
>
> I imagine having a set of functional tests for the API, that don't do any
> CLI calls at all. With that we test that the API works properly. Then have
> a separate set for the CLI, which only calls CLI for the command being
> tested, everything else to set up and tear down the test done by API calls.
> This would be done with the rationale that because the entire API
> functionality is tested separately, we can safely use it for setup/teardown
> with the intent to isolate the CLI test to the command being tested and
> avoid introducing side effects from the CLI commands.
>
> But I suppose one could make the same argument for using CLI everywhere
> (if they are all tested, they can also be trusted not to introduce side
> effects). I tend to favor using the API because it's the most bare bones
> setup/teardown we could use. At the same time I understand the idea of
> performing an entire test using the CLI, as a way of replicating the
> experience a real user might have using the CLI, from start to end. I don't
> think I feel strongly either way.


 I guess its time to revisit the actual status of novaclient and if we want
to actively move away from it to openstacksdk/OSC as well. If we are
actively trying to move away from novaclient, using the python API as much
as possible makes a lot of sense.



>
> For the --poll stuff, I agree the API should have it and the CLI uses it.
> And with and without "poll" functionality should be tested separately, API
> and CLI.
>
> melanie (melwitt)
>
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] novaclient functional test guidelines

[Joe Gordon]

On Tue, Feb 24, 2015 at 12:30 PM, Sean Dague  wrote:

> On 02/24/2015 03:28 PM, Ed Leafe wrote:
> > On Feb 24, 2015, at 1:49 PM, Sean Dague  wrote:
> >
> >>> IMHO the CLI should have an option to returned raw JSON back instead of
> >>> pretty tabled results as well.
> >>
> >> Um... isn't that just the API calls?
> >>
> >> I'm not sure creating a 3rd functional surface is really the answer
> >> here, because we still need to actually test the CLI / pretty table
> output.
> >
> > The python-openstacksdk project was originally envisioned to wrap the
> API calls and return usable Python objects. The nova client CLI (or any
> other CLI, for that matter) would then just provide the command line input
> parsing and output presentation. It's been a while since I was involved
> with that project, but it seems that decoupling the command line interface
> from the Python API wrapper would make testing much, much easier.
>
> Right, I think to some degree novaclient is legacy code, and we should
> focus on specific regressions and bugs without doing to much code change.
>
> The future should be more focussed on openstacksdk and openstackclient.
>

While I don't disagree with this, it seems like we have been waiting for
openstackclient and openstacksdk for a while now, do we even have a
timeline for moving off of novaclient?


>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] novaclient functional test guidelines

[melanie witt]

On Feb 24, 2015, at 9:47, Sean Dague  wrote:

> I'm happy if there are other theories about how we do these things,
> being the first functional test in the python-novaclient tree that
> creates and destroys real resources, there isn't an established pattern
> yet. But I think doing all CLI calls in CLI tests is actually really
> cumbersome, especially in the amount of output parsing code needed if
> you are going to setup any complicated resource structure.

I think I'm in agreement with the pattern you describe.

I imagine having a set of functional tests for the API, that don't do any CLI 
calls at all. With that we test that the API works properly. Then have a 
separate set for the CLI, which only calls CLI for the command being tested, 
everything else to set up and tear down the test done by API calls. This would 
be done with the rationale that because the entire API functionality is tested 
separately, we can safely use it for setup/teardown with the intent to isolate 
the CLI test to the command being tested and avoid introducing side effects 
from the CLI commands.

But I suppose one could make the same argument for using CLI everywhere (if 
they are all tested, they can also be trusted not to introduce side effects). I 
tend to favor using the API because it's the most bare bones setup/teardown we 
could use. At the same time I understand the idea of performing an entire test 
using the CLI, as a way of replicating the experience a real user might have 
using the CLI, from start to end. I don't think I feel strongly either way.

For the --poll stuff, I agree the API should have it and the CLI uses it. And 
with and without "poll" functionality should be tested separately, API and CLI.

melanie (melwitt)






signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] novaclient functional test guidelines

[Ed Leafe]

On Feb 24, 2015, at 2:30 PM, Sean Dague  wrote:

> Right, I think to some degree novaclient is legacy code, and we should
> focus on specific regressions and bugs without doing to much code change.
> 
> The future should be more focussed on openstacksdk and openstackclient.

IMO, openstackclient has an impossible task: taking the varied (and flawed) CLI 
clients and unite them under a single CLI interface. It is better to create the 
clean separation of the API wrapping into a Python library, and keep the CLI 
completely separate.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] novaclient functional test guidelines

[Sean Dague]

On 02/24/2015 03:28 PM, Ed Leafe wrote:
> On Feb 24, 2015, at 1:49 PM, Sean Dague  wrote:
> 
>>> IMHO the CLI should have an option to returned raw JSON back instead of
>>> pretty tabled results as well.
>>
>> Um... isn't that just the API calls?
>>
>> I'm not sure creating a 3rd functional surface is really the answer
>> here, because we still need to actually test the CLI / pretty table output.
> 
> The python-openstacksdk project was originally envisioned to wrap the API 
> calls and return usable Python objects. The nova client CLI (or any other 
> CLI, for that matter) would then just provide the command line input parsing 
> and output presentation. It's been a while since I was involved with that 
> project, but it seems that decoupling the command line interface from the 
> Python API wrapper would make testing much, much easier.

Right, I think to some degree novaclient is legacy code, and we should
focus on specific regressions and bugs without doing to much code change.

The future should be more focussed on openstacksdk and openstackclient.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] novaclient functional test guidelines

[Ed Leafe]

On Feb 24, 2015, at 1:49 PM, Sean Dague  wrote:

>> IMHO the CLI should have an option to returned raw JSON back instead of
>> pretty tabled results as well.
> 
> Um... isn't that just the API calls?
> 
> I'm not sure creating a 3rd functional surface is really the answer
> here, because we still need to actually test the CLI / pretty table output.

The python-openstacksdk project was originally envisioned to wrap the API calls 
and return usable Python objects. The nova client CLI (or any other CLI, for 
that matter) would then just provide the command line input parsing and output 
presentation. It's been a while since I was involved with that project, but it 
seems that decoupling the command line interface from the Python API wrapper 
would make testing much, much easier.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] novaclient functional test guidelines

[Sean Dague]

On 02/24/2015 01:47 PM, Joe Gordon wrote:
> 
> 
> On Tue, Feb 24, 2015 at 9:47 AM, Sean Dague  > wrote:
> 
> Towards the end of merging the regression test for the nova
> volume-attach bug - https://review.openstack.org/#/c/157959/ there was a
> discussion around what style the functional tests should take.
> Especially as that had a mix of CLI and API calls in it.
> 
> 
> 
> Thanks for starting this thread.  Once we reach general agreement lets
> put this in a in tree README for record keeping.

Absolutely.

> Here are my thoughts for why that test ended up that way:
> 
> 1) All resource setup that is table stakes for the test should be done
> via the API, regardless if it's a CLI or API test.
> 
> The reason for this is that structured data is returned, which removes
> one possible error in the tests by parsing incorrectly. The API objects
> returned also include things like .delete methods in most cases, which
> means that addCleanup is a little more clean.
> 
> 
> IMHO the CLI should have an option to returned raw JSON back instead of
> pretty tabled results as well.

Um... isn't that just the API calls?

I'm not sure creating a 3rd functional surface is really the answer
here, because we still need to actually test the CLI / pretty table output.

> 2) Main logic should touch which ever interface you are trying to test.
> This was demonstrating a CLI regression, so the volume-attach call was
> important to be done over the CLI.
> 
> 
> Now... here's where theory runs into issues.
> 
> #1 - nova boot is table stakes. Under the above guidelines it should be
> called via API. However --poll is a CLI construct and so saved a custom
> wait loop here. We should implement that custom wait loop down the road
> and make that an API call
> 
> 
> https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L116
> 
> 
> This issue is from a real short coming in the python client. So this
> isn't really an issue with your theory, just an issue with novaclient.

Sure, though, if we support the async form in the API we need to test
it. So while adding poll support is a good UX add, it doesn't actually
mean we get to not test the async versions. It just adds another feature
set we need to test.

> #2 - the volume create command is table stakes. It should be an API
> call. However, it can't be because the service catalog redirection only
> works at the CLI layer. This is actually also the crux of bug #1423695.
> The same reason the completion cache code failed is the reason we can't
> use the API for that.
> 
> 
> https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L129
> 
> 
> Issues like this are why I wrote the read only nova CLI tests in the
> first place. Unit testing the python API is doable, but unit testing the
> CLI is a little bit more tricky. So I expect issues like this to come up
> over and over again.

So this is actually an issue with the API code. We have a chunk of code
that's directly callable by consumers, but if they do, it will error.

> #3 - the cleanup of the volume should have been API call. See reason
> for #2.
> 
> 
> https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L131
> 
> #4 - the cleanup of the attachment should be an addCleanup via the API.
> See reason for #2 why it's not.
> 
> 
> https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L155
> 
> 
> I'm happy if there are other theories about how we do these things,
> being the first functional test in the python-novaclient tree that
> creates and destroys real resources, there isn't an established pattern
> yet. But I think doing all CLI calls in CLI tests is actually really
> cumbersome, especially in the amount of output parsing code needed if
> you are going to setup any complicated resource structure.
> 
> 
> Here is an alternate theory:
> 
> We should have both python API and CLI functional tests. But they should
> be kept separate.
> 
> This is to help us make sure both the CLI and python API are actually
> usable interfaces. As the exercise above has shown, they both have
> really major short comings.  I think having in tree functional testing
> covering both the CLI and python API will make it easier for us to
> review new client features in terms of usability.
> 
> Here is a very rough proof of concept patch showing the same
> tests: 
> https://review.openstack.org/#/c/157974/2/novaclient/tests/functional/test_volumes.py,cm
> 
> No matter how we define this functional testing model, I think its clear
> novaclient needs a decent amount of work before it can really be usable.

Agreed, but I think we should focus on testing the code we actually have
to prevent 

Re: [openstack-dev] [nova] novaclient functional test guidelines

[Joe Gordon]

On Tue, Feb 24, 2015 at 9:47 AM, Sean Dague  wrote:

> Towards the end of merging the regression test for the nova
> volume-attach bug - https://review.openstack.org/#/c/157959/ there was a
> discussion around what style the functional tests should take.
> Especially as that had a mix of CLI and API calls in it.
>


Thanks for starting this thread.  Once we reach general agreement lets put
this in a in tree README for record keeping.


>
> Here are my thoughts for why that test ended up that way:
>
> 1) All resource setup that is table stakes for the test should be done
> via the API, regardless if it's a CLI or API test.
>
> The reason for this is that structured data is returned, which removes
> one possible error in the tests by parsing incorrectly. The API objects
> returned also include things like .delete methods in most cases, which
> means that addCleanup is a little more clean.
>

IMHO the CLI should have an option to returned raw JSON back instead of
pretty tabled results as well.


>
> 2) Main logic should touch which ever interface you are trying to test.
> This was demonstrating a CLI regression, so the volume-attach call was
> important to be done over the CLI.
>
>
> Now... here's where theory runs into issues.
>
> #1 - nova boot is table stakes. Under the above guidelines it should be
> called via API. However --poll is a CLI construct and so saved a custom
> wait loop here. We should implement that custom wait loop down the road
> and make that an API call
>
>
> https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L116


This issue is from a real short coming in the python client. So this isn't
really an issue with your theory, just an issue with novaclient.


>
>
> #2 - the volume create command is table stakes. It should be an API
> call. However, it can't be because the service catalog redirection only
> works at the CLI layer. This is actually also the crux of bug #1423695.
> The same reason the completion cache code failed is the reason we can't
> use the API for that.
>
>
> https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L129


Issues like this are why I wrote the read only nova CLI tests in the first
place. Unit testing the python API is doable, but unit testing the CLI is a
little bit more tricky. So I expect issues like this to come up over and
over again.


>
>
> #3 - the cleanup of the volume should have been API call. See reason for
> #2.
>
>
> https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L131
>
> #4 - the cleanup of the attachment should be an addCleanup via the API.
> See reason for #2 why it's not.
>
>
> https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/test_instances.py#L155
>
>
> I'm happy if there are other theories about how we do these things,
> being the first functional test in the python-novaclient tree that
> creates and destroys real resources, there isn't an established pattern
> yet. But I think doing all CLI calls in CLI tests is actually really
> cumbersome, especially in the amount of output parsing code needed if
> you are going to setup any complicated resource structure.
>
>
Here is an alternate theory:

We should have both python API and CLI functional tests. But they should be
kept separate.

This is to help us make sure both the CLI and python API are actually
usable interfaces. As the exercise above has shown, they both have really
major short comings.  I think having in tree functional testing covering
both the CLI and python API will make it easier for us to review new client
features in terms of usability.

Here is a very rough proof of concept patch showing the same tests:
https://review.openstack.org/#/c/157974/2/novaclient/tests/functional/test_volumes.py,cm

No matter how we define this functional testing model, I think its clear
novaclient needs a decent amount of work before it can really be usable.



> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev