Re: [openstack-dev] [ironic][tests] approach to functional/integration tests

2016-01-12 Thread Dmitry Tantsur 

On 01/11/2016 03:49 PM, Serge Kovaleff wrote:

Hi All,

Last week I had a noble goal to write "one-more" functional test in Ironic.
I did find a folder "func" but it was empty.

Friends helped me to find a WIP patch
https://review.openstack.org/#/c/235612/

and here comes the question of this email: what approach we would like
to implement:
Option 1 - write infrastructure code that starts/configure/stops the
services
Option 2 - rely on installed DevStack and run the tests over it

Both options have their Cons and Pros. Both options are implemented
across the OpenStack umbrella.
Option 1 - Glance, Nova, the patch above
Option 2 - HEAT and my favorite at the moment.

Any ideas?


I think we should eventually end up with supporting both standalone 
functional tests (#1) and tempest-based (#3).


A bit of context on #1. We've been using it in ironic-inspector since 
nearly its inception, when devstack plugins didn't exist and our project 
was on stackforge, so we had no way of implementing our dsvm gate. The 
basic idea is to start a full-featured service with mocked access to 
other services and simplified environment. We've written a decorator [1] 
that starts the service in __enter__ and stops in __exit__. It uses a 
temporary configuration file [2] with authentication disabled and 
database in temporary SQLite file. The service is started in a new green 
thread and exits when the test exits. We mock ironicclient with the 
usual 'mock' library to avoid requirements on a running ironic instance.


We do 2 kinds of tests: just an API test like [3] or full-flow 
introspection tests like [4]. In the latter case we first start 
introspection via API, verify that status is "in progress", then call 
the ramdisk callback endpoint with a fake data, and verify that 
introspection ends successfully.


Applying the same thing to ironic might be somewhat trickier. We can run 
conductor and API in the same process and use oslo.messaging fake driver 
[5] to avoid AMQP dependency. We'll have to use a fake network 
implementation and either mock glance or make sure we use local file 
system URL's for all images (IIRC we do support it).


Going further, if we create a simple fake IPA, we can even do a 
full-flow test with fake_agent driver. We will start deployment, make 
sure fake IPA was called with a right image, make it report success and 
see deployment finish. We might want to modify fake interfaces to record 
all calls, so that we can verify that boot interface was called properly.


[1] 
https://github.com/openstack/ironic-inspector/blob/master/ironic_inspector/test/functional.py#L358-L393
[2] 
https://github.com/openstack/ironic-inspector/blob/master/ironic_inspector/test/functional.py#L36-L51
[3] 
https://github.com/openstack/ironic-inspector/blob/master/ironic_inspector/test/functional.py#L249-L291
[4] 
https://github.com/openstack/ironic-inspector/blob/master/ironic_inspector/test/functional.py#L177-L196

[5] http://docs.openstack.org/developer/oslo.messaging/drivers.html#fake



Cheers,
Serge Kovaleff
http://www.mirantis.com 
cell: +38 (063) 83-155-70


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][tests] approach to functional/integration tests

2016-01-11 Thread Christopher_Dearborn 
I like #1 as well since it would enable quick coding iteration, but I would 
also like to be able to run the tests against a running OpenStack cluster that 
includes Ironic, so put me down for #3 (Tempest) as well.


Chris Dearborn
Dell Inc

From: John Villalovos [mailto:openstack@sodarock.com]
Sent: Monday, January 11, 2016 10:42 AM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [ironic][tests] approach to functional/integration 
tests

I personally like Option #1 as an additional goal is to make it easy for 
developers to also run the functional tests with 'tox -efunctional'.  And also 
run the functional tests as part of a normal tox run.

On Mon, Jan 11, 2016 at 6:54 AM, Serge Kovaleff 
<skoval...@mirantis.com<mailto:skoval...@mirantis.com>> wrote:
Option 3 - to concentrate on Tempest tests 
https://review.openstack.org/#/c/253982/

Cheers,
Serge Kovaleff
http://www.mirantis.com<http://www.mirantis.com/>
cell: +38 (063) 83-155-70<tel:%2B38%20%28063%29%2083-155-70>

On Mon, Jan 11, 2016 at 4:49 PM, Serge Kovaleff 
<skoval...@mirantis.com<mailto:skoval...@mirantis.com>> wrote:
Hi All,

Last week I had a noble goal to write "one-more" functional test in Ironic.
I did find a folder "func" but it was empty.

Friends helped me to find a WIP patch https://review.openstack.org/#/c/235612/

and here comes the question of this email: what approach we would like to 
implement:
Option 1 - write infrastructure code that starts/configure/stops the services
Option 2 - rely on installed DevStack and run the tests over it

Both options have their Cons and Pros. Both options are implemented across the 
OpenStack umbrella.
Option 1 - Glance, Nova, the patch above
Option 2 - HEAT and my favorite at the moment.

Any ideas?

Cheers,
Serge Kovaleff
http://www.mirantis.com<http://www.mirantis.com/>
cell: +38 (063) 83-155-70<tel:%2B38%20%28063%29%2083-155-70>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][tests] approach to functional/integration tests

2016-01-11 Thread Serge Kovaleff 
Option 3 - to concentrate on Tempest tests
https://review.openstack.org/#/c/253982/

Cheers,
Serge Kovaleff
http://www.mirantis.com
cell: +38 (063) 83-155-70

On Mon, Jan 11, 2016 at 4:49 PM, Serge Kovaleff 
wrote:

> Hi All,
>
> Last week I had a noble goal to write "one-more" functional test in Ironic.
> I did find a folder "func" but it was empty.
>
> Friends helped me to find a WIP patch
> https://review.openstack.org/#/c/235612/
>
> and here comes the question of this email: what approach we would like to
> implement:
> Option 1 - write infrastructure code that starts/configure/stops the
> services
> Option 2 - rely on installed DevStack and run the tests over it
>
> Both options have their Cons and Pros. Both options are implemented across
> the OpenStack umbrella.
> Option 1 - Glance, Nova, the patch above
> Option 2 - HEAT and my favorite at the moment.
>
> Any ideas?
>
> Cheers,
> Serge Kovaleff
> http://www.mirantis.com
> cell: +38 (063) 83-155-70
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][tests] approach to functional/integration tests

2016-01-11 Thread Serge Kovaleff 
We might want to implement all three options. The more tests the better :)

Cheers,
Serge Kovaleff
http://www.mirantis.com
cell: +38 (063) 83-155-70

On Mon, Jan 11, 2016 at 5:41 PM, John Villalovos  wrote:

> I personally like Option #1 as an additional goal is to make it easy for
> developers to also run the functional tests with 'tox -efunctional'.  And
> also run the functional tests as part of a normal tox run.
>
> On Mon, Jan 11, 2016 at 6:54 AM, Serge Kovaleff 
> wrote:
>
>> Option 3 - to concentrate on Tempest tests
>> https://review.openstack.org/#/c/253982/
>>
>> Cheers,
>> Serge Kovaleff
>> http://www.mirantis.com
>> cell: +38 (063) 83-155-70
>>
>> On Mon, Jan 11, 2016 at 4:49 PM, Serge Kovaleff 
>> wrote:
>>
>>> Hi All,
>>>
>>> Last week I had a noble goal to write "one-more" functional test in
>>> Ironic.
>>> I did find a folder "func" but it was empty.
>>>
>>> Friends helped me to find a WIP patch
>>> https://review.openstack.org/#/c/235612/
>>>
>>> and here comes the question of this email: what approach we would like
>>> to implement:
>>> Option 1 - write infrastructure code that starts/configure/stops the
>>> services
>>> Option 2 - rely on installed DevStack and run the tests over it
>>>
>>> Both options have their Cons and Pros. Both options are implemented
>>> across the OpenStack umbrella.
>>> Option 1 - Glance, Nova, the patch above
>>> Option 2 - HEAT and my favorite at the moment.
>>>
>>> Any ideas?
>>>
>>> Cheers,
>>> Serge Kovaleff
>>> http://www.mirantis.com
>>> cell: +38 (063) 83-155-70
>>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][tests] approach to functional/integration tests

2016-01-11 Thread John Villalovos 
I personally like Option #1 as an additional goal is to make it easy for
developers to also run the functional tests with 'tox -efunctional'.  And
also run the functional tests as part of a normal tox run.

On Mon, Jan 11, 2016 at 6:54 AM, Serge Kovaleff 
wrote:

> Option 3 - to concentrate on Tempest tests
> https://review.openstack.org/#/c/253982/
>
> Cheers,
> Serge Kovaleff
> http://www.mirantis.com
> cell: +38 (063) 83-155-70
>
> On Mon, Jan 11, 2016 at 4:49 PM, Serge Kovaleff 
> wrote:
>
>> Hi All,
>>
>> Last week I had a noble goal to write "one-more" functional test in
>> Ironic.
>> I did find a folder "func" but it was empty.
>>
>> Friends helped me to find a WIP patch
>> https://review.openstack.org/#/c/235612/
>>
>> and here comes the question of this email: what approach we would like to
>> implement:
>> Option 1 - write infrastructure code that starts/configure/stops the
>> services
>> Option 2 - rely on installed DevStack and run the tests over it
>>
>> Both options have their Cons and Pros. Both options are implemented
>> across the OpenStack umbrella.
>> Option 1 - Glance, Nova, the patch above
>> Option 2 - HEAT and my favorite at the moment.
>>
>> Any ideas?
>>
>> Cheers,
>> Serge Kovaleff
>> http://www.mirantis.com
>> cell: +38 (063) 83-155-70
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic][tests] approach to functional/integration tests

2016-01-11 Thread Serge Kovaleff 
Hi All,

Last week I had a noble goal to write "one-more" functional test in Ironic.
I did find a folder "func" but it was empty.

Friends helped me to find a WIP patch
https://review.openstack.org/#/c/235612/

and here comes the question of this email: what approach we would like to
implement:
Option 1 - write infrastructure code that starts/configure/stops the
services
Option 2 - rely on installed DevStack and run the tests over it

Both options have their Cons and Pros. Both options are implemented across
the OpenStack umbrella.
Option 1 - Glance, Nova, the patch above
Option 2 - HEAT and my favorite at the moment.

Any ideas?

Cheers,
Serge Kovaleff
http://www.mirantis.com
cell: +38 (063) 83-155-70
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev