Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?

2013-12-11 Thread David Kranz 

On 12/10/2013 08:41 PM, Devananda van der Veen wrote:
 Tue, Dec 10, 2013 at 12:43 PM, David Kranz > wrote:


On 12/09/2013 01:37 PM, Devananda van der Veen wrote:

On Fri, Dec 6, 2013 at 2:13 PM, Clark Boylan
mailto:clark.boy...@gmail.com>> wrote:

On Fri, Dec 6, 2013 at 1:53 PM, David Kranz
mailto:dkr...@redhat.com>> wrote:
> It's great that tempest tests for ironic have been
submitted! I was
> reviewing https://review.openstack.org/#/c/48109/ and
noticed that the tests
> do not actually run. They are skipped because baremetal is
not enabled. This
> is not terribly surprising but we have had a policy in
tempest to only merge
> code that has demonstrated that it works. For services that
cannot run in
> the single-vm environment of the upstream gate we said
there could be a
> system running somewhere that would run them and report a
result to gerrit.
> Is there a plan for this, or to make an exception for ironic?
>
>  -David
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org

>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

There is a change[0] to openstack-infra/config to add
experimental
tempest jobs to test ironic. I think that change is close to
being
ready, but I need to give it time for a proper review. Once
in that
will allow you to test 48109 (in theory, not sure if all the
bits will
just work). I don't think these tests fall under the cannot
run in a
single vm environment umbrella, we should be able to test the
baremetal code via the pxe booting of VMs within the single VM
environment.

[0] https://review.openstack.org/#/c/53917/


Clark


We can test the ironic services, database, and the driver
interfaces by using our "fake" driver within a single devstack VM
today (I'm not sure the exercises for all of this have been
written yet, but it's practical to test it). OTOH, I don't
believe we can test a PXE deploy within a single VM today, and
need to resume discussions with infra about this.

There are some other aspects of Ironic (IPMI, SOL access, any
vendor-specific drivers) which we'll need real hardware to test
because they can't effectively be virtualized. TripleO should
cover some (much?) of those needs, once they are able to switch
to using Ironic instead of nova-baremetal.

-Devananda

So it seems that the code in the submitted tempest tests can run
in a regular job if devstack is configured to enable ironic, but
that this cannot be the default. So I propose that we create a
regular devstack+ironic job that will run in the ironic and
tempest gates, and run just the ironic tests. When third-party
bare-metal results can be reported for ironic, tempest can then
accept tests that require bare-metal.  Does any one have a problem
with this approach?

 -David


As I understand it, the infra/config patch which Clark already linked 
(https://review.openstack.org/#/c/53917), which has gone through 
several iterations, should be enabling Ironic within devstack -- and 
thus causing tempest to run the relevant tests -- within the Ironic 
and Tempest check and gate pipelines. This will exercise Ironic's API 
by performing CRUD actions on resources. It doesn't do any more than 
that yet.
It looks like that patch is adding ironic jobs to the experimental queue 
but I think we want them on check/gate.


David, I'm not sure what you mean by "when third-party bare-metal 
results can be reported for ironic" -- I don't see any reason why we 
couldn't accept third-party smoke tests right now, except that none of 
the tempest tests are written... Am I missing something?
I was assuming there were some ironic tests that actually need bare 
metal resources to run. Perhaps there are not. Either way, we just want 
to make sure that when tests are submitted to tempest we have evidence 
that they have successfully run. Sounds like the CRUD tests will just 
work the same way as our existing tests once ironic is enabled in devstack.


In the longer term, we are planning to enable tempest testing of 
deployment by ironic within devstack-gate as all the pieces come 
together. This will take a fair bit more work / time, but I'm going to 
start nudging resources in this direction very soon. In fact, we just 
talked about this in #infra for a bit. Here's an attempt to summarize 
what came of it w.r.t. Ironic's testing plans. We will need:


- some changes in devstack-gate to prepare a new environment by...
-- install sshd + firewall it to only allow connections

Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?

2013-12-10 Thread Devananda van der Veen 
 Tue, Dec 10, 2013 at 12:43 PM, David Kranz  wrote:

>  On 12/09/2013 01:37 PM, Devananda van der Veen wrote:
>
>  On Fri, Dec 6, 2013 at 2:13 PM, Clark Boylan wrote:
>
>>  On Fri, Dec 6, 2013 at 1:53 PM, David Kranz  wrote:
>> > It's great that tempest tests for ironic have been submitted! I was
>> > reviewing https://review.openstack.org/#/c/48109/ and noticed that the
>> tests
>> > do not actually run. They are skipped because baremetal is not enabled.
>> This
>> > is not terribly surprising but we have had a policy in tempest to only
>> merge
>> > code that has demonstrated that it works. For services that cannot run
>> in
>> > the single-vm environment of the upstream gate we said there could be a
>> > system running somewhere that would run them and report a result to
>> gerrit.
>> > Is there a plan for this, or to make an exception for ironic?
>> >
>> >  -David
>> >
>> > ___
>> > OpenStack-dev mailing list
>> > OpenStack-dev@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>  There is a change[0] to openstack-infra/config to add experimental
>> tempest jobs to test ironic. I think that change is close to being
>> ready, but I need to give it time for a proper review. Once in that
>> will allow you to test 48109 (in theory, not sure if all the bits will
>> just work). I don't think these tests fall under the cannot run in a
>> single vm environment umbrella, we should be able to test the
>> baremetal code via the pxe booting of VMs within the single VM
>> environment.
>>
>> [0] https://review.openstack.org/#/c/53917/
>>
>>
>> Clark
>>
>>
>  We can test the ironic services, database, and the driver interfaces by
> using our "fake" driver within a single devstack VM today (I'm not sure the
> exercises for all of this have been written yet, but it's practical to test
> it). OTOH, I don't believe we can test a PXE deploy within a single VM
> today, and need to resume discussions with infra about this.
>
>  There are some other aspects of Ironic (IPMI, SOL access, any
> vendor-specific drivers) which we'll need real hardware to test because
> they can't effectively be virtualized. TripleO should cover some (much?) of
> those needs, once they are able to switch to using Ironic instead of
> nova-baremetal.
>
>  -Devananda
>
> So it seems that the code in the submitted tempest tests can run in a
> regular job if devstack is configured to enable ironic, but that this
> cannot be the default. So I propose that we create a regular
> devstack+ironic job that will run in the ironic and tempest gates, and run
> just the ironic tests. When third-party bare-metal results can be reported
> for ironic, tempest can then accept tests that require bare-metal.  Does
> any one have a problem with this approach?
>
>  -David
>
>
As I understand it, the infra/config patch which Clark already linked (
https://review.openstack.org/#/c/53917), which has gone through several
iterations, should be enabling Ironic within devstack -- and thus causing
tempest to run the relevant tests -- within the Ironic and Tempest check
and gate pipelines. This will exercise Ironic's API by performing CRUD
actions on resources. It doesn't do any more than that yet.

David, I'm not sure what you mean by "when third-party bare-metal results
can be reported for ironic" -- I don't see any reason why we couldn't
accept third-party smoke tests right now, except that none of the tempest
tests are written... Am I missing something?

In the longer term, we are planning to enable tempest testing of deployment
by ironic within devstack-gate as all the pieces come together. This will
take a fair bit more work / time, but I'm going to start nudging resources
in this direction very soon. In fact, we just talked about this in #infra
for a bit. Here's an attempt to summarize what came of it w.r.t. Ironic's
testing plans. We will need:

- some changes in devstack-gate to prepare a new environment by...
-- install sshd + firewall it to only allow connections from localhost
-- create a bunch of tiny qemu VMs (of configurable size and number)
- some changes in devstack to...
-- suck up a list of those VM's MAC addresses and enroll them in Ironic
-- configure nova to use ironic
-- configure ironic to use the pxe+ssh driver
- a new test job that turns all this on, thus allowing tempest to do all
its usual work against a "virtual baremetal" cloud

Also, it's worth mentioning, the above-described plan won't exercise the
CRUD of Ironic resources -- I think they need to be pre-enrolled with
Ironic before tempest starts (maybe not? maybe tempest can enroll them,
instead of devstack?). This is one reason why we have proposed separate
tempest tests for exercising Ironic's API. Another reason is, testing our
API is a valuable thing all by itself, and has a much lower development
cost than all the changes above.

-Devananda
___
OpenStack-dev mail

Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?

2013-12-10 Thread Sean Dague 
On 12/10/2013 07:05 PM, Robert Collins wrote:
> On 11 December 2013 09:43, David Kranz  wrote:
> 
>> So it seems that the code in the submitted tempest tests can run in a
>> regular job if devstack is configured to enable ironic, but that this cannot
>> be the default. So I propose that we create a regular devstack+ironic job
>> that will run in the ironic and tempest gates, and run just the ironic
>> tests. When third-party bare-metal results can be reported for ironic,
>> tempest can then accept tests that require bare-metal.  Does any one have a
>> problem with this approach?
> 
> Whats the connection between accepting baremetal tests and third-party
> testing? AIUI the criteria for acceptance is 'the thing is incubated
> or integrated'.

We also take the general policy of not landing things we've not seen
execute, otherwise we've found we end up with a bunch of non working
tests, because they aren't executed, a refactor happens, they don't get
caught, fail

So landing additional function means we need some data on it running.
That means either something in infra (ideally, it can be a periodic and
not an every patch sort of thing) though we can also handle that with
something coming back on 3rd party testing.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?

2013-12-10 Thread Robert Collins 
On 11 December 2013 09:43, David Kranz  wrote:

> So it seems that the code in the submitted tempest tests can run in a
> regular job if devstack is configured to enable ironic, but that this cannot
> be the default. So I propose that we create a regular devstack+ironic job
> that will run in the ironic and tempest gates, and run just the ironic
> tests. When third-party bare-metal results can be reported for ironic,
> tempest can then accept tests that require bare-metal.  Does any one have a
> problem with this approach?

Whats the connection between accepting baremetal tests and third-party
testing? AIUI the criteria for acceptance is 'the thing is incubated
or integrated'.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?

2013-12-10 Thread David Kranz 

On 12/09/2013 01:37 PM, Devananda van der Veen wrote:
On Fri, Dec 6, 2013 at 2:13 PM, Clark Boylan > wrote:


On Fri, Dec 6, 2013 at 1:53 PM, David Kranz mailto:dkr...@redhat.com>> wrote:
> It's great that tempest tests for ironic have been submitted! I was
> reviewing https://review.openstack.org/#/c/48109/ and noticed
that the tests
> do not actually run. They are skipped because baremetal is not
enabled. This
> is not terribly surprising but we have had a policy in tempest
to only merge
> code that has demonstrated that it works. For services that
cannot run in
> the single-vm environment of the upstream gate we said there
could be a
> system running somewhere that would run them and report a result
to gerrit.
> Is there a plan for this, or to make an exception for ironic?
>
>  -David
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org

> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

There is a change[0] to openstack-infra/config to add experimental
tempest jobs to test ironic. I think that change is close to being
ready, but I need to give it time for a proper review. Once in that
will allow you to test 48109 (in theory, not sure if all the bits will
just work). I don't think these tests fall under the cannot run in a
single vm environment umbrella, we should be able to test the
baremetal code via the pxe booting of VMs within the single VM
environment.

[0] https://review.openstack.org/#/c/53917/


Clark


We can test the ironic services, database, and the driver interfaces 
by using our "fake" driver within a single devstack VM today (I'm not 
sure the exercises for all of this have been written yet, but it's 
practical to test it). OTOH, I don't believe we can test a PXE deploy 
within a single VM today, and need to resume discussions with infra 
about this.


There are some other aspects of Ironic (IPMI, SOL access, any 
vendor-specific drivers) which we'll need real hardware to test 
because they can't effectively be virtualized. TripleO should cover 
some (much?) of those needs, once they are able to switch to using 
Ironic instead of nova-baremetal.


-Devananda
So it seems that the code in the submitted tempest tests can run in a 
regular job if devstack is configured to enable ironic, but that this 
cannot be the default. So I propose that we create a regular 
devstack+ironic job that will run in the ironic and tempest gates, and 
run just the ironic tests. When third-party bare-metal results can be 
reported for ironic, tempest can then accept tests that require 
bare-metal.  Does any one have a problem with this approach?


 -David
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?

2013-12-10 Thread Jay Pipes 

On 12/09/2013 01:37 PM, Devananda van der Veen wrote:

On Fri, Dec 6, 2013 at 2:13 PM, Clark Boylan mailto:clark.boy...@gmail.com>> wrote:



We can test the ironic services, database, and the driver interfaces by
using our "fake" driver within a single devstack VM today (I'm not sure
the exercises for all of this have been written yet, but it's practical
to test it). OTOH, I don't believe we can test a PXE deploy within a
single VM today, and need to resume discussions with infra about this.


FWIW, this is the main issue that we have struggled to overcome in our 
continuous testing process here at AT&T for the better part of 18 
months. It is exceedingly difficult to test bare-metal setup of RAID, 
network interface configuration, partitioning, and other such stuffs in 
a deployment gate.


We eventually moved on to work on other things that, frankly, gave us 
much more bang for the proverbial buck than testing raw bare-metal 
configuration -- things like our OpenStack and infrastructure services 
Chef cookbooks. The reason I say that the bare-metal testing didn't give 
us as much bang for the buck is because the bare-metal configuration 
was, frankly, something we do very seldom -- usually just once when a 
deployment is initialized. Changes to bare-metal base configuration of 
RAID, network interfaces, and partitioning, are exceedingly rare; really 
thing change only when vendors screw up or we need to deal with a 
different set of hardware. Comparatively, OpenStack projects (installed 
and configured with our Chef cookbooks) change at a rapid pace, bugs are 
fixed and features are added continually. Therefore, it makes much more 
sense to focus our "iterative testing loop" on the things that change 
the most and provide the most benefit.



There are some other aspects of Ironic (IPMI, SOL access, any
vendor-specific drivers) which we'll need real hardware to test because
they can't effectively be virtualized. TripleO should cover some (much?)
of those needs, once they are able to switch to using Ironic instead of
nova-baremetal.


I'm very interested in how Triple-O will innovate in this space in the 
next year or so. The promise of a continually-delivered bare-metal 
undercloud is one thing. The promise of having a fully-automated "Metal 
as a Service" is another. I suspect it is the latter promise that most 
product people are salivating over right now.


Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?

2013-12-09 Thread Robert Collins 
On 10 December 2013 07:37, Devananda van der Veen
 wrote:

>
> We can test the ironic services, database, and the driver interfaces by
> using our "fake" driver within a single devstack VM today (I'm not sure the
> exercises for all of this have been written yet, but it's practical to test
> it). OTOH, I don't believe we can test a PXE deploy within a single VM
> today, and need to resume discussions with infra about this.

I think you can with qemu, but only VM's that are super lightweight
(like cirros) heavier things like OpenStack itself will be
prohibitively slow.

> There are some other aspects of Ironic (IPMI, SOL access, any
> vendor-specific drivers) which we'll need real hardware to test because they
> can't effectively be virtualized. TripleO should cover some (much?) of those
> needs, once they are able to switch to using Ironic instead of
> nova-baremetal.

We can cover that two ways - post deploy feedback, and using machines
to run tests against during gating. The latter is clearly better, but
we'll need a broader set of small machines to do that (we currently
have a set of big machines, which are fantastic for divide and conquer
workloads, but not for anything that needs a full machine.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?

2013-12-09 Thread Devananda van der Veen 
On Fri, Dec 6, 2013 at 2:13 PM, Clark Boylan  wrote:

> On Fri, Dec 6, 2013 at 1:53 PM, David Kranz  wrote:
> > It's great that tempest tests for ironic have been submitted! I was
> > reviewing https://review.openstack.org/#/c/48109/ and noticed that the
> tests
> > do not actually run. They are skipped because baremetal is not enabled.
> This
> > is not terribly surprising but we have had a policy in tempest to only
> merge
> > code that has demonstrated that it works. For services that cannot run in
> > the single-vm environment of the upstream gate we said there could be a
> > system running somewhere that would run them and report a result to
> gerrit.
> > Is there a plan for this, or to make an exception for ironic?
> >
> >  -David
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> There is a change[0] to openstack-infra/config to add experimental
> tempest jobs to test ironic. I think that change is close to being
> ready, but I need to give it time for a proper review. Once in that
> will allow you to test 48109 (in theory, not sure if all the bits will
> just work). I don't think these tests fall under the cannot run in a
> single vm environment umbrella, we should be able to test the
> baremetal code via the pxe booting of VMs within the single VM
> environment.
>
> [0] https://review.openstack.org/#/c/53917/
>
>
> Clark
>
>
We can test the ironic services, database, and the driver interfaces by
using our "fake" driver within a single devstack VM today (I'm not sure the
exercises for all of this have been written yet, but it's practical to test
it). OTOH, I don't believe we can test a PXE deploy within a single VM
today, and need to resume discussions with infra about this.

There are some other aspects of Ironic (IPMI, SOL access, any
vendor-specific drivers) which we'll need real hardware to test because
they can't effectively be virtualized. TripleO should cover some (much?) of
those needs, once they are able to switch to using Ironic instead of
nova-baremetal.

-Devananda
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?

2013-12-06 Thread Clark Boylan 
On Fri, Dec 6, 2013 at 1:53 PM, David Kranz  wrote:
> It's great that tempest tests for ironic have been submitted! I was
> reviewing https://review.openstack.org/#/c/48109/ and noticed that the tests
> do not actually run. They are skipped because baremetal is not enabled. This
> is not terribly surprising but we have had a policy in tempest to only merge
> code that has demonstrated that it works. For services that cannot run in
> the single-vm environment of the upstream gate we said there could be a
> system running somewhere that would run them and report a result to gerrit.
> Is there a plan for this, or to make an exception for ironic?
>
>  -David
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

There is a change[0] to openstack-infra/config to add experimental
tempest jobs to test ironic. I think that change is close to being
ready, but I need to give it time for a proper review. Once in that
will allow you to test 48109 (in theory, not sure if all the bits will
just work). I don't think these tests fall under the cannot run in a
single vm environment umbrella, we should be able to test the
baremetal code via the pxe booting of VMs within the single VM
environment.

[0] https://review.openstack.org/#/c/53917/


Clark

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev