Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?
On 12/10/2013 08:41 PM, Devananda van der Veen wrote: Tue, Dec 10, 2013 at 12:43 PM, David Kranz dkr...@redhat.com mailto:dkr...@redhat.com wrote: On 12/09/2013 01:37 PM, Devananda van der Veen wrote: On Fri, Dec 6, 2013 at 2:13 PM, Clark Boylan clark.boy...@gmail.com mailto:clark.boy...@gmail.com wrote: On Fri, Dec 6, 2013 at 1:53 PM, David Kranz dkr...@redhat.com mailto:dkr...@redhat.com wrote: It's great that tempest tests for ironic have been submitted! I was reviewing https://review.openstack.org/#/c/48109/ and noticed that the tests do not actually run. They are skipped because baremetal is not enabled. This is not terribly surprising but we have had a policy in tempest to only merge code that has demonstrated that it works. For services that cannot run in the single-vm environment of the upstream gate we said there could be a system running somewhere that would run them and report a result to gerrit. Is there a plan for this, or to make an exception for ironic? -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev There is a change[0] to openstack-infra/config to add experimental tempest jobs to test ironic. I think that change is close to being ready, but I need to give it time for a proper review. Once in that will allow you to test 48109 (in theory, not sure if all the bits will just work). I don't think these tests fall under the cannot run in a single vm environment umbrella, we should be able to test the baremetal code via the pxe booting of VMs within the single VM environment. [0] https://review.openstack.org/#/c/53917/ Clark We can test the ironic services, database, and the driver interfaces by using our fake driver within a single devstack VM today (I'm not sure the exercises for all of this have been written yet, but it's practical to test it). OTOH, I don't believe we can test a PXE deploy within a single VM today, and need to resume discussions with infra about this. There are some other aspects of Ironic (IPMI, SOL access, any vendor-specific drivers) which we'll need real hardware to test because they can't effectively be virtualized. TripleO should cover some (much?) of those needs, once they are able to switch to using Ironic instead of nova-baremetal. -Devananda So it seems that the code in the submitted tempest tests can run in a regular job if devstack is configured to enable ironic, but that this cannot be the default. So I propose that we create a regular devstack+ironic job that will run in the ironic and tempest gates, and run just the ironic tests. When third-party bare-metal results can be reported for ironic, tempest can then accept tests that require bare-metal. Does any one have a problem with this approach? -David As I understand it, the infra/config patch which Clark already linked (https://review.openstack.org/#/c/53917), which has gone through several iterations, should be enabling Ironic within devstack -- and thus causing tempest to run the relevant tests -- within the Ironic and Tempest check and gate pipelines. This will exercise Ironic's API by performing CRUD actions on resources. It doesn't do any more than that yet. It looks like that patch is adding ironic jobs to the experimental queue but I think we want them on check/gate. David, I'm not sure what you mean by when third-party bare-metal results can be reported for ironic -- I don't see any reason why we couldn't accept third-party smoke tests right now, except that none of the tempest tests are written... Am I missing something? I was assuming there were some ironic tests that actually need bare metal resources to run. Perhaps there are not. Either way, we just want to make sure that when tests are submitted to tempest we have evidence that they have successfully run. Sounds like the CRUD tests will just work the same way as our existing tests once ironic is enabled in devstack. In the longer term, we are planning to enable tempest testing of deployment by ironic within devstack-gate as all the pieces come together. This will take a fair bit more work / time, but I'm going to start nudging resources in this direction very soon. In fact, we just talked about this in #infra for a bit. Here's an attempt to summarize what came of it w.r.t. Ironic's testing plans. We will need: - some changes in devstack-gate to prepare a new environment by... -- install sshd +
Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?
On 12/09/2013 01:37 PM, Devananda van der Veen wrote: On Fri, Dec 6, 2013 at 2:13 PM, Clark Boylan clark.boy...@gmail.com mailto:clark.boy...@gmail.com wrote: snip We can test the ironic services, database, and the driver interfaces by using our fake driver within a single devstack VM today (I'm not sure the exercises for all of this have been written yet, but it's practical to test it). OTOH, I don't believe we can test a PXE deploy within a single VM today, and need to resume discussions with infra about this. FWIW, this is the main issue that we have struggled to overcome in our continuous testing process here at ATT for the better part of 18 months. It is exceedingly difficult to test bare-metal setup of RAID, network interface configuration, partitioning, and other such stuffs in a deployment gate. We eventually moved on to work on other things that, frankly, gave us much more bang for the proverbial buck than testing raw bare-metal configuration -- things like our OpenStack and infrastructure services Chef cookbooks. The reason I say that the bare-metal testing didn't give us as much bang for the buck is because the bare-metal configuration was, frankly, something we do very seldom -- usually just once when a deployment is initialized. Changes to bare-metal base configuration of RAID, network interfaces, and partitioning, are exceedingly rare; really thing change only when vendors screw up or we need to deal with a different set of hardware. Comparatively, OpenStack projects (installed and configured with our Chef cookbooks) change at a rapid pace, bugs are fixed and features are added continually. Therefore, it makes much more sense to focus our iterative testing loop on the things that change the most and provide the most benefit. There are some other aspects of Ironic (IPMI, SOL access, any vendor-specific drivers) which we'll need real hardware to test because they can't effectively be virtualized. TripleO should cover some (much?) of those needs, once they are able to switch to using Ironic instead of nova-baremetal. I'm very interested in how Triple-O will innovate in this space in the next year or so. The promise of a continually-delivered bare-metal undercloud is one thing. The promise of having a fully-automated Metal as a Service is another. I suspect it is the latter promise that most product people are salivating over right now. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?
On 11 December 2013 09:43, David Kranz dkr...@redhat.com wrote: So it seems that the code in the submitted tempest tests can run in a regular job if devstack is configured to enable ironic, but that this cannot be the default. So I propose that we create a regular devstack+ironic job that will run in the ironic and tempest gates, and run just the ironic tests. When third-party bare-metal results can be reported for ironic, tempest can then accept tests that require bare-metal. Does any one have a problem with this approach? Whats the connection between accepting baremetal tests and third-party testing? AIUI the criteria for acceptance is 'the thing is incubated or integrated'. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?
On 12/10/2013 07:05 PM, Robert Collins wrote: On 11 December 2013 09:43, David Kranz dkr...@redhat.com wrote: So it seems that the code in the submitted tempest tests can run in a regular job if devstack is configured to enable ironic, but that this cannot be the default. So I propose that we create a regular devstack+ironic job that will run in the ironic and tempest gates, and run just the ironic tests. When third-party bare-metal results can be reported for ironic, tempest can then accept tests that require bare-metal. Does any one have a problem with this approach? Whats the connection between accepting baremetal tests and third-party testing? AIUI the criteria for acceptance is 'the thing is incubated or integrated'. We also take the general policy of not landing things we've not seen execute, otherwise we've found we end up with a bunch of non working tests, because they aren't executed, a refactor happens, they don't get caught, fail So landing additional function means we need some data on it running. That means either something in infra (ideally, it can be a periodic and not an every patch sort of thing) though we can also handle that with something coming back on 3rd party testing. -Sean -- Sean Dague http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?
Tue, Dec 10, 2013 at 12:43 PM, David Kranz dkr...@redhat.com wrote: On 12/09/2013 01:37 PM, Devananda van der Veen wrote: On Fri, Dec 6, 2013 at 2:13 PM, Clark Boylan clark.boy...@gmail.comwrote: On Fri, Dec 6, 2013 at 1:53 PM, David Kranz dkr...@redhat.com wrote: It's great that tempest tests for ironic have been submitted! I was reviewing https://review.openstack.org/#/c/48109/ and noticed that the tests do not actually run. They are skipped because baremetal is not enabled. This is not terribly surprising but we have had a policy in tempest to only merge code that has demonstrated that it works. For services that cannot run in the single-vm environment of the upstream gate we said there could be a system running somewhere that would run them and report a result to gerrit. Is there a plan for this, or to make an exception for ironic? -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev There is a change[0] to openstack-infra/config to add experimental tempest jobs to test ironic. I think that change is close to being ready, but I need to give it time for a proper review. Once in that will allow you to test 48109 (in theory, not sure if all the bits will just work). I don't think these tests fall under the cannot run in a single vm environment umbrella, we should be able to test the baremetal code via the pxe booting of VMs within the single VM environment. [0] https://review.openstack.org/#/c/53917/ Clark We can test the ironic services, database, and the driver interfaces by using our fake driver within a single devstack VM today (I'm not sure the exercises for all of this have been written yet, but it's practical to test it). OTOH, I don't believe we can test a PXE deploy within a single VM today, and need to resume discussions with infra about this. There are some other aspects of Ironic (IPMI, SOL access, any vendor-specific drivers) which we'll need real hardware to test because they can't effectively be virtualized. TripleO should cover some (much?) of those needs, once they are able to switch to using Ironic instead of nova-baremetal. -Devananda So it seems that the code in the submitted tempest tests can run in a regular job if devstack is configured to enable ironic, but that this cannot be the default. So I propose that we create a regular devstack+ironic job that will run in the ironic and tempest gates, and run just the ironic tests. When third-party bare-metal results can be reported for ironic, tempest can then accept tests that require bare-metal. Does any one have a problem with this approach? -David As I understand it, the infra/config patch which Clark already linked ( https://review.openstack.org/#/c/53917), which has gone through several iterations, should be enabling Ironic within devstack -- and thus causing tempest to run the relevant tests -- within the Ironic and Tempest check and gate pipelines. This will exercise Ironic's API by performing CRUD actions on resources. It doesn't do any more than that yet. David, I'm not sure what you mean by when third-party bare-metal results can be reported for ironic -- I don't see any reason why we couldn't accept third-party smoke tests right now, except that none of the tempest tests are written... Am I missing something? In the longer term, we are planning to enable tempest testing of deployment by ironic within devstack-gate as all the pieces come together. This will take a fair bit more work / time, but I'm going to start nudging resources in this direction very soon. In fact, we just talked about this in #infra for a bit. Here's an attempt to summarize what came of it w.r.t. Ironic's testing plans. We will need: - some changes in devstack-gate to prepare a new environment by... -- install sshd + firewall it to only allow connections from localhost -- create a bunch of tiny qemu VMs (of configurable size and number) - some changes in devstack to... -- suck up a list of those VM's MAC addresses and enroll them in Ironic -- configure nova to use ironic -- configure ironic to use the pxe+ssh driver - a new test job that turns all this on, thus allowing tempest to do all its usual work against a virtual baremetal cloud Also, it's worth mentioning, the above-described plan won't exercise the CRUD of Ironic resources -- I think they need to be pre-enrolled with Ironic before tempest starts (maybe not? maybe tempest can enroll them, instead of devstack?). This is one reason why we have proposed separate tempest tests for exercising Ironic's API. Another reason is, testing our API is a valuable thing all by itself, and has a much lower development cost than all the changes above. -Devananda ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?
On Fri, Dec 6, 2013 at 2:13 PM, Clark Boylan clark.boy...@gmail.com wrote: On Fri, Dec 6, 2013 at 1:53 PM, David Kranz dkr...@redhat.com wrote: It's great that tempest tests for ironic have been submitted! I was reviewing https://review.openstack.org/#/c/48109/ and noticed that the tests do not actually run. They are skipped because baremetal is not enabled. This is not terribly surprising but we have had a policy in tempest to only merge code that has demonstrated that it works. For services that cannot run in the single-vm environment of the upstream gate we said there could be a system running somewhere that would run them and report a result to gerrit. Is there a plan for this, or to make an exception for ironic? -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev There is a change[0] to openstack-infra/config to add experimental tempest jobs to test ironic. I think that change is close to being ready, but I need to give it time for a proper review. Once in that will allow you to test 48109 (in theory, not sure if all the bits will just work). I don't think these tests fall under the cannot run in a single vm environment umbrella, we should be able to test the baremetal code via the pxe booting of VMs within the single VM environment. [0] https://review.openstack.org/#/c/53917/ Clark We can test the ironic services, database, and the driver interfaces by using our fake driver within a single devstack VM today (I'm not sure the exercises for all of this have been written yet, but it's practical to test it). OTOH, I don't believe we can test a PXE deploy within a single VM today, and need to resume discussions with infra about this. There are some other aspects of Ironic (IPMI, SOL access, any vendor-specific drivers) which we'll need real hardware to test because they can't effectively be virtualized. TripleO should cover some (much?) of those needs, once they are able to switch to using Ironic instead of nova-baremetal. -Devananda ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?
On 10 December 2013 07:37, Devananda van der Veen devananda@gmail.com wrote: We can test the ironic services, database, and the driver interfaces by using our fake driver within a single devstack VM today (I'm not sure the exercises for all of this have been written yet, but it's practical to test it). OTOH, I don't believe we can test a PXE deploy within a single VM today, and need to resume discussions with infra about this. I think you can with qemu, but only VM's that are super lightweight (like cirros) heavier things like OpenStack itself will be prohibitively slow. There are some other aspects of Ironic (IPMI, SOL access, any vendor-specific drivers) which we'll need real hardware to test because they can't effectively be virtualized. TripleO should cover some (much?) of those needs, once they are able to switch to using Ironic instead of nova-baremetal. We can cover that two ways - post deploy feedback, and using machines to run tests against during gating. The latter is clearly better, but we'll need a broader set of small machines to do that (we currently have a set of big machines, which are fantastic for divide and conquer workloads, but not for anything that needs a full machine. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ironic][qa] How will ironic tests run in tempest?
It's great that tempest tests for ironic have been submitted! I was reviewing https://review.openstack.org/#/c/48109/ and noticed that the tests do not actually run. They are skipped because baremetal is not enabled. This is not terribly surprising but we have had a policy in tempest to only merge code that has demonstrated that it works. For services that cannot run in the single-vm environment of the upstream gate we said there could be a system running somewhere that would run them and report a result to gerrit. Is there a plan for this, or to make an exception for ironic? -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ironic][qa] How will ironic tests run in tempest?
On Fri, Dec 6, 2013 at 1:53 PM, David Kranz dkr...@redhat.com wrote: It's great that tempest tests for ironic have been submitted! I was reviewing https://review.openstack.org/#/c/48109/ and noticed that the tests do not actually run. They are skipped because baremetal is not enabled. This is not terribly surprising but we have had a policy in tempest to only merge code that has demonstrated that it works. For services that cannot run in the single-vm environment of the upstream gate we said there could be a system running somewhere that would run them and report a result to gerrit. Is there a plan for this, or to make an exception for ironic? -David ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev There is a change[0] to openstack-infra/config to add experimental tempest jobs to test ironic. I think that change is close to being ready, but I need to give it time for a proper review. Once in that will allow you to test 48109 (in theory, not sure if all the bits will just work). I don't think these tests fall under the cannot run in a single vm environment umbrella, we should be able to test the baremetal code via the pxe booting of VMs within the single VM environment. [0] https://review.openstack.org/#/c/53917/ Clark ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
