Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
-Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: 25 November 2013 22:37 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan On 11/25/2013 05:19 PM, Matt Riedemann wrote: I'll play devil's advocate here and ask this question before someone else does. I'm assuming that the requirement of a 'full' tempest run means running this [1]. Is that correct? It's just confusing sometimes because there are other things in Tempest that aren't in the 'full' run, like stress tests. [1] https://github.com/openstack/tempest/blob/master/tox.ini#L33 I think the short answer is, whatever we're running against all Nova changes in the gate. Can we strip this down a bit? I don't see any benefit in running tests that verify Swift's behaviour other than where it interacts with Nova. I hope we can safely say that we should run against all gating tests which require Nova? Currently we run quite a number of tests in the gate that succeed even when Nova is not running as the gate isn't just for Nova but for all projects. It would be trivial to kill Nova, run the full tempest and only enable those jobs for a new flavour which we must run. Thoughts? Bob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On 11/25/2013 08:00 PM, Matt Riedemann wrote: On Monday, November 25, 2013 4:37:29 PM, Russell Bryant wrote: I think the short answer is, whatever we're running against all Nova changes in the gate. Maybe a silly question, but is what is run against the check queue any different from the gate queue? A better question, where can I look to see the current configuration used for jenkins jobs? First look at the jenkins_job_builder config to see what options are set for various jobs: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/devstack-gate.yaml Then, see devstack-gate here and search for TEMPEST. You'll see what the knobs set in the job configs do. http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On 11/26/2013 04:48 AM, Bob Ball wrote: -Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: 25 November 2013 22:37 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan On 11/25/2013 05:19 PM, Matt Riedemann wrote: I'll play devil's advocate here and ask this question before someone else does. I'm assuming that the requirement of a 'full' tempest run means running this [1]. Is that correct? It's just confusing sometimes because there are other things in Tempest that aren't in the 'full' run, like stress tests. [1] https://github.com/openstack/tempest/blob/master/tox.ini#L33 I think the short answer is, whatever we're running against all Nova changes in the gate. Can we strip this down a bit? I don't see any benefit in running tests that verify Swift's behaviour other than where it interacts with Nova. I hope we can safely say that we should run against all gating tests which require Nova? Currently we run quite a number of tests in the gate that succeed even when Nova is not running as the gate isn't just for Nova but for all projects. It would be trivial to kill Nova, run the full tempest and only enable those jobs for a new flavour which we must run. Thoughts? Would you like to come up with a more detailed proposal? What tests would you cut, and how much time does it save? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
-Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: 26 November 2013 13:56 To: openstack-dev@lists.openstack.org Cc: Sean Dague Subject: Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan On 11/26/2013 04:48 AM, Bob Ball wrote: I hope we can safely say that we should run against all gating tests which require Nova? Currently we run quite a number of tests in the gate that succeed even when Nova is not running as the gate isn't just for Nova but for all projects. Would you like to come up with a more detailed proposal? What tests would you cut, and how much time does it save? I don't have a detailed proposal yet - but it's very possible that we'll want one in the coming weeks. In terms of the time saved, I noticed that a tempest smoke run with Nova absent took 400 seconds on one of my machines (a particularly slow one) - so I imagine that would translate to maybe a 300 second / 5 minute reduction in overall time. Total smoke took approximately 800 seconds on the same machine. If the approach could be acceptable then yes, I'm happy to come up with a detailed set of tests that I would propose cutting. My primary hesitation with the approach is it would need Tempest reviewers to be aware of this extra type of test, and flag up if a test is added to the full tempest suite which should also be in the nova tempest suite. Bob ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On 11/26/2013 09:38 AM, Bob Ball wrote: -Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: 26 November 2013 13:56 To: openstack-dev@lists.openstack.org Cc: Sean Dague Subject: Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan On 11/26/2013 04:48 AM, Bob Ball wrote: I hope we can safely say that we should run against all gating tests which require Nova? Currently we run quite a number of tests in the gate that succeed even when Nova is not running as the gate isn't just for Nova but for all projects. Would you like to come up with a more detailed proposal? What tests would you cut, and how much time does it save? I don't have a detailed proposal yet - but it's very possible that we'll want one in the coming weeks. In terms of the time saved, I noticed that a tempest smoke run with Nova absent took 400 seconds on one of my machines (a particularly slow one) - so I imagine that would translate to maybe a 300 second / 5 minute reduction in overall time. Total smoke took approximately 800 seconds on the same machine. I don't think the smoke tests are really relevant here. That's not related to Nova vs non-Nova tests, right? If the approach could be acceptable then yes, I'm happy to come up with a detailed set of tests that I would propose cutting. My primary hesitation with the approach is it would need Tempest reviewers to be aware of this extra type of test, and flag up if a test is added to the full tempest suite which should also be in the nova tempest suite. Right now I don't think it's acceptable. I was suggesting a more detailed proposal to help convince me. :-) -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On Tuesday, November 26, 2013 10:07:02 AM, Sean Dague wrote: On 11/26/2013 09:56 AM, Russell Bryant wrote: On 11/26/2013 09:38 AM, Bob Ball wrote: -Original Message- From: Russell Bryant [mailto:rbry...@redhat.com] Sent: 26 November 2013 13:56 To: openstack-dev@lists.openstack.org Cc: Sean Dague Subject: Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan On 11/26/2013 04:48 AM, Bob Ball wrote: I hope we can safely say that we should run against all gating tests which require Nova? Currently we run quite a number of tests in the gate that succeed even when Nova is not running as the gate isn't just for Nova but for all projects. Would you like to come up with a more detailed proposal? What tests would you cut, and how much time does it save? I don't have a detailed proposal yet - but it's very possible that we'll want one in the coming weeks. In terms of the time saved, I noticed that a tempest smoke run with Nova absent took 400 seconds on one of my machines (a particularly slow one) - so I imagine that would translate to maybe a 300 second / 5 minute reduction in overall time. Total smoke took approximately 800 seconds on the same machine. I don't think the smoke tests are really relevant here. That's not related to Nova vs non-Nova tests, right? If the approach could be acceptable then yes, I'm happy to come up with a detailed set of tests that I would propose cutting. My primary hesitation with the approach is it would need Tempest reviewers to be aware of this extra type of test, and flag up if a test is added to the full tempest suite which should also be in the nova tempest suite. Right now I don't think it's acceptable. I was suggesting a more detailed proposal to help convince me. :-) So we already have the beginnings of service tags in Tempest, that would let you slice exactly like this. I don't think the infrastructure is fully complete yet, but the idea being that you could run the subset of tests that interact with compute or networking in any real way. Realize... that's not going to drop that many tests for something like compute, it's touched a lot. -Sean ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Good to know about the service tags, I think I remember being broken at some point after those tempest.conf.sample changes. :) My overall concern, and I think the other guys doing this for virt drivers will agree, is trying to scope down the exposure to unrelated failures. For example, if there is a bug in swift breaking the gate, it could start breaking the nova virt driver CI as well. When things get bad in the gate, it takes some monstrous effort to rally people across the projects to come together to unblock it (like what Joe Gordon was doing last week). I'm running Tempest internally about once per day when we rebase code with the community and that's to cover running with the PowerVM driver for nova, Storwize driver for cinder, OVS for neutron, with qpid and DB2. We're running almost a full run except for the third party boto tests and swift API tests. The thing is, when something fails, I have to figure out if it's environmental (infra), a problem with tempest (think instability with neutron in the gate), a configuration issue, or a code bug. That's a lot for one person to have to cover, even a small team. That's why at some points we just have to ignore/exclude tests that continuously fail but we can't figure out (think intermittent gate breaker bugs that are open for months). Now multiply this out across all the nova virt drivers, the neutron plugins and I'm assuming at some point the various glance backends and cinder drivers (haven't heard if they are planning on the same types of CI requirements yet). I think either we're going to have a lot of flaky/instable driver CI going on so the scores can't be trusted, or we're going to develop a lot of people that get really good at infra/QA (which would be a plus in the long-run, but maybe not what those teams set out to be). I don't have any good answers, I'm just trying to raise the issue since this is complicated. I think it's also hard for people that aren't forced to invest in infra/QA on a daily basis to understand and appreciate the amount of effort it takes just to keep the wheels spinning, so I want to keep expectations at a reasonable level. Don't get me wrong, I absolutely agree with requiring third party CI for the various vendor-specific drivers and plugins, that's a no-brainer for openstack to scale. I think it will just be very interesting to see the kinds of results coming out of all of these disconnected teams come icehouse-3. -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
My overall concern, and I think the other guys doing this for virt drivers will agree, is trying to scope down the exposure to unrelated failures. But, if it's not related to your driver, then it also failed in the upstream gate, right? If it didn't fail the upstream gate, then it is some weird interaction. Of course, for spurious failures, you'll still have the case where you recheck something and it passes the next time, but that's exactly what we have today. I don't see a new CI job for a given driver being a monstrous amount of work above and beyond what we have to do today to triage the issues. Of course, driver CI is not a single-time task that you never have to babysit... I think that running as close as possible to the upstream gate is crucial for us to gain the level of comfort we have with the stuff that is actually tested upstream. If you pass more often because you don't include any of the tests you don't strictly have to run, and don't have a set of folks ready to triage failures, then you don't have quality parity with the other stuff. That's kinda the whole point of us doing this, no? --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On 11/15/2013 9:28 AM, Dan Smith wrote: Hi all, As you know, Nova adopted a plan to require CI testing for all our in-tree hypervisors by the Icehouse release. At the summit last week, we determined the actual plan for deprecating non-compliant drivers. I put together a page detailing the specific requirements we're putting in place as well as a plan and timeline for how the deprecation process will proceed: https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan I also listed the various drivers and whether we've heard any concrete plans from them. Driver owners should feel free to add details to that and correct any of the statements if incorrect. Thanks! --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I'll play devil's advocate here and ask this question before someone else does. I'm assuming that the requirement of a 'full' tempest run means running this [1]. Is that correct? It's just confusing sometimes because there are other things in Tempest that aren't in the 'full' run, like stress tests. Assuming that's what 'full' means, it's running API, CLI, third party (boto), and scenario tests. Does it make sense to require a nova virt driver's CI to run API tests for keystone, heat and swift? Or couldn't the nova virt driver CI be scoped down to just the compute API tests? The argument against that is probably that the network/image/volume tests may create instances using nova to do their API testing also. The same would apply for the CLI tests since those are broken down by service, i.e. why would I need to run keystone and ceilometer CLI tests for a nova virt driver? If nothing else, I think we could firm up the wording on the wiki a bit around the requirements and what that means for scope. [1] https://github.com/openstack/tempest/blob/master/tox.ini#L33 -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On 11/25/2013 05:19 PM, Matt Riedemann wrote: I'll play devil's advocate here and ask this question before someone else does. I'm assuming that the requirement of a 'full' tempest run means running this [1]. Is that correct? It's just confusing sometimes because there are other things in Tempest that aren't in the 'full' run, like stress tests. Assuming that's what 'full' means, it's running API, CLI, third party (boto), and scenario tests. Does it make sense to require a nova virt driver's CI to run API tests for keystone, heat and swift? Or couldn't the nova virt driver CI be scoped down to just the compute API tests? The argument against that is probably that the network/image/volume tests may create instances using nova to do their API testing also. The same would apply for the CLI tests since those are broken down by service, i.e. why would I need to run keystone and ceilometer CLI tests for a nova virt driver? If nothing else, I think we could firm up the wording on the wiki a bit around the requirements and what that means for scope. [1] https://github.com/openstack/tempest/blob/master/tox.ini#L33 I think the short answer is, whatever we're running against all Nova changes in the gate. I expect that for some drivers, a more specific configuration is going to be needed to exclude tests for features not implemented in that driver. That's fine. Soon we also need to start solidifying criteria for what features *must* be implemented in a driver. I think we've let some drivers in with far too many features not supported. That's a separate issue from the CI requirement, though. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On Monday, November 25, 2013 4:37:29 PM, Russell Bryant wrote: On 11/25/2013 05:19 PM, Matt Riedemann wrote: I'll play devil's advocate here and ask this question before someone else does. I'm assuming that the requirement of a 'full' tempest run means running this [1]. Is that correct? It's just confusing sometimes because there are other things in Tempest that aren't in the 'full' run, like stress tests. Assuming that's what 'full' means, it's running API, CLI, third party (boto), and scenario tests. Does it make sense to require a nova virt driver's CI to run API tests for keystone, heat and swift? Or couldn't the nova virt driver CI be scoped down to just the compute API tests? The argument against that is probably that the network/image/volume tests may create instances using nova to do their API testing also. The same would apply for the CLI tests since those are broken down by service, i.e. why would I need to run keystone and ceilometer CLI tests for a nova virt driver? If nothing else, I think we could firm up the wording on the wiki a bit around the requirements and what that means for scope. [1] https://github.com/openstack/tempest/blob/master/tox.ini#L33 I think the short answer is, whatever we're running against all Nova changes in the gate. Maybe a silly question, but is what is run against the check queue any different from the gate queue? I expect that for some drivers, a more specific configuration is going to be needed to exclude tests for features not implemented in that driver. That's fine. Soon we also need to start solidifying criteria for what features *must* be implemented in a driver. I think we've let some drivers in with far too many features not supported. That's a separate issue from the CI requirement, though. -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On Sat 16 Nov 2013 (17:43), Russell Bryant wrote: On 11/16/2013 11:18 AM, Álvaro López García wrote: On Fri 15 Nov 2013 (07:28), Dan Smith wrote: Hi all, Hi Dan. As you know, Nova adopted a plan to require CI testing for all our in-tree hypervisors by the Icehouse release. At the summit last week, we determined the actual plan for deprecating non-compliant drivers. I put together a page detailing the specific requirements we're putting in place as well as a plan and timeline for how the deprecation process will proceed: https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan I also listed the various drivers and whether we've heard any concrete plans from them. Driver owners should feel free to add details to that and correct any of the statements if incorrect. I've posted an email one month ago about this and the support of Xen via libvirt [1]. We are happy users of this ad we would love to see it being promoted form the use-at-your-own-risk group C to group B. Should we add it to the list of drivers in the group C, and add another column to the table so that we can start filling it with the supported/working features? Should I fill a blueprint for this? [1] http://lists.openstack.org/pipermail/openstack-dev/2013-October/016661.html I don't think we need a blueprint. Yes, I would list it under group C and add a column for it. That's fine, we will update the wiki and start filling the information. Is there any chance you would be interested in working on setting up CI for it? Definitely yes. We are interested in the libvirt+Xen support, so having it promoted from group C to B (and eventually A) would be great. What would be the necessary steps? Is there any document/guidance we can follow? Cheers, -- Álvaro López García al...@ifca.unican.es Instituto de Física de Cantabria http://alvarolopez.github.io Ed. Juan Jordá, Campus UC tel: (+34) 942 200 969 Avda. de los Castros s/n 39005 Santander (SPAIN) _ There are two ways to write error-free programs, but only the third one works. -- Alan Perlis ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On 11/18/2013 09:55 AM, Álvaro López García wrote: On Sat 16 Nov 2013 (17:43), Russell Bryant wrote: On 11/16/2013 11:18 AM, Álvaro López García wrote: On Fri 15 Nov 2013 (07:28), Dan Smith wrote: Hi all, Hi Dan. As you know, Nova adopted a plan to require CI testing for all our in-tree hypervisors by the Icehouse release. At the summit last week, we determined the actual plan for deprecating non-compliant drivers. I put together a page detailing the specific requirements we're putting in place as well as a plan and timeline for how the deprecation process will proceed: https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan I also listed the various drivers and whether we've heard any concrete plans from them. Driver owners should feel free to add details to that and correct any of the statements if incorrect. I've posted an email one month ago about this and the support of Xen via libvirt [1]. We are happy users of this ad we would love to see it being promoted form the use-at-your-own-risk group C to group B. Should we add it to the list of drivers in the group C, and add another column to the table so that we can start filling it with the supported/working features? Should I fill a blueprint for this? [1] http://lists.openstack.org/pipermail/openstack-dev/2013-October/016661.html I don't think we need a blueprint. Yes, I would list it under group C and add a column for it. That's fine, we will update the wiki and start filling the information. Is there any chance you would be interested in working on setting up CI for it? Definitely yes. We are interested in the libvirt+Xen support, so having it promoted from group C to B (and eventually A) would be great. What would be the necessary steps? Is there any document/guidance we can follow? This is a good place to start: http://ci.openstack.org/third_party.html -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On Fri 15 Nov 2013 (07:28), Dan Smith wrote: Hi all, Hi Dan. As you know, Nova adopted a plan to require CI testing for all our in-tree hypervisors by the Icehouse release. At the summit last week, we determined the actual plan for deprecating non-compliant drivers. I put together a page detailing the specific requirements we're putting in place as well as a plan and timeline for how the deprecation process will proceed: https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan I also listed the various drivers and whether we've heard any concrete plans from them. Driver owners should feel free to add details to that and correct any of the statements if incorrect. I've posted an email one month ago about this and the support of Xen via libvirt [1]. We are happy users of this ad we would love to see it being promoted form the use-at-your-own-risk group C to group B. Should we add it to the list of drivers in the group C, and add another column to the table so that we can start filling it with the supported/working features? Should I fill a blueprint for this? [1] http://lists.openstack.org/pipermail/openstack-dev/2013-October/016661.html Thanks! --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Álvaro López García al...@ifca.unican.es Instituto de Física de Cantabria http://alvarolopez.github.io Ed. Juan Jordá, Campus UC tel: (+34) 942 200 969 Avda. de los Castros s/n 39005 Santander (SPAIN) _ http://xkcd.com/571/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
On 11/16/2013 11:18 AM, Álvaro López García wrote: On Fri 15 Nov 2013 (07:28), Dan Smith wrote: Hi all, Hi Dan. As you know, Nova adopted a plan to require CI testing for all our in-tree hypervisors by the Icehouse release. At the summit last week, we determined the actual plan for deprecating non-compliant drivers. I put together a page detailing the specific requirements we're putting in place as well as a plan and timeline for how the deprecation process will proceed: https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan I also listed the various drivers and whether we've heard any concrete plans from them. Driver owners should feel free to add details to that and correct any of the statements if incorrect. I've posted an email one month ago about this and the support of Xen via libvirt [1]. We are happy users of this ad we would love to see it being promoted form the use-at-your-own-risk group C to group B. Should we add it to the list of drivers in the group C, and add another column to the table so that we can start filling it with the supported/working features? Should I fill a blueprint for this? [1] http://lists.openstack.org/pipermail/openstack-dev/2013-October/016661.html I don't think we need a blueprint. Yes, I would list it under group C and add a column for it. Is there any chance you would be interested in working on setting up CI for it? -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Nova] Hypervisor CI requirement and deprecation plan
Hi all, As you know, Nova adopted a plan to require CI testing for all our in-tree hypervisors by the Icehouse release. At the summit last week, we determined the actual plan for deprecating non-compliant drivers. I put together a page detailing the specific requirements we're putting in place as well as a plan and timeline for how the deprecation process will proceed: https://wiki.openstack.org/wiki/HypervisorSupportMatrix/DeprecationPlan I also listed the various drivers and whether we've heard any concrete plans from them. Driver owners should feel free to add details to that and correct any of the statements if incorrect. Thanks! --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
