Re: [openstack-dev] [neutron] Skip-only tests
John Schwarz wrote: >Hi all, > >The Neutron community has recently started contributing Tempest >scenario tests in the Neutron tree and I'd like to discuss a general >issue we're hitting. Mainly, we're talking about required hardware for >some features and/or VM images that support more features, which makes >it hard (or impossible) to test certain features at the gate >end-to-end. Specifically, tests that require SR-IOV or OVS-DPDK, and >tests relating to VLAN Aware VMs. >[Š] Resurrecting this thread with another example where this would be useful: IPv6 Prefix Delegation This feature still lacks end-to-end testing because it depends upon having a compatible DHCPv6 server installed and running with an appropriate config, and specifically dibbler[1] to act as the DHCPv6 client. I¹d be interested in assisting with this effort. John [1] https://github.com/tomaszmrugalski/dibbler Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com and delete the original message. Your cooperation is appreciated. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Skip-only tests
On Thu, Sep 8, 2016 at 12:44 PM, Joe Hakim Rahme wrote: > On Thu, 2016-09-08 at 10:36 +0200, Andreas Jaeger wrote: > > > > Toni, sorry, I wasn't clear. I'm not advocating for this, I wanted to > > bring Joe into this discussion to see whether the solution that John > > suggested will help them as well - I wanted to broaden the scope of > > this since it's not only neutron that faces this. > > > > Hi, > > Just to be clear, my intention with the repo I'm trying to create is > to hold tests that are not acceptable in upstream Tempest by design > (mainly whitebox testing). I think what John was referring to (correct > me if I'm wrong) are tests that are acceptable Tempest tests but we > lack the required infrastrcuture (hardware, images, ...) to run them > in the gate. > > I realize now that the name I chose "nova-tempest-staging" might be > misleading and I think it should be changed. Maybe something like > "tempest-whitebox-plugin" might be more descriptive? > This sounds like a generic repo, not specific or tied to service. Not being specific sounds much better to me. I liked the idea and would be willing to help out in the effort, but agree it needs more thorough discussion. > > As for the topic of skip-only tests, I think that it'd be a shame to > reject valid and valuable tests simply because the gate is not equipped > to run them. There are use cases for Tempest outside of the upstream > gate, so marking certain tests as valid-but-skipped-in-the-gate seems > like an acceptable solution to me. > > (I'm not a maintainer of the Tempest repo so please take my opinion > with a grain of salt). > > Cheers, > Joe > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Rodrigo Duarte Sousa Senior Quality Engineer @ Red Hat MSc in Computer Science http://rodrigods.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Skip-only tests
On Thu, 2016-09-08 at 10:36 +0200, Andreas Jaeger wrote: > > Toni, sorry, I wasn't clear. I'm not advocating for this, I wanted to > bring Joe into this discussion to see whether the solution that John > suggested will help them as well - I wanted to broaden the scope of > this since it's not only neutron that faces this. > Hi, Just to be clear, my intention with the repo I'm trying to create is to hold tests that are not acceptable in upstream Tempest by design (mainly whitebox testing). I think what John was referring to (correct me if I'm wrong) are tests that are acceptable Tempest tests but we lack the required infrastrcuture (hardware, images, ...) to run them in the gate. I realize now that the name I chose "nova-tempest-staging" might be misleading and I think it should be changed. Maybe something like "tempest-whitebox-plugin" might be more descriptive? As for the topic of skip-only tests, I think that it'd be a shame to reject valid and valuable tests simply because the gate is not equipped to run them. There are use cases for Tempest outside of the upstream gate, so marking certain tests as valid-but-skipped-in-the-gate seems like an acceptable solution to me. (I'm not a maintainer of the Tempest repo so please take my opinion with a grain of salt). Cheers, Joe __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Skip-only tests
On 2016-09-08 10:23, Toni Freger wrote: > On Thu, Sep 8, 2016 at 10:52 AM, Andreas Jaeger wrote: >> >> This reminds me of this review for a nova-tempest-staging repository: >> https://review.openstack.org/366091 >> >> Joe seems to face similar challenges that you articulate, John, >> >> Andreas >> > > Hi Andreas, > > I understand other projects are going in the direction of using a > different repo for these skip-only tests. This seems > counter-productive for us for the following reasons: Toni, sorry, I wasn't clear. I'm not advocating for this, I wanted to bring Joe into this discussion to see whether the solution that John suggested will help them as well - I wanted to broaden the scope of this since it's not only neutron that faces this. If you review my comments on that linked review, you see that I'm not in favor of a separate repository at all. > 1. We believe that eventually some of these pre-conditions John > mentioned will be solved, thus requiring even more patches to migrate > tests back to the Neutron repos. > 2. Since the infra is still maturing (and since new features are > written all the time), contributing tests will eventually require > multiple patches in multiple repos. For example, adding a test for > VLAN Aware VMs will probably require making changes to the current > tempest infrastructure (which resides in the Neutron repo), and then > additional "Depends-On" patches in the new repo. > > These will only slow down the process which can be made fluent if > tests will be all be done in the same repo as the actual features and > infra. Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Skip-only tests
On Thu, Sep 8, 2016 at 10:52 AM, Andreas Jaeger wrote: > > This reminds me of this review for a nova-tempest-staging repository: > https://review.openstack.org/366091 > > Joe seems to face similar challenges that you articulate, John, > > Andreas > Hi Andreas, I understand other projects are going in the direction of using a different repo for these skip-only tests. This seems counter-productive for us for the following reasons: 1. We believe that eventually some of these pre-conditions John mentioned will be solved, thus requiring even more patches to migrate tests back to the Neutron repos. 2. Since the infra is still maturing (and since new features are written all the time), contributing tests will eventually require multiple patches in multiple repos. For example, adding a test for VLAN Aware VMs will probably require making changes to the current tempest infrastructure (which resides in the Neutron repo), and then additional "Depends-On" patches in the new repo. These will only slow down the process which can be made fluent if tests will be all be done in the same repo as the actual features and infra. Thanks, Toni. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] Skip-only tests
On 2016-09-08 09:38, John Schwarz wrote: > Hi all, > > The Neutron community has recently started contributing Tempest > scenario tests in the Neutron tree and I'd like to discuss a general > issue we're hitting. Mainly, we're talking about required hardware for > some features and/or VM images that support more features, which makes > it hard (or impossible) to test certain features at the gate > end-to-end. Specifically, tests that require SR-IOV or OVS-DPDK, and > tests relating to VLAN Aware VMs. > > For this reason, we would like to bring up the idea of introducing in > a new concept to the "testing arena" - skip-only tests. These will be > tests that will be merged upstream and skipped unless some > pre-conditions are met (example: "@skip_unless(has_sriov=True)"). The > key idea here though, is that we don't expect upstream gates to add > support for these pre-conditions any time soon. Instead, the tests > will be merged upstream and used on the various downstream gates most > of us have (with specific hardware/deployment that match our > respective interests). > > Another way to look at it is: "I want to contribute tests that require > specific hardware/software/deployments, and I want it to be robust and > shared amongst the community as per the Open Source way, even though > it won't be actually run on the upstream gates". Obviously, we would > continue to investigate ways to "unskip" these kind of tests where > possible. > [...] This reminds me of this review for a nova-tempest-staging repository: https://review.openstack.org/366091 Joe seems to face similar challenges that you articulate, John, Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev