Re: [openstack-dev] [tempest][qa][congress] tempest test conditioning on release version
On Tue, Aug 14, 2018 at 8:59 PM, Ghanshyam Mann wrote: > On Wed, 15 Aug 2018 06:40:57 +0900 Eric K > wrote > > Anyone have an example handy of a tempest test conditioning on service > > release version (because new features not available in past versions)? > > Seems like it could get pretty messy and haphazard, so I'm curious to > > see best practices. Thanks lots! > > Thanks Eric for query. We do it in many times in Tempest and similar approach > can be adopt by tempest plugins. There are 2 ways we can handle this- > > 1. Using feature flag. Tempest documentation is here [1]. > Step1- This is simply adding a config options(feature flag) for new/old > feature. > Example- https://review.openstack.org/#/c/545627/ > https://github.com/openstack/tempest/blob/6a8d495192632fd18dce4baf1a4b213f401a0167/tempest/config.py#L242 > Step2- Based on that flag you can skip the tests where that feature is not > available. > Example- > https://github.com/openstack/tempest/blob/d5058a8a9c8c1c5383699d04296087b6d5a24efd/tempest/api/identity/base.py#L315 > Step3- For gate, devstack plugin on project side (congress is your case [2]) > which is branch aware can set that flag to true and false based on which > branch that test is running. For tempest we do the same from > devstack/lib/tempest > Example - https://review.openstack.org/#/c/545680/ > https://github.com/openstack-dev/devstack/blob/8c1052001629d62f001d04c182500fa293858f47/lib/tempest#L308 > Step4- For cloud testing(non-gate), tester can manually configure the those > flag based on what service version they are testing. > > 2. Detecting service version via version API > - If you can get the service version info from API then you can use that > while skipping the tests. > - One example if for compute where based on microversion, it can be detected > that test running against which release. > - Example- > https://github.com/openstack/tempest/blob/d5058a8a9c8c1c5383699d04296087b6d5a24efd/tempest/api/compute/base.py#L114 > > > [1] > https://docs.openstack.org/tempest/latest/HACKING.html#branchless-tempest-considerations > [2] > https://github.com/openstack/congress/blob/014361c809517661264d0364eaf1e261e449ea80/devstack/plugin.sh#L88 > > > > > Eric Kao Thank you so much, Ghanshyam! __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tempest][qa][congress] tempest test conditioning on release version
On Wed, 15 Aug 2018 06:40:57 +0900 Eric K wrote > Anyone have an example handy of a tempest test conditioning on service > release version (because new features not available in past versions)? > Seems like it could get pretty messy and haphazard, so I'm curious to > see best practices. Thanks lots! Thanks Eric for query. We do it in many times in Tempest and similar approach can be adopt by tempest plugins. There are 2 ways we can handle this- 1. Using feature flag. Tempest documentation is here [1]. Step1- This is simply adding a config options(feature flag) for new/old feature. Example- https://review.openstack.org/#/c/545627/ https://github.com/openstack/tempest/blob/6a8d495192632fd18dce4baf1a4b213f401a0167/tempest/config.py#L242 Step2- Based on that flag you can skip the tests where that feature is not available. Example- https://github.com/openstack/tempest/blob/d5058a8a9c8c1c5383699d04296087b6d5a24efd/tempest/api/identity/base.py#L315 Step3- For gate, devstack plugin on project side (congress is your case [2]) which is branch aware can set that flag to true and false based on which branch that test is running. For tempest we do the same from devstack/lib/tempest Example - https://review.openstack.org/#/c/545680/ https://github.com/openstack-dev/devstack/blob/8c1052001629d62f001d04c182500fa293858f47/lib/tempest#L308 Step4- For cloud testing(non-gate), tester can manually configure the those flag based on what service version they are testing. 2. Detecting service version via version API - If you can get the service version info from API then you can use that while skipping the tests. - One example if for compute where based on microversion, it can be detected that test running against which release. - Example- https://github.com/openstack/tempest/blob/d5058a8a9c8c1c5383699d04296087b6d5a24efd/tempest/api/compute/base.py#L114 [1] https://docs.openstack.org/tempest/latest/HACKING.html#branchless-tempest-considerations [2] https://github.com/openstack/congress/blob/014361c809517661264d0364eaf1e261e449ea80/devstack/plugin.sh#L88 > > Eric Kao > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tempest][qa][congress] tempest test conditioning on release version
Anyone have an example handy of a tempest test conditioning on service release version (because new features not available in past versions)? Seems like it could get pretty messy and haphazard, so I'm curious to see best practices. Thanks lots! Eric Kao __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
