Re: [openstack-dev] [tempest] How to run tempest tests

2014-11-28 Thread om prakash pandey
Hi Folks,

I would like to know about the best practices followed for skipping tests
not applicable for my environment.

I know one of the ways is to use the below decorator over the test method:
 @test.skip_because(bug=BUG_ID)

However, what if my deployment doesn't support VPNAAS and I want to skip
those tests. Similarly, what if I want to skip the entire suite of
sahara(data processing) tests.

Are there any options in testr to customize running of tempest tests as per
my environment/requirements?

Regards,
Om

On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon mvineetme...@gmail.com
wrote:

 Hi,
 Thanjs for clearing that up... I had a hard time understanding the screws
 before I went with testr.

 Regards,
 Vineet
 On 25 Nov 2014 17:46, Matthew Treinish mtrein...@kortar.org wrote:

 On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
  Sorry for my previous message with wrong subject
 
  Hi all,
  By reading the tempest documentation page [1] a user can run tempest
 tests
  by using whether testr or run_tempest.sh or tox.
 
  What is the best practice?
  run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is
 my
  preferred way, currently.
  Any thought?

 So the options are there for different reasons and fit different
 purposes. The
 run_tempest.sh script exists mostly for legacy reasons as some people
 prefer to
 use it, and it predates the usage of tox in tempest. It also has some
 advantages
 like that it can run without a venv and provides some other options.

 Tox is what we use for gating, and we keep most of job definitions for
 gating in
 the tox.ini file. If you're trying to reproduce a gate run locally using
 tox is
 what is recommended to use. Personally I use it to run everything just
 because
 I often mix unit tests and tempest runs and I like having separate venvs
 for
 both being created on demand.

 Calling testr directly is just what all of these tools are doing under the
 covers, and it'll always be an option.

 One thing we're looking to do this cycle is to add a single entry point
 for
 running tempest which will hopefully clear up this confusion, and make the
 interface for interacting with tempest a bit nicer. When this work is
 done, the
 run_tempest.sh script will most likely disappear and tox will probably
 just be
 used for gating job definitions and just call the new entry-point instead
 of
 testr directly.

 
  BR,
  Angelo
 
  [1]
 http://docs.openstack.org/developer/tempest/overview.html#quickstart
 

 -Matt Treinish

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tempest] How to run tempest tests

2014-11-28 Thread Sridhar Gaddam
If the deployment does not support IPv6, we use the following convention 
to skip the tests at class level.

https://github.com/openstack/tempest/blob/master/tempest/api/network/base.py#L65

Regards,
--Sridhar.


On 11/28/2014 01:50 PM, om prakash pandey wrote:

Hi Folks,

I would like to know about the best practices followed for skipping 
tests not applicable for my environment.


I know one of the ways is to use the below decorator over the test method:
 @test.skip_because(bug=BUG_ID)

However, what if my deployment doesn't support VPNAAS and I want to 
skip those tests. Similarly, what if I want to skip the entire suite 
of sahara(data processing) tests.


Are there any options in testr to customize running of tempest tests 
as per my environment/requirements?


Regards,
Om

On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon mvineetme...@gmail.com 
mailto:mvineetme...@gmail.com wrote:


Hi,
Thanjs for clearing that up... I had a hard time understanding the
screws before I went with testr.

Regards,
Vineet

On 25 Nov 2014 17:46, Matthew Treinish mtrein...@kortar.org
mailto:mtrein...@kortar.org wrote:

On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
 Sorry for my previous message with wrong subject

 Hi all,
 By reading the tempest documentation page [1] a user can run
tempest tests
 by using whether testr or run_tempest.sh or tox.

 What is the best practice?
 run_tempest.sh has several options (e.g. ./run_tempest.sh
-h) and it is my
 preferred way, currently.
 Any thought?

So the options are there for different reasons and fit
different purposes. The
run_tempest.sh script exists mostly for legacy reasons as some
people prefer to
use it, and it predates the usage of tox in tempest. It also
has some advantages
like that it can run without a venv and provides some other
options.

Tox is what we use for gating, and we keep most of job
definitions for gating in
the tox.ini file. If you're trying to reproduce a gate run
locally using tox is
what is recommended to use. Personally I use it to run
everything just because
I often mix unit tests and tempest runs and I like having
separate venvs for
both being created on demand.

Calling testr directly is just what all of these tools are
doing under the
covers, and it'll always be an option.

One thing we're looking to do this cycle is to add a single
entry point for
running tempest which will hopefully clear up this confusion,
and make the
interface for interacting with tempest a bit nicer. When this
work is done, the
run_tempest.sh script will most likely disappear and tox will
probably just be
used for gating job definitions and just call the new
entry-point instead of
testr directly.


 BR,
 Angelo

 [1]
http://docs.openstack.org/developer/tempest/overview.html#quickstart


-Matt Treinish

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tempest] How to run tempest tests

2014-11-28 Thread om prakash pandey
Thanks Sridhar.

I'm aware of using skipException for skipping tests at class level.
However, this involves putting it in every class to skip tests which are
not desired.

I was looking for a way to control the tests I want to run through some
kind of a configuration file, the options to pass to test runner.

Regards
Om

On Fri, Nov 28, 2014 at 2:02 PM, Sridhar Gaddam sridhar.gad...@enovance.com
 wrote:

  If the deployment does not support IPv6, we use the following convention
 to skip the tests at class level.

 https://github.com/openstack/tempest/blob/master/tempest/api/network/base.py#L65

 Regards,
 --Sridhar.



 On 11/28/2014 01:50 PM, om prakash pandey wrote:

 Hi Folks,

  I would like to know about the best practices followed for skipping
 tests not applicable for my environment.

  I know one of the ways is to use the below decorator over the test
 method:
  @test.skip_because(bug=BUG_ID)

  However, what if my deployment doesn't support VPNAAS and I want to skip
 those tests. Similarly, what if I want to skip the entire suite of
 sahara(data processing) tests.

  Are there any options in testr to customize running of tempest tests as
 per my environment/requirements?

  Regards,
 Om

 On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon mvineetme...@gmail.com
 wrote:

 Hi,
 Thanjs for clearing that up... I had a hard time understanding the screws
 before I went with testr.

 Regards,
 Vineet
  On 25 Nov 2014 17:46, Matthew Treinish mtrein...@kortar.org wrote:

  On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
  Sorry for my previous message with wrong subject
 
  Hi all,
  By reading the tempest documentation page [1] a user can run tempest
 tests
  by using whether testr or run_tempest.sh or tox.
 
  What is the best practice?
  run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it
 is my
  preferred way, currently.
  Any thought?

 So the options are there for different reasons and fit different
 purposes. The
 run_tempest.sh script exists mostly for legacy reasons as some people
 prefer to
 use it, and it predates the usage of tox in tempest. It also has some
 advantages
 like that it can run without a venv and provides some other options.

 Tox is what we use for gating, and we keep most of job definitions for
 gating in
 the tox.ini file. If you're trying to reproduce a gate run locally using
 tox is
 what is recommended to use. Personally I use it to run everything just
 because
 I often mix unit tests and tempest runs and I like having separate venvs
 for
 both being created on demand.

 Calling testr directly is just what all of these tools are doing under
 the
 covers, and it'll always be an option.

 One thing we're looking to do this cycle is to add a single entry point
 for
 running tempest which will hopefully clear up this confusion, and make
 the
 interface for interacting with tempest a bit nicer. When this work is
 done, the
 run_tempest.sh script will most likely disappear and tox will probably
 just be
 used for gating job definitions and just call the new entry-point
 instead of
 testr directly.

 
  BR,
  Angelo
 
  [1]
 http://docs.openstack.org/developer/tempest/overview.html#quickstart
 

 -Matt Treinish

  ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 ___
 OpenStack-dev mailing 
 listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tempest] How to run tempest tests

2014-11-28 Thread GHANSHYAM MANN
Tests can be skipped based on configuration file options. Its all depend on
what all set of tests you want to skip for your environment.

For example- to skip all sahara tests (as you mentioned)

You can make config option - 'sahara' to false which will skip all sahara
tests.

Particular feature of services can be skipped by same way if those have
specific configuration options (like IPv6) or through extension list
''api_extension of specific service.



On Fri, Nov 28, 2014 at 6:24 PM, om prakash pandey pande...@gmail.com
wrote:

 Thanks Sridhar.

 I'm aware of using skipException for skipping tests at class level.
 However, this involves putting it in every class to skip tests which are
 not desired.

 I was looking for a way to control the tests I want to run through some
 kind of a configuration file, the options to pass to test runner.

 Regards
 Om

 On Fri, Nov 28, 2014 at 2:02 PM, Sridhar Gaddam 
 sridhar.gad...@enovance.com wrote:

  If the deployment does not support IPv6, we use the following convention
 to skip the tests at class level.

 https://github.com/openstack/tempest/blob/master/tempest/api/network/base.py#L65

 Regards,
 --Sridhar.



 On 11/28/2014 01:50 PM, om prakash pandey wrote:

 Hi Folks,

  I would like to know about the best practices followed for skipping
 tests not applicable for my environment.

  I know one of the ways is to use the below decorator over the test
 method:
  @test.skip_because(bug=BUG_ID)

  However, what if my deployment doesn't support VPNAAS and I want to
 skip those tests. Similarly, what if I want to skip the entire suite of
 sahara(data processing) tests.

  Are there any options in testr to customize running of tempest tests as
 per my environment/requirements?

  Regards,
 Om

 On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon mvineetme...@gmail.com
 wrote:

 Hi,
 Thanjs for clearing that up... I had a hard time understanding the
 screws before I went with testr.

 Regards,
 Vineet
  On 25 Nov 2014 17:46, Matthew Treinish mtrein...@kortar.org wrote:

  On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
  Sorry for my previous message with wrong subject
 
  Hi all,
  By reading the tempest documentation page [1] a user can run tempest
 tests
  by using whether testr or run_tempest.sh or tox.
 
  What is the best practice?
  run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it
 is my
  preferred way, currently.
  Any thought?

 So the options are there for different reasons and fit different
 purposes. The
 run_tempest.sh script exists mostly for legacy reasons as some people
 prefer to
 use it, and it predates the usage of tox in tempest. It also has some
 advantages
 like that it can run without a venv and provides some other options.

 Tox is what we use for gating, and we keep most of job definitions for
 gating in
 the tox.ini file. If you're trying to reproduce a gate run locally
 using tox is
 what is recommended to use. Personally I use it to run everything just
 because
 I often mix unit tests and tempest runs and I like having separate
 venvs for
 both being created on demand.

 Calling testr directly is just what all of these tools are doing under
 the
 covers, and it'll always be an option.

 One thing we're looking to do this cycle is to add a single entry point
 for
 running tempest which will hopefully clear up this confusion, and make
 the
 interface for interacting with tempest a bit nicer. When this work is
 done, the
 run_tempest.sh script will most likely disappear and tox will probably
 just be
 used for gating job definitions and just call the new entry-point
 instead of
 testr directly.

 
  BR,
  Angelo
 
  [1]
 http://docs.openstack.org/developer/tempest/overview.html#quickstart
 

 -Matt Treinish

  ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 ___
 OpenStack-dev mailing 
 listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Thanks  Regards
Ghanshyam Mann
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tempest] How to run tempest tests

2014-11-25 Thread Matthew Treinish
On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
 Sorry for my previous message with wrong subject
 
 Hi all,
 By reading the tempest documentation page [1] a user can run tempest tests
 by using whether testr or run_tempest.sh or tox.
 
 What is the best practice?
 run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is my
 preferred way, currently.
 Any thought?

So the options are there for different reasons and fit different purposes. The
run_tempest.sh script exists mostly for legacy reasons as some people prefer to
use it, and it predates the usage of tox in tempest. It also has some advantages
like that it can run without a venv and provides some other options.

Tox is what we use for gating, and we keep most of job definitions for gating in
the tox.ini file. If you're trying to reproduce a gate run locally using tox is
what is recommended to use. Personally I use it to run everything just because
I often mix unit tests and tempest runs and I like having separate venvs for
both being created on demand.

Calling testr directly is just what all of these tools are doing under the
covers, and it'll always be an option. 

One thing we're looking to do this cycle is to add a single entry point for
running tempest which will hopefully clear up this confusion, and make the
interface for interacting with tempest a bit nicer. When this work is done, the
run_tempest.sh script will most likely disappear and tox will probably just be
used for gating job definitions and just call the new entry-point instead of
testr directly.

 
 BR,
 Angelo
 
 [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart
 

-Matt Treinish


pgpccVuLwrFTD.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tempest] How to run tempest tests

2014-11-25 Thread Vineet Menon
Hi,
Thanjs for clearing that up... I had a hard time understanding the screws
before I went with testr.

Regards,
Vineet
On 25 Nov 2014 17:46, Matthew Treinish mtrein...@kortar.org wrote:

 On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
  Sorry for my previous message with wrong subject
 
  Hi all,
  By reading the tempest documentation page [1] a user can run tempest
 tests
  by using whether testr or run_tempest.sh or tox.
 
  What is the best practice?
  run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is
 my
  preferred way, currently.
  Any thought?

 So the options are there for different reasons and fit different purposes.
 The
 run_tempest.sh script exists mostly for legacy reasons as some people
 prefer to
 use it, and it predates the usage of tox in tempest. It also has some
 advantages
 like that it can run without a venv and provides some other options.

 Tox is what we use for gating, and we keep most of job definitions for
 gating in
 the tox.ini file. If you're trying to reproduce a gate run locally using
 tox is
 what is recommended to use. Personally I use it to run everything just
 because
 I often mix unit tests and tempest runs and I like having separate venvs
 for
 both being created on demand.

 Calling testr directly is just what all of these tools are doing under the
 covers, and it'll always be an option.

 One thing we're looking to do this cycle is to add a single entry point for
 running tempest which will hopefully clear up this confusion, and make the
 interface for interacting with tempest a bit nicer. When this work is
 done, the
 run_tempest.sh script will most likely disappear and tox will probably
 just be
 used for gating job definitions and just call the new entry-point instead
 of
 testr directly.

 
  BR,
  Angelo
 
  [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart
 

 -Matt Treinish

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tempest] How to run tempest tests

2014-11-24 Thread Vineet Menon
Hi,

I cannot comment on the best practice.

But I can point to you a few more methods and links.

1. https://dague.net//presentations/tempest-101/#/
2.
http://www.slideshare.net/kamesh001/open-stack-qa-and-tempest?next_slideshow=1
3.
https://docs.google.com/presentation/d/1M3XhAco_0u7NZQn3Gz53z9VOHHrkQBzEs5gt43ZvhOc/edit#slide=id.p



Regards,

Vineet Menon


On 24 November 2014 at 10:49, Angelo Matarazzo 
angelo.matara...@dektech.com.au wrote:

 Sorry for my previous message with wrong subject

 Hi all,
 By reading the tempest documentation page [1] a user can run tempest tests
 by using whether testr or run_tempest.sh or tox.

 What is the best practice?
 run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is my
 preferred way, currently.
 Any thought?

 BR,
 Angelo

 [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tempest] How to run tempest tests

2014-11-24 Thread Boris Pavlovic
Angelo,


One more way to run Tempest is to run it via Rally.
Rally will take care about installation, generating tempest.conf, running
tempest, parsing  storing output results.

Here is manual:
https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/

As a bonus you'll get ability to compare results of 2 runs (if you need
that)

Best regards,
Boris Pavlovic



On Mon, Nov 24, 2014 at 6:05 PM, Vineet Menon mvineetme...@gmail.com
wrote:

 Hi,

 I cannot comment on the best practice.

 But I can point to you a few more methods and links.

 1. https://dague.net//presentations/tempest-101/#/
 2.
 http://www.slideshare.net/kamesh001/open-stack-qa-and-tempest?next_slideshow=1
 3.
 https://docs.google.com/presentation/d/1M3XhAco_0u7NZQn3Gz53z9VOHHrkQBzEs5gt43ZvhOc/edit#slide=id.p



 Regards,

 Vineet Menon


 On 24 November 2014 at 10:49, Angelo Matarazzo 
 angelo.matara...@dektech.com.au wrote:

 Sorry for my previous message with wrong subject

 Hi all,
 By reading the tempest documentation page [1] a user can run tempest
 tests by using whether testr or run_tempest.sh or tox.

 What is the best practice?
 run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is
 my preferred way, currently.
 Any thought?

 BR,
 Angelo

 [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev