Re: [openstack-dev] [tempest] How to run tempest tests

[GHANSHYAM MANN]

Tests can be skipped based on configuration file options. Its all depend on
what all set of tests you want to skip for your environment.

For example- to skip all sahara tests (as you mentioned)

You can make config option - 'sahara' to false which will skip all sahara
tests.

Particular feature of services can be skipped by same way if those have
specific configuration options (like IPv6) or through extension list
''api_extension" of specific service.



On Fri, Nov 28, 2014 at 6:24 PM, om prakash pandey 
wrote:

> Thanks Sridhar.
>
> I'm aware of using "skipException" for skipping tests at class level.
> However, this involves putting it in every class to skip tests which are
> not desired.
>
> I was looking for a way to control the tests I want to run through some
> kind of a configuration file, the options to pass to test runner.
>
> Regards
> Om
>
> On Fri, Nov 28, 2014 at 2:02 PM, Sridhar Gaddam <
> sridhar.gad...@enovance.com> wrote:
>
>>  If the deployment does not support IPv6, we use the following convention
>> to skip the tests at class level.
>>
>> https://github.com/openstack/tempest/blob/master/tempest/api/network/base.py#L65
>>
>> Regards,
>> --Sridhar.
>>
>>
>>
>> On 11/28/2014 01:50 PM, om prakash pandey wrote:
>>
>> Hi Folks,
>>
>>  I would like to know about the "best practices" followed for skipping
>> tests not applicable for my environment.
>>
>>  I know one of the ways is to use the below decorator over the test
>> method:
>>  @test.skip_because(bug="BUG_ID")
>>
>>  However, what if my deployment doesn't support VPNAAS and I want to
>> skip those tests. Similarly, what if I want to skip the entire suite of
>> sahara(data processing) tests.
>>
>>  Are there any options in testr to customize running of tempest tests as
>> per my environment/requirements?
>>
>>  Regards,
>> Om
>>
>> On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon 
>> wrote:
>>
>>> Hi,
>>> Thanjs for clearing that up... I had a hard time understanding the
>>> screws before I went with testr.
>>>
>>> Regards,
>>> Vineet
>>>  On 25 Nov 2014 17:46, "Matthew Treinish"  wrote:
>>>
  On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
 > Sorry for my previous message with wrong subject
 >
 > Hi all,
 > By reading the tempest documentation page [1] a user can run tempest
 tests
 > by using whether testr or run_tempest.sh or tox.
 >
 > What is the best practice?
 > run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it
 is my
 > preferred way, currently.
 > Any thought?

 So the options are there for different reasons and fit different
 purposes. The
 run_tempest.sh script exists mostly for legacy reasons as some people
 prefer to
 use it, and it predates the usage of tox in tempest. It also has some
 advantages
 like that it can run without a venv and provides some other options.

 Tox is what we use for gating, and we keep most of job definitions for
 gating in
 the tox.ini file. If you're trying to reproduce a gate run locally
 using tox is
 what is recommended to use. Personally I use it to run everything just
 because
 I often mix unit tests and tempest runs and I like having separate
 venvs for
 both being created on demand.

 Calling testr directly is just what all of these tools are doing under
 the
 covers, and it'll always be an option.

 One thing we're looking to do this cycle is to add a single entry point
 for
 running tempest which will hopefully clear up this confusion, and make
 the
 interface for interacting with tempest a bit nicer. When this work is
 done, the
 run_tempest.sh script will most likely disappear and tox will probably
 just be
 used for gating job definitions and just call the new entry-point
 instead of
 testr directly.

 >
 > BR,
 > Angelo
 >
 > [1]
 http://docs.openstack.org/developer/tempest/overview.html#quickstart
 >

 -Matt Treinish

  ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> ___
>> OpenStack-dev mailing 
>> listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http:

Re: [openstack-dev] [tempest] How to run tempest tests

[om prakash pandey]

Thanks Sridhar.

I'm aware of using "skipException" for skipping tests at class level.
However, this involves putting it in every class to skip tests which are
not desired.

I was looking for a way to control the tests I want to run through some
kind of a configuration file, the options to pass to test runner.

Regards
Om

On Fri, Nov 28, 2014 at 2:02 PM, Sridhar Gaddam  wrote:

>  If the deployment does not support IPv6, we use the following convention
> to skip the tests at class level.
>
> https://github.com/openstack/tempest/blob/master/tempest/api/network/base.py#L65
>
> Regards,
> --Sridhar.
>
>
>
> On 11/28/2014 01:50 PM, om prakash pandey wrote:
>
> Hi Folks,
>
>  I would like to know about the "best practices" followed for skipping
> tests not applicable for my environment.
>
>  I know one of the ways is to use the below decorator over the test
> method:
>  @test.skip_because(bug="BUG_ID")
>
>  However, what if my deployment doesn't support VPNAAS and I want to skip
> those tests. Similarly, what if I want to skip the entire suite of
> sahara(data processing) tests.
>
>  Are there any options in testr to customize running of tempest tests as
> per my environment/requirements?
>
>  Regards,
> Om
>
> On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon 
> wrote:
>
>> Hi,
>> Thanjs for clearing that up... I had a hard time understanding the screws
>> before I went with testr.
>>
>> Regards,
>> Vineet
>>  On 25 Nov 2014 17:46, "Matthew Treinish"  wrote:
>>
>>>  On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
>>> > Sorry for my previous message with wrong subject
>>> >
>>> > Hi all,
>>> > By reading the tempest documentation page [1] a user can run tempest
>>> tests
>>> > by using whether testr or run_tempest.sh or tox.
>>> >
>>> > What is the best practice?
>>> > run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it
>>> is my
>>> > preferred way, currently.
>>> > Any thought?
>>>
>>> So the options are there for different reasons and fit different
>>> purposes. The
>>> run_tempest.sh script exists mostly for legacy reasons as some people
>>> prefer to
>>> use it, and it predates the usage of tox in tempest. It also has some
>>> advantages
>>> like that it can run without a venv and provides some other options.
>>>
>>> Tox is what we use for gating, and we keep most of job definitions for
>>> gating in
>>> the tox.ini file. If you're trying to reproduce a gate run locally using
>>> tox is
>>> what is recommended to use. Personally I use it to run everything just
>>> because
>>> I often mix unit tests and tempest runs and I like having separate venvs
>>> for
>>> both being created on demand.
>>>
>>> Calling testr directly is just what all of these tools are doing under
>>> the
>>> covers, and it'll always be an option.
>>>
>>> One thing we're looking to do this cycle is to add a single entry point
>>> for
>>> running tempest which will hopefully clear up this confusion, and make
>>> the
>>> interface for interacting with tempest a bit nicer. When this work is
>>> done, the
>>> run_tempest.sh script will most likely disappear and tox will probably
>>> just be
>>> used for gating job definitions and just call the new entry-point
>>> instead of
>>> testr directly.
>>>
>>> >
>>> > BR,
>>> > Angelo
>>> >
>>> > [1]
>>> http://docs.openstack.org/developer/tempest/overview.html#quickstart
>>> >
>>>
>>> -Matt Treinish
>>>
>>>  ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> ___
> OpenStack-dev mailing 
> listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] How to run tempest tests

[Sridhar Gaddam]

If the deployment does not support IPv6, we use the following convention 
to skip the tests at class level.

https://github.com/openstack/tempest/blob/master/tempest/api/network/base.py#L65

Regards,
--Sridhar.


On 11/28/2014 01:50 PM, om prakash pandey wrote:

Hi Folks,

I would like to know about the "best practices" followed for skipping 
tests not applicable for my environment.


I know one of the ways is to use the below decorator over the test method:
 @test.skip_because(bug="BUG_ID")

However, what if my deployment doesn't support VPNAAS and I want to 
skip those tests. Similarly, what if I want to skip the entire suite 
of sahara(data processing) tests.


Are there any options in testr to customize running of tempest tests 
as per my environment/requirements?


Regards,
Om

On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon > wrote:


Hi,
Thanjs for clearing that up... I had a hard time understanding the
screws before I went with testr.

Regards,
Vineet

On 25 Nov 2014 17:46, "Matthew Treinish" mailto:mtrein...@kortar.org>> wrote:

On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
> Sorry for my previous message with wrong subject
>
> Hi all,
> By reading the tempest documentation page [1] a user can run
tempest tests
> by using whether testr or run_tempest.sh or tox.
>
> What is the best practice?
> run_tempest.sh has several options (e.g. ./run_tempest.sh
-h) and it is my
> preferred way, currently.
> Any thought?

So the options are there for different reasons and fit
different purposes. The
run_tempest.sh script exists mostly for legacy reasons as some
people prefer to
use it, and it predates the usage of tox in tempest. It also
has some advantages
like that it can run without a venv and provides some other
options.

Tox is what we use for gating, and we keep most of job
definitions for gating in
the tox.ini file. If you're trying to reproduce a gate run
locally using tox is
what is recommended to use. Personally I use it to run
everything just because
I often mix unit tests and tempest runs and I like having
separate venvs for
both being created on demand.

Calling testr directly is just what all of these tools are
doing under the
covers, and it'll always be an option.

One thing we're looking to do this cycle is to add a single
entry point for
running tempest which will hopefully clear up this confusion,
and make the
interface for interacting with tempest a bit nicer. When this
work is done, the
run_tempest.sh script will most likely disappear and tox will
probably just be
used for gating job definitions and just call the new
entry-point instead of
testr directly.

>
> BR,
> Angelo
>
> [1]
http://docs.openstack.org/developer/tempest/overview.html#quickstart
>

-Matt Treinish

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] How to run tempest tests

[om prakash pandey]

Hi Folks,

I would like to know about the "best practices" followed for skipping tests
not applicable for my environment.

I know one of the ways is to use the below decorator over the test method:
 @test.skip_because(bug="BUG_ID")

However, what if my deployment doesn't support VPNAAS and I want to skip
those tests. Similarly, what if I want to skip the entire suite of
sahara(data processing) tests.

Are there any options in testr to customize running of tempest tests as per
my environment/requirements?

Regards,
Om

On Wed, Nov 26, 2014 at 3:13 AM, Vineet Menon 
wrote:

> Hi,
> Thanjs for clearing that up... I had a hard time understanding the screws
> before I went with testr.
>
> Regards,
> Vineet
> On 25 Nov 2014 17:46, "Matthew Treinish"  wrote:
>
>> On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
>> > Sorry for my previous message with wrong subject
>> >
>> > Hi all,
>> > By reading the tempest documentation page [1] a user can run tempest
>> tests
>> > by using whether testr or run_tempest.sh or tox.
>> >
>> > What is the best practice?
>> > run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is
>> my
>> > preferred way, currently.
>> > Any thought?
>>
>> So the options are there for different reasons and fit different
>> purposes. The
>> run_tempest.sh script exists mostly for legacy reasons as some people
>> prefer to
>> use it, and it predates the usage of tox in tempest. It also has some
>> advantages
>> like that it can run without a venv and provides some other options.
>>
>> Tox is what we use for gating, and we keep most of job definitions for
>> gating in
>> the tox.ini file. If you're trying to reproduce a gate run locally using
>> tox is
>> what is recommended to use. Personally I use it to run everything just
>> because
>> I often mix unit tests and tempest runs and I like having separate venvs
>> for
>> both being created on demand.
>>
>> Calling testr directly is just what all of these tools are doing under the
>> covers, and it'll always be an option.
>>
>> One thing we're looking to do this cycle is to add a single entry point
>> for
>> running tempest which will hopefully clear up this confusion, and make the
>> interface for interacting with tempest a bit nicer. When this work is
>> done, the
>> run_tempest.sh script will most likely disappear and tox will probably
>> just be
>> used for gating job definitions and just call the new entry-point instead
>> of
>> testr directly.
>>
>> >
>> > BR,
>> > Angelo
>> >
>> > [1]
>> http://docs.openstack.org/developer/tempest/overview.html#quickstart
>> >
>>
>> -Matt Treinish
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] How to run tempest tests

[Vineet Menon]

Hi,
Thanjs for clearing that up... I had a hard time understanding the screws
before I went with testr.

Regards,
Vineet
On 25 Nov 2014 17:46, "Matthew Treinish"  wrote:

> On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
> > Sorry for my previous message with wrong subject
> >
> > Hi all,
> > By reading the tempest documentation page [1] a user can run tempest
> tests
> > by using whether testr or run_tempest.sh or tox.
> >
> > What is the best practice?
> > run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is
> my
> > preferred way, currently.
> > Any thought?
>
> So the options are there for different reasons and fit different purposes.
> The
> run_tempest.sh script exists mostly for legacy reasons as some people
> prefer to
> use it, and it predates the usage of tox in tempest. It also has some
> advantages
> like that it can run without a venv and provides some other options.
>
> Tox is what we use for gating, and we keep most of job definitions for
> gating in
> the tox.ini file. If you're trying to reproduce a gate run locally using
> tox is
> what is recommended to use. Personally I use it to run everything just
> because
> I often mix unit tests and tempest runs and I like having separate venvs
> for
> both being created on demand.
>
> Calling testr directly is just what all of these tools are doing under the
> covers, and it'll always be an option.
>
> One thing we're looking to do this cycle is to add a single entry point for
> running tempest which will hopefully clear up this confusion, and make the
> interface for interacting with tempest a bit nicer. When this work is
> done, the
> run_tempest.sh script will most likely disappear and tox will probably
> just be
> used for gating job definitions and just call the new entry-point instead
> of
> testr directly.
>
> >
> > BR,
> > Angelo
> >
> > [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart
> >
>
> -Matt Treinish
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] How to run tempest tests

[Matthew Treinish]

On Mon, Nov 24, 2014 at 10:49:27AM +0100, Angelo Matarazzo wrote:
> Sorry for my previous message with wrong subject
> 
> Hi all,
> By reading the tempest documentation page [1] a user can run tempest tests
> by using whether testr or run_tempest.sh or tox.
> 
> What is the best practice?
> run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is my
> preferred way, currently.
> Any thought?

So the options are there for different reasons and fit different purposes. The
run_tempest.sh script exists mostly for legacy reasons as some people prefer to
use it, and it predates the usage of tox in tempest. It also has some advantages
like that it can run without a venv and provides some other options.

Tox is what we use for gating, and we keep most of job definitions for gating in
the tox.ini file. If you're trying to reproduce a gate run locally using tox is
what is recommended to use. Personally I use it to run everything just because
I often mix unit tests and tempest runs and I like having separate venvs for
both being created on demand.

Calling testr directly is just what all of these tools are doing under the
covers, and it'll always be an option. 

One thing we're looking to do this cycle is to add a single entry point for
running tempest which will hopefully clear up this confusion, and make the
interface for interacting with tempest a bit nicer. When this work is done, the
run_tempest.sh script will most likely disappear and tox will probably just be
used for gating job definitions and just call the new entry-point instead of
testr directly.

> 
> BR,
> Angelo
> 
> [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart
> 

-Matt Treinish


pgpccVuLwrFTD.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] How to run tempest tests

[Boris Pavlovic]

Angelo,


One more way to run Tempest is to run it via Rally.
Rally will take care about installation, generating tempest.conf, running
tempest, parsing & storing output results.

Here is manual:
https://www.mirantis.com/blog/rally-openstack-tempest-testing-made-simpler/

As a bonus you'll get ability to compare results of 2 runs (if you need
that)

Best regards,
Boris Pavlovic



On Mon, Nov 24, 2014 at 6:05 PM, Vineet Menon 
wrote:

> Hi,
>
> I cannot comment on the best practice.
>
> But I can point to you a few more methods and links.
>
> 1. https://dague.net//presentations/tempest-101/#/
> 2.
> http://www.slideshare.net/kamesh001/open-stack-qa-and-tempest?next_slideshow=1
> 3.
> https://docs.google.com/presentation/d/1M3XhAco_0u7NZQn3Gz53z9VOHHrkQBzEs5gt43ZvhOc/edit#slide=id.p
>
>
>
> Regards,
>
> Vineet Menon
>
>
> On 24 November 2014 at 10:49, Angelo Matarazzo <
> angelo.matara...@dektech.com.au> wrote:
>
>> Sorry for my previous message with wrong subject
>>
>> Hi all,
>> By reading the tempest documentation page [1] a user can run tempest
>> tests by using whether testr or run_tempest.sh or tox.
>>
>> What is the best practice?
>> run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is
>> my preferred way, currently.
>> Any thought?
>>
>> BR,
>> Angelo
>>
>> [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tempest] How to run tempest tests

[Vineet Menon]

Hi,

I cannot comment on the best practice.

But I can point to you a few more methods and links.

1. https://dague.net//presentations/tempest-101/#/
2.
http://www.slideshare.net/kamesh001/open-stack-qa-and-tempest?next_slideshow=1
3.
https://docs.google.com/presentation/d/1M3XhAco_0u7NZQn3Gz53z9VOHHrkQBzEs5gt43ZvhOc/edit#slide=id.p



Regards,

Vineet Menon


On 24 November 2014 at 10:49, Angelo Matarazzo <
angelo.matara...@dektech.com.au> wrote:

> Sorry for my previous message with wrong subject
>
> Hi all,
> By reading the tempest documentation page [1] a user can run tempest tests
> by using whether testr or run_tempest.sh or tox.
>
> What is the best practice?
> run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is my
> preferred way, currently.
> Any thought?
>
> BR,
> Angelo
>
> [1] http://docs.openstack.org/developer/tempest/overview.html#quickstart
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tempest] How to run tempest tests

[Angelo Matarazzo]

Sorry for my previous message with wrong subject

Hi all,
By reading the tempest documentation page [1] a user can run tempest 
tests by using whether testr or run_tempest.sh or tox.


What is the best practice?
run_tempest.sh has several options (e.g. ./run_tempest.sh -h) and it is 
my preferred way, currently.

Any thought?

BR,
Angelo

[1] http://docs.openstack.org/developer/tempest/overview.html#quickstart

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev