Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[David_Paterson]

Dell - Internal Use - Confidential
 I don’t really understand the friction, it’s incredibly useful, especially for 
new users.

Use it or don’t use it it’s up to you,  nowhere is it implied that the user is 
forced to configure tempest with the tool.

Without a tool like this it is complex for new users to get up and going with 
tempest.  It is also more difficult for other parties to integrate with 
Tempest, Rally had to write their own configuration tooling for instance.

Thanks,
dp



From: Andrea Frittoli [mailto:andrea.fritt...@gmail.com]
Sent: Monday, November 30, 2015 8:34 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [QA][Tempest] Use tempest-config for 
tempest-cli-improvements

Looking at the tool, it seems to me that is servers a combination of functions:
- provision test resources
- support for distribution specific / cloud specific overrides to default
- support for configuration override via CLI
- discovery of configuration

Test resource provisioning is something that I agree is useful to have.
We plan in Mitaka to separate out of tempest.conf the configuration of test 
resources, and have a CLI tool to provision them [0]. We could re-use code from 
this tool to achieve that.

Support for distribution specific / cloud specific overrides is also something 
that is useful. In clouds where I control the deployment process I inject extra 
configs in tempest.conf based on the deployment options. In clouds where I 
don't, I maintain a partial tempest.conf with the list of options which I 
expects I have to modify to match the target cloud.

That's pretty easy to achieve though - simply append the extra configs to the 
bottom of tempest.conf and it's done. Duplicate configuration options are not 
an issue, the last one wins. Still we could support specifying a number of 
configuration files to the non-yet implemented "tempest run" command.

Support for configuration override via CLI is something that I can see it can 
be useful during development or troubleshooting, we could support that as 
options of the non-yet implemented "tempest run" command.

The last point is discovery - I believe we should use that only as we use it 
today in the gate - i.e. fail fast if the generated configuration does not 
match what can be discovered from the cloud.

So I would not get the script as is into tempest, but I think many of the 
functions implemented by it can fit into tempest - and some are already there.

andrea

[0] https://review.openstack.org/#/c/173334/

On Mon, Nov 30, 2015 at 7:39 AM Masayuki Igawa 
<masayuki.ig...@gmail.com<mailto:masayuki.ig...@gmail.com>> wrote:
Hi,

I agree with Ken'ichi's opinion, basically. Tempest users should know "what do 
we test for?" and we shouldn't discover values that we test for automatically.
If users seems that "My current cloud is good. This is what I expect.", 
discovering function could work. But I suppose many of users would use 
tempest-config-generator for a new their cloud. So I feel the tool users could 
be misunderstanding easily.

But I also think that tempest users don't need to know all of the 
configurations.
So, how about something like introducing "a configuration wizard" for tempest 
configuration?
This is a just idea, though..


Anyway, if you have the motivation to introduce tempest-config, how about 
writing a spec for the feature for a concrete discussion?
(I think we should have agreements about the target issues, users, solutions, 
etc.)

Best Regards,
-- Masayuki Igawa

2015年11月29日(日) 22:07 Yair Fried <yfr...@redhat.com<mailto:yfr...@redhat.com>>:
Hi,
I agree with Jordan.
We don't have to use the tool as part of the gate. It's target audience is 
people and not CI systems. More specifically - new users.
However, we could add a gate (or a few) for the tool that makes sure a proper 
conf file is generated. It doesn't have to run the tests, just compare the 
output of the script to the conf generated by devstack.

Re Rally - I believe the best place for tempest configuration script is within 
tempest. That said, if the Tempest community doesn't want this tool, we'll have 
to settle for the Rally solution.

Regards
Yair

On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier 
<jordan.pitt...@scality.com<mailto:jordan.pitt...@scality.com>> wrote:
Hi,
I think this script is valuable to some users: Rally and Red Hat expressed 
their needs, they seem clear.

This tool is far from bullet proof and if used blindly or in case of bugs, 
Tempest could be misconfigured. So, we could have this tool inside the Tempest 
repository (in the tools/) but not use it at all for the Gate.

I am not sure I fully understand the resistance for this, if we don"t use this 
config generator for the gate, what's the risk ?

Jordan

On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi 
<ken1ohmi...@gmail.com<mailto:ken1

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Andrea Frittoli]

Looking at the tool, it seems to me that is servers a combination of
functions:
- provision test resources
- support for distribution specific / cloud specific overrides to default
- support for configuration override via CLI
- discovery of configuration

Test resource provisioning is something that I agree is useful to have.
We plan in Mitaka to separate out of tempest.conf the configuration of test
resources, and have a CLI tool to provision them [0]. We could re-use code
from this tool to achieve that.

Support for distribution specific / cloud specific overrides is also
something that is useful. In clouds where I control the deployment process
I inject extra configs in tempest.conf based on the deployment options. In
clouds where I don't, I maintain a partial tempest.conf with the list of
options which I expects I have to modify to match the target cloud.

That's pretty easy to achieve though - simply append the extra configs to
the bottom of tempest.conf and it's done. Duplicate configuration options
are not an issue, the last one wins. Still we could support specifying a
number of configuration files to the non-yet implemented "tempest run"
command.

Support for configuration override via CLI is something that I can see it
can be useful during development or troubleshooting, we could support that
as options of the non-yet implemented "tempest run" command.

The last point is discovery - I believe we should use that only as we use
it today in the gate - i.e. fail fast if the generated configuration does
not match what can be discovered from the cloud.

So I would not get the script as is into tempest, but I think many of the
functions implemented by it can fit into tempest - and some are already
there.

andrea

[0] https://review.openstack.org/#/c/173334/

On Mon, Nov 30, 2015 at 7:39 AM Masayuki Igawa 
wrote:

> Hi,
>
> I agree with Ken'ichi's opinion, basically. Tempest users should know
> "what do we test for?" and we shouldn't discover values that we test for
> automatically.
> If users seems that "My current cloud is good. This is what I expect.",
> discovering function could work. But I suppose many of users would use
> tempest-config-generator for a new their cloud. So I feel the tool users
> could be misunderstanding easily.
>
> But I also think that tempest users don't need to know all of the
> configurations.
> So, how about something like introducing "a configuration wizard" for
> tempest configuration?
> This is a just idea, though..
>
>
> Anyway, if you have the motivation to introduce tempest-config, how about
> writing a spec for the feature for a concrete discussion?
> (I think we should have agreements about the target issues, users,
> solutions, etc.)
>
> Best Regards,
> -- Masayuki Igawa
>
>
> 2015年11月29日(日) 22:07 Yair Fried :
>
>> Hi,
>> I agree with Jordan.
>> We don't have to use the tool as part of the gate. It's target audience
>> is people and not CI systems. More specifically - new users.
>> However, we could add a gate (or a few) for the tool that makes sure a
>> proper conf file is generated. It doesn't have to run the tests, just
>> compare the output of the script to the conf generated by devstack.
>>
>> Re Rally - I believe the best place for tempest configuration script is
>> within tempest. That said, if the Tempest community doesn't want this tool,
>> we'll have to settle for the Rally solution.
>>
>> Regards
>> Yair
>>
>> On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier <
>> jordan.pitt...@scality.com> wrote:
>>
>>> Hi,
>>> I think this script is valuable to some users: Rally and Red Hat
>>> expressed their needs, they seem clear.
>>>
>>> This tool is far from bullet proof and if used blindly or in case of
>>> bugs, Tempest could be misconfigured. So, we could have this tool inside
>>> the Tempest repository (in the tools/) but not use it at all for the Gate.
>>>
>>> I am not sure I fully understand the resistance for this, if we don"t
>>> use this config generator for the gate, what's the risk ?
>>>
>>> Jordan
>>>
>>> On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi >> > wrote:
>>>
 2015-11-27 15:40 GMT+09:00 Daniel Mellado :
 > I still do think that even if there are some issues addressed to the
 > feature, such as skipping tests in the gate, the feature itself it's
 still
 > good -we just won't use it for the gates-
 > Instead it'd be used as a wrapper for a user who would be interested
 on
 > trying it against a real/reals clouds.
 >
 > Ken, do you really think a tempest user should know all tempest
 options?
 > As you pointed out there are quite a few of them and even if they
 should at
 > least know their environment, this script would set a minimum
 acceptable
 > default. Do you think PTL and Pre-PTL concerns that we spoke of would
 still
 > apply to that scenario?

 If Tempest users 

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Yair Fried]

Hi,
I agree with Jordan.
We don't have to use the tool as part of the gate. It's target audience is
people and not CI systems. More specifically - new users.
However, we could add a gate (or a few) for the tool that makes sure a
proper conf file is generated. It doesn't have to run the tests, just
compare the output of the script to the conf generated by devstack.

Re Rally - I believe the best place for tempest configuration script is
within tempest. That said, if the Tempest community doesn't want this tool,
we'll have to settle for the Rally solution.

Regards
Yair

On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier  wrote:

> Hi,
> I think this script is valuable to some users: Rally and Red Hat expressed
> their needs, they seem clear.
>
> This tool is far from bullet proof and if used blindly or in case of bugs,
> Tempest could be misconfigured. So, we could have this tool inside the
> Tempest repository (in the tools/) but not use it at all for the Gate.
>
> I am not sure I fully understand the resistance for this, if we don"t use
> this config generator for the gate, what's the risk ?
>
> Jordan
>
> On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi 
> wrote:
>
>> 2015-11-27 15:40 GMT+09:00 Daniel Mellado :
>> > I still do think that even if there are some issues addressed to the
>> > feature, such as skipping tests in the gate, the feature itself it's
>> still
>> > good -we just won't use it for the gates-
>> > Instead it'd be used as a wrapper for a user who would be interested on
>> > trying it against a real/reals clouds.
>> >
>> > Ken, do you really think a tempest user should know all tempest options?
>> > As you pointed out there are quite a few of them and even if they
>> should at
>> > least know their environment, this script would set a minimum acceptable
>> > default. Do you think PTL and Pre-PTL concerns that we spoke of would
>> still
>> > apply to that scenario?
>>
>> If Tempest users run part of tests of Tempest, they need to know the
>> options which are used with these tests only.
>> For example, current Tempest contains ironic API tests and the
>> corresponding options.
>> If users don't want to run these tests because the cloud don't support
>> ironic API, they don't need to know/setup these options.
>> I feel users need to know necessary options which are used on tests
>> they want, because they need to investigate the reason if facing a
>> problem during Tempest tests.
>>
>> Now Tempest options contain their default values, but you need a
>> script for changing them from the default.
>> Don't these default values work for your cloud at all?
>> If so, these values should be changed to better.
>>
>> Thanks
>> Ken Ohmichi
>>
>> ---
>>
>> > Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
>> > tempest-cli improvements? What do you think about this, Masayuki?
>> >
>> > Thanks for all your feedback! ;)
>> >
>> > El 27/11/15 a las 00:15, Andrey Kurilin escribió:
>> >
>> > Sorry for wrong numbers. The bug-fix for issue with counters is merged.
>> > Correct numbers(latest result from rally's gate[1]):
>> >  - total number of executed tests: 1689
>> >  - success: 1155
>> >  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
>> > enable them)
>> >  - failed: 0
>> >
>> > [1] -
>> >
>> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
>> > [2] - https://review.openstack.org/#/c/250540/
>> >
>> > On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
>> yloban...@mirantis.com>
>> > wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >> Yes, I am working on this now. We have some success already, but there
>> is
>> >> a lot of work to do. Of course, some things don't work ideally. For
>> example,
>> >> in [2] from the previous letter we have not 24 skipped tests, actually
>> much
>> >> more. So we have a bug somewhere :)
>> >>
>> >> Regards,
>> >> Yaroslav Lobankov.
>> >>
>> >> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin > >
>> >> wrote:
>> >>>
>> >>> Hi!
>> >>> Boris P. and I tried to push a spec[1] for automation tempest config
>> >>> generator, but we did not succeed to merge it. Imo, qa-team doesn't
>> want to
>> >>> have such tool:(
>> >>>
>> >>> >However, there is a big concern:
>> >>> >If the script contain a bug and creates the configuration which makes
>> >>> >most tests skipped, we cannot do enough tests on the gate.
>> >>> >Tempest contains 1432 tests and difficult to detect which tests are
>> >>> >skipped as unexpected.
>> >>>
>> >>> Yaroslav Lobankov is working on improvement for tempest config
>> generator
>> >>> in Rally. Last time when we launch full tempest run[2], we got 1154
>> success
>> >>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
>> >>> mechanism(it based on subunit-filter): you can transmit a file with
>> test
>> >>> names + 

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Masayuki Igawa]

Hi,

I agree with Ken'ichi's opinion, basically. Tempest users should know "what
do we test for?" and we shouldn't discover values that we test for
automatically.
If users seems that "My current cloud is good. This is what I expect.",
discovering function could work. But I suppose many of users would use
tempest-config-generator for a new their cloud. So I feel the tool users
could be misunderstanding easily.

But I also think that tempest users don't need to know all of the
configurations.
So, how about something like introducing "a configuration wizard" for
tempest configuration?
This is a just idea, though..


Anyway, if you have the motivation to introduce tempest-config, how about
writing a spec for the feature for a concrete discussion?
(I think we should have agreements about the target issues, users,
solutions, etc.)

Best Regards,
-- Masayuki Igawa


2015年11月29日(日) 22:07 Yair Fried :

> Hi,
> I agree with Jordan.
> We don't have to use the tool as part of the gate. It's target audience is
> people and not CI systems. More specifically - new users.
> However, we could add a gate (or a few) for the tool that makes sure a
> proper conf file is generated. It doesn't have to run the tests, just
> compare the output of the script to the conf generated by devstack.
>
> Re Rally - I believe the best place for tempest configuration script is
> within tempest. That said, if the Tempest community doesn't want this tool,
> we'll have to settle for the Rally solution.
>
> Regards
> Yair
>
> On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier <
> jordan.pitt...@scality.com> wrote:
>
>> Hi,
>> I think this script is valuable to some users: Rally and Red Hat
>> expressed their needs, they seem clear.
>>
>> This tool is far from bullet proof and if used blindly or in case of
>> bugs, Tempest could be misconfigured. So, we could have this tool inside
>> the Tempest repository (in the tools/) but not use it at all for the Gate.
>>
>> I am not sure I fully understand the resistance for this, if we don"t use
>> this config generator for the gate, what's the risk ?
>>
>> Jordan
>>
>> On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi 
>> wrote:
>>
>>> 2015-11-27 15:40 GMT+09:00 Daniel Mellado :
>>> > I still do think that even if there are some issues addressed to the
>>> > feature, such as skipping tests in the gate, the feature itself it's
>>> still
>>> > good -we just won't use it for the gates-
>>> > Instead it'd be used as a wrapper for a user who would be interested on
>>> > trying it against a real/reals clouds.
>>> >
>>> > Ken, do you really think a tempest user should know all tempest
>>> options?
>>> > As you pointed out there are quite a few of them and even if they
>>> should at
>>> > least know their environment, this script would set a minimum
>>> acceptable
>>> > default. Do you think PTL and Pre-PTL concerns that we spoke of would
>>> still
>>> > apply to that scenario?
>>>
>>> If Tempest users run part of tests of Tempest, they need to know the
>>> options which are used with these tests only.
>>> For example, current Tempest contains ironic API tests and the
>>> corresponding options.
>>> If users don't want to run these tests because the cloud don't support
>>> ironic API, they don't need to know/setup these options.
>>> I feel users need to know necessary options which are used on tests
>>> they want, because they need to investigate the reason if facing a
>>> problem during Tempest tests.
>>>
>>> Now Tempest options contain their default values, but you need a
>>> script for changing them from the default.
>>> Don't these default values work for your cloud at all?
>>> If so, these values should be changed to better.
>>>
>>> Thanks
>>> Ken Ohmichi
>>>
>>> ---
>>>
>>> > Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it
>>> to
>>> > tempest-cli improvements? What do you think about this, Masayuki?
>>> >
>>> > Thanks for all your feedback! ;)
>>> >
>>> > El 27/11/15 a las 00:15, Andrey Kurilin escribió:
>>> >
>>> > Sorry for wrong numbers. The bug-fix for issue with counters is merged.
>>> > Correct numbers(latest result from rally's gate[1]):
>>> >  - total number of executed tests: 1689
>>> >  - success: 1155
>>> >  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2]
>>> should
>>> > enable them)
>>> >  - failed: 0
>>> >
>>> > [1] -
>>> >
>>> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
>>> > [2] - https://review.openstack.org/#/c/250540/
>>> >
>>> > On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
>>> yloban...@mirantis.com>
>>> > wrote:
>>> >>
>>> >> Hello everyone,
>>> >>
>>> >> Yes, I am working on this now. We have some success already, but
>>> there is
>>> >> a lot of work to do. Of course, some things don't work ideally. For
>>> example,
>>> >> in [2] from the previous letter we have not 24 skipped tests,
>>> 

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Jordan Pittier]

Hi,
I think this script is valuable to some users: Rally and Red Hat expressed
their needs, they seem clear.

This tool is far from bullet proof and if used blindly or in case of bugs,
Tempest could be misconfigured. So, we could have this tool inside the
Tempest repository (in the tools/) but not use it at all for the Gate.

I am not sure I fully understand the resistance for this, if we don"t use
this config generator for the gate, what's the risk ?

Jordan

On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi 
wrote:

> 2015-11-27 15:40 GMT+09:00 Daniel Mellado :
> > I still do think that even if there are some issues addressed to the
> > feature, such as skipping tests in the gate, the feature itself it's
> still
> > good -we just won't use it for the gates-
> > Instead it'd be used as a wrapper for a user who would be interested on
> > trying it against a real/reals clouds.
> >
> > Ken, do you really think a tempest user should know all tempest options?
> > As you pointed out there are quite a few of them and even if they should
> at
> > least know their environment, this script would set a minimum acceptable
> > default. Do you think PTL and Pre-PTL concerns that we spoke of would
> still
> > apply to that scenario?
>
> If Tempest users run part of tests of Tempest, they need to know the
> options which are used with these tests only.
> For example, current Tempest contains ironic API tests and the
> corresponding options.
> If users don't want to run these tests because the cloud don't support
> ironic API, they don't need to know/setup these options.
> I feel users need to know necessary options which are used on tests
> they want, because they need to investigate the reason if facing a
> problem during Tempest tests.
>
> Now Tempest options contain their default values, but you need a
> script for changing them from the default.
> Don't these default values work for your cloud at all?
> If so, these values should be changed to better.
>
> Thanks
> Ken Ohmichi
>
> ---
>
> > Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
> > tempest-cli improvements? What do you think about this, Masayuki?
> >
> > Thanks for all your feedback! ;)
> >
> > El 27/11/15 a las 00:15, Andrey Kurilin escribió:
> >
> > Sorry for wrong numbers. The bug-fix for issue with counters is merged.
> > Correct numbers(latest result from rally's gate[1]):
> >  - total number of executed tests: 1689
> >  - success: 1155
> >  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
> > enable them)
> >  - failed: 0
> >
> > [1] -
> >
> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
> > [2] - https://review.openstack.org/#/c/250540/
> >
> > On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov <
> yloban...@mirantis.com>
> > wrote:
> >>
> >> Hello everyone,
> >>
> >> Yes, I am working on this now. We have some success already, but there
> is
> >> a lot of work to do. Of course, some things don't work ideally. For
> example,
> >> in [2] from the previous letter we have not 24 skipped tests, actually
> much
> >> more. So we have a bug somewhere :)
> >>
> >> Regards,
> >> Yaroslav Lobankov.
> >>
> >> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin 
> >> wrote:
> >>>
> >>> Hi!
> >>> Boris P. and I tried to push a spec[1] for automation tempest config
> >>> generator, but we did not succeed to merge it. Imo, qa-team doesn't
> want to
> >>> have such tool:(
> >>>
> >>> >However, there is a big concern:
> >>> >If the script contain a bug and creates the configuration which makes
> >>> >most tests skipped, we cannot do enough tests on the gate.
> >>> >Tempest contains 1432 tests and difficult to detect which tests are
> >>> >skipped as unexpected.
> >>>
> >>> Yaroslav Lobankov is working on improvement for tempest config
> generator
> >>> in Rally. Last time when we launch full tempest run[2], we got 1154
> success
> >>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
> >>> mechanism(it based on subunit-filter): you can transmit a file with
> test
> >>> names + reasons and rally will modify results.
> >>>
> >>> [1] - https://review.openstack.org/#/c/94473/
> >>>
> >>> [2] -
> >>>
> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
> >>>
> >>> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi <
> ken1ohmi...@gmail.com>
> >>> wrote:
> 
>  Hi Daniel,
> 
>  Thanks for pointing this up.
> 
>  2015-11-25 1:40 GMT+09:00 Daniel Mellado  >:
>  > Hi All,
>  >
>  > As you might already know, within Red Hat's tempest fork, we do have
>  > one
>  > tempest configuration script which was built in the past by David
>  > Kranz [1]
>  > and that's been actively used in our CI system. Regarding this
> topic,
> 

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Andrey Kurilin]

Sorry for wrong numbers. The bug-fix for issue with counters is merged.
Correct numbers(latest result from rally's gate[1]):
 - total number of executed tests: 1689
 - success: 1155
 - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
enable them)
 - failed: 0

[1] -
http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
[2] - https://review.openstack.org/#/c/250540/

On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov 
wrote:

> Hello everyone,
>
> Yes, I am working on this now. We have some success already, but there is
> a lot of work to do. Of course, some things don't work ideally. For
> example, in [2] from the previous letter we have not 24 skipped tests,
> actually much more. So we have a bug somewhere :)
>
> Regards,
> Yaroslav Lobankov.
>
> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin 
> wrote:
>
>> Hi!
>> Boris P. and I tried to push a spec[1] for automation tempest config
>> generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
>> have such tool:(
>>
>> >However, there is a big concern:
>> >If the script contain a bug and creates the configuration which makes
>> >most tests skipped, we cannot do enough tests on the gate.
>> >Tempest contains 1432 tests and difficult to detect which tests are
>> >skipped as unexpected.
>>
>> Yaroslav Lobankov is working on improvement for tempest config generator
>> in Rally. Last time when we launch full tempest run[2], we got 1154 success
>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
>> mechanism(it based on subunit-filter): you can transmit a file with test
>> names + reasons and rally will modify results.
>>
>> [1] - https://review.openstack.org/#/c/94473/
>>
>> [2] -
>> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>>
>> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi 
>> wrote:
>>
>>> Hi Daniel,
>>>
>>> Thanks for pointing this up.
>>>
>>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado :
>>> > Hi All,
>>> >
>>> > As you might already know, within Red Hat's tempest fork, we do have
>>> one
>>> > tempest configuration script which was built in the past by David
>>> Kranz [1]
>>> > and that's been actively used in our CI system. Regarding this topic,
>>> I'm
>>> > aware that quite some effort has been done in the past [2] and I would
>>> like
>>> > to complete the implementation of this blueprint/spec.
>>> >
>>> > My plan would be to have this script under the /tempest/cmd or
>>> > /tempest/tools folder from tempest so it can be used to configure not
>>> the
>>> > tempest gate but any cloud we'd like to run tempest against.
>>> >
>>> > Adding the configuration script was discussed briefly at the Mitaka
>>> summit
>>> > in the QA Priorities meting [3]. I propose we use the existing
>>> etherpad to
>>> > continue the discussion around and tracking of implementing "tempest
>>> > config-create" using the downstream config script as a starting point.
>>> [4]
>>> >
>>> > If you have any questions, comments or opinion, please let me know.
>>>
>>> This topic have happened several times, and I also felt this kind of
>>> tool was very useful for Tempest users, because Tempest contains 296
>>> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
>>> set the configuration up.
>>> However, there is a big concern:
>>> If the script contain a bug and creates the configuration which makes
>>> most tests skipped, we cannot do enough tests on the gate.
>>> Tempest contains 1432 tests and difficult to detect which tests are
>>> skipped as unexpected.
>>> Actually we faced unexpected skipped tests on the gate before due to
>>> some bug, then the problem has been fixed.
>>> But I can imagine this kind of problem happens after implementing this
>>> kind of script.
>>>
>>> So now I am feeling Tempest users need to know what cloud they want to
>>> test with Tempest, and need to know what tests run with Tempest.
>>> Testers need to know what test target/items they are testing basically.
>>>
>>> Thanks
>>> Ken Ohmichi
>>>
>>> ---
>>>
>>> > ---
>>> > [1]
>>> >
>>> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
>>> > [2]
>>> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
>>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
>>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
>>> >
>>> >
>>> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>>> >
>>> >
>>> __
>>> > OpenStack Development Mailing List (not for usage questions)
>>> > Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> > 

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Ken'ichi Ohmichi]

2015-11-27 15:40 GMT+09:00 Daniel Mellado :
> I still do think that even if there are some issues addressed to the
> feature, such as skipping tests in the gate, the feature itself it's still
> good -we just won't use it for the gates-
> Instead it'd be used as a wrapper for a user who would be interested on
> trying it against a real/reals clouds.
>
> Ken, do you really think a tempest user should know all tempest options?
> As you pointed out there are quite a few of them and even if they should at
> least know their environment, this script would set a minimum acceptable
> default. Do you think PTL and Pre-PTL concerns that we spoke of would still
> apply to that scenario?

If Tempest users run part of tests of Tempest, they need to know the
options which are used with these tests only.
For example, current Tempest contains ironic API tests and the
corresponding options.
If users don't want to run these tests because the cloud don't support
ironic API, they don't need to know/setup these options.
I feel users need to know necessary options which are used on tests
they want, because they need to investigate the reason if facing a
problem during Tempest tests.

Now Tempest options contain their default values, but you need a
script for changing them from the default.
Don't these default values work for your cloud at all?
If so, these values should be changed to better.

Thanks
Ken Ohmichi

---

> Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
> tempest-cli improvements? What do you think about this, Masayuki?
>
> Thanks for all your feedback! ;)
>
> El 27/11/15 a las 00:15, Andrey Kurilin escribió:
>
> Sorry for wrong numbers. The bug-fix for issue with counters is merged.
> Correct numbers(latest result from rally's gate[1]):
>  - total number of executed tests: 1689
>  - success: 1155
>  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should
> enable them)
>  - failed: 0
>
> [1] -
> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
> [2] - https://review.openstack.org/#/c/250540/
>
> On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov 
> wrote:
>>
>> Hello everyone,
>>
>> Yes, I am working on this now. We have some success already, but there is
>> a lot of work to do. Of course, some things don't work ideally. For example,
>> in [2] from the previous letter we have not 24 skipped tests, actually much
>> more. So we have a bug somewhere :)
>>
>> Regards,
>> Yaroslav Lobankov.
>>
>> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin 
>> wrote:
>>>
>>> Hi!
>>> Boris P. and I tried to push a spec[1] for automation tempest config
>>> generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
>>> have such tool:(
>>>
>>> >However, there is a big concern:
>>> >If the script contain a bug and creates the configuration which makes
>>> >most tests skipped, we cannot do enough tests on the gate.
>>> >Tempest contains 1432 tests and difficult to detect which tests are
>>> >skipped as unexpected.
>>>
>>> Yaroslav Lobankov is working on improvement for tempest config generator
>>> in Rally. Last time when we launch full tempest run[2], we got 1154 success
>>> tests and only 24 skipped. Also, there is a patch, which adds x-fail
>>> mechanism(it based on subunit-filter): you can transmit a file with test
>>> names + reasons and rally will modify results.
>>>
>>> [1] - https://review.openstack.org/#/c/94473/
>>>
>>> [2] -
>>> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>>>
>>> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi 
>>> wrote:

 Hi Daniel,

 Thanks for pointing this up.

 2015-11-25 1:40 GMT+09:00 Daniel Mellado :
 > Hi All,
 >
 > As you might already know, within Red Hat's tempest fork, we do have
 > one
 > tempest configuration script which was built in the past by David
 > Kranz [1]
 > and that's been actively used in our CI system. Regarding this topic,
 > I'm
 > aware that quite some effort has been done in the past [2] and I would
 > like
 > to complete the implementation of this blueprint/spec.
 >
 > My plan would be to have this script under the /tempest/cmd or
 > /tempest/tools folder from tempest so it can be used to configure not
 > the
 > tempest gate but any cloud we'd like to run tempest against.
 >
 > Adding the configuration script was discussed briefly at the Mitaka
 > summit
 > in the QA Priorities meting [3]. I propose we use the existing
 > etherpad to
 > continue the discussion around and tracking of implementing "tempest
 > config-create" using the downstream config script as a starting point.
 > [4]
 >
 > If you have any questions, 

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Daniel Mellado]

I still do think that even if there are some issues addressed to the
feature, such as skipping tests in the gate, the feature itself it's
still good -we just won't use it for the gates-
Instead it'd be used as a wrapper for a user who would be interested on
trying it against a real/reals clouds.

Ken, do you really think a tempest user should know all tempest options?
As you pointed out there are quite a few of them and even if they should
at least know their environment, this script would set a minimum
acceptable default. Do you think PTL and Pre-PTL concerns that we spoke
of would still apply to that scenario?

Andrey, Yaroslav. Would you like to revisit the blueprint to adapt it to
tempest-cli improvements? What do you think about this, Masayuki?

Thanks for all your feedback! ;)

El 27/11/15 a las 00:15, Andrey Kurilin escribió:
> Sorry for wrong numbers. The bug-fix for issue with counters is merged.
> Correct numbers(latest result from rally's gate[1]):
>  - total number of executed tests: 1689
>  - success: 1155
>  - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2]
> should enable them)
>  - failed: 0
>
> [1] -
> http://logs.openstack.org/27/246627/11/gate/gate-rally-dsvm-verify-full/800bad0/rally-verify/7_verify_results_--html.html.gz
> [2] - https://review.openstack.org/#/c/250540/
>
> On Thu, Nov 26, 2015 at 3:23 PM, Yaroslav Lobankov
> > wrote:
>
> Hello everyone,
>
> Yes, I am working on this now. We have some success already, but
> there is a lot of work to do. Of course, some things don't work
> ideally. For example, in [2] from the previous letter we have not
> 24 skipped tests, actually much more. So we have a bug somewhere :)
>
> Regards,
> Yaroslav Lobankov.  
>
> On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin
> > wrote:
>
> Hi!
> Boris P. and I tried to push a spec[1] for automation tempest
> config generator, but we did not succeed to merge it. Imo,
> qa-team doesn't want to have such tool:(
>
> >However, there is a big concern:
> >If the script contain a bug and creates the configuration
> which makes
> >most tests skipped, we cannot do enough tests on the gate.
> >Tempest contains 1432 tests and difficult to detect which
> tests are
> >skipped as unexpected.
>
> Yaroslav Lobankov is working on improvement for tempest config
> generator in Rally. Last time when we launch full tempest
> run[2], we got 1154 success tests and only 24 skipped. Also,
> there is a patch, which adds x-fail mechanism(it based on
> subunit-filter): you can transmit a file with test names +
> reasons and rally will modify results.
>
> [1] - https://review.openstack.org/#/c/94473/
>
> [2] -
> 
> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>
> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi
> > wrote:
>
> Hi Daniel,
>
> Thanks for pointing this up.
>
> 2015-11-25 1:40 GMT+09:00 Daniel Mellado
>  >:
> > Hi All,
> >
> > As you might already know, within Red Hat's tempest
> fork, we do have one
> > tempest configuration script which was built in the past
> by David Kranz [1]
> > and that's been actively used in our CI system.
> Regarding this topic, I'm
> > aware that quite some effort has been done in the past
> [2] and I would like
> > to complete the implementation of this blueprint/spec.
> >
> > My plan would be to have this script under the
> /tempest/cmd or
> > /tempest/tools folder from tempest so it can be used to
> configure not the
> > tempest gate but any cloud we'd like to run tempest against.
> >
> > Adding the configuration script was discussed briefly at
> the Mitaka summit
> > in the QA Priorities meting [3]. I propose we use the
> existing etherpad to
> > continue the discussion around and tracking of
> implementing "tempest
> > config-create" using the downstream config script as a
> starting point. [4]
> >
> > If you have any questions, comments or opinion, please
> let me know.
>
> This topic have happened several times, and I also felt
> this kind of
> tool was very useful for Tempest users, because Tempest
> contains 296
>  

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Ken'ichi Ohmichi]

Hi Daniel,

Thanks for pointing this up.

2015-11-25 1:40 GMT+09:00 Daniel Mellado :
> Hi All,
>
> As you might already know, within Red Hat's tempest fork, we do have one
> tempest configuration script which was built in the past by David Kranz [1]
> and that's been actively used in our CI system. Regarding this topic, I'm
> aware that quite some effort has been done in the past [2] and I would like
> to complete the implementation of this blueprint/spec.
>
> My plan would be to have this script under the /tempest/cmd or
> /tempest/tools folder from tempest so it can be used to configure not the
> tempest gate but any cloud we'd like to run tempest against.
>
> Adding the configuration script was discussed briefly at the Mitaka summit
> in the QA Priorities meting [3]. I propose we use the existing etherpad to
> continue the discussion around and tracking of implementing "tempest
> config-create" using the downstream config script as a starting point. [4]
>
> If you have any questions, comments or opinion, please let me know.

This topic have happened several times, and I also felt this kind of
tool was very useful for Tempest users, because Tempest contains 296
options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
set the configuration up.
However, there is a big concern:
If the script contain a bug and creates the configuration which makes
most tests skipped, we cannot do enough tests on the gate.
Tempest contains 1432 tests and difficult to detect which tests are
skipped as unexpected.
Actually we faced unexpected skipped tests on the gate before due to
some bug, then the problem has been fixed.
But I can imagine this kind of problem happens after implementing this
kind of script.

So now I am feeling Tempest users need to know what cloud they want to
test with Tempest, and need to know what tests run with Tempest.
Testers need to know what test target/items they are testing basically.

Thanks
Ken Ohmichi

---

> ---
> [1]
> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
> [2] https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
> [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
> [4] https://etherpad.openstack.org/p/tempest-cli-improvements
>
> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Andrey Kurilin]

Hi!
Boris P. and I tried to push a spec[1] for automation tempest config
generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
have such tool:(

>However, there is a big concern:
>If the script contain a bug and creates the configuration which makes
>most tests skipped, we cannot do enough tests on the gate.
>Tempest contains 1432 tests and difficult to detect which tests are
>skipped as unexpected.

Yaroslav Lobankov is working on improvement for tempest config generator in
Rally. Last time when we launch full tempest run[2], we got 1154 success
tests and only 24 skipped. Also, there is a patch, which adds x-fail
mechanism(it based on subunit-filter): you can transmit a file with test
names + reasons and rally will modify results.

[1] - https://review.openstack.org/#/c/94473/

[2] -
http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz

On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi 
wrote:

> Hi Daniel,
>
> Thanks for pointing this up.
>
> 2015-11-25 1:40 GMT+09:00 Daniel Mellado :
> > Hi All,
> >
> > As you might already know, within Red Hat's tempest fork, we do have one
> > tempest configuration script which was built in the past by David Kranz
> [1]
> > and that's been actively used in our CI system. Regarding this topic, I'm
> > aware that quite some effort has been done in the past [2] and I would
> like
> > to complete the implementation of this blueprint/spec.
> >
> > My plan would be to have this script under the /tempest/cmd or
> > /tempest/tools folder from tempest so it can be used to configure not the
> > tempest gate but any cloud we'd like to run tempest against.
> >
> > Adding the configuration script was discussed briefly at the Mitaka
> summit
> > in the QA Priorities meting [3]. I propose we use the existing etherpad
> to
> > continue the discussion around and tracking of implementing "tempest
> > config-create" using the downstream config script as a starting point.
> [4]
> >
> > If you have any questions, comments or opinion, please let me know.
>
> This topic have happened several times, and I also felt this kind of
> tool was very useful for Tempest users, because Tempest contains 296
> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
> set the configuration up.
> However, there is a big concern:
> If the script contain a bug and creates the configuration which makes
> most tests skipped, we cannot do enough tests on the gate.
> Tempest contains 1432 tests and difficult to detect which tests are
> skipped as unexpected.
> Actually we faced unexpected skipped tests on the gate before due to
> some bug, then the problem has been fixed.
> But I can imagine this kind of problem happens after implementing this
> kind of script.
>
> So now I am feeling Tempest users need to know what cloud they want to
> test with Tempest, and need to know what tests run with Tempest.
> Testers need to know what test target/items they are testing basically.
>
> Thanks
> Ken Ohmichi
>
> ---
>
> > ---
> > [1]
> >
> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
> > [2]
> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
> >
> >
> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Best regards,
Andrey Kurilin.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Yaroslav Lobankov]

Hello everyone,

Yes, I am working on this now. We have some success already, but there is a
lot of work to do. Of course, some things don't work ideally. For example,
in [2] from the previous letter we have not 24 skipped tests, actually much
more. So we have a bug somewhere :)

Regards,
Yaroslav Lobankov.

On Thu, Nov 26, 2015 at 3:59 PM, Andrey Kurilin 
wrote:

> Hi!
> Boris P. and I tried to push a spec[1] for automation tempest config
> generator, but we did not succeed to merge it. Imo, qa-team doesn't want to
> have such tool:(
>
> >However, there is a big concern:
> >If the script contain a bug and creates the configuration which makes
> >most tests skipped, we cannot do enough tests on the gate.
> >Tempest contains 1432 tests and difficult to detect which tests are
> >skipped as unexpected.
>
> Yaroslav Lobankov is working on improvement for tempest config generator
> in Rally. Last time when we launch full tempest run[2], we got 1154 success
> tests and only 24 skipped. Also, there is a patch, which adds x-fail
> mechanism(it based on subunit-filter): you can transmit a file with test
> names + reasons and rally will modify results.
>
> [1] - https://review.openstack.org/#/c/94473/
>
> [2] -
> http://logs.openstack.org/49/242849/8/check/gate-rally-dsvm-verify/e91992e/rally-verify/7_verify_results_--html.html.gz
>
> On Thu, Nov 26, 2015 at 1:52 PM, Ken'ichi Ohmichi 
> wrote:
>
>> Hi Daniel,
>>
>> Thanks for pointing this up.
>>
>> 2015-11-25 1:40 GMT+09:00 Daniel Mellado :
>> > Hi All,
>> >
>> > As you might already know, within Red Hat's tempest fork, we do have one
>> > tempest configuration script which was built in the past by David Kranz
>> [1]
>> > and that's been actively used in our CI system. Regarding this topic,
>> I'm
>> > aware that quite some effort has been done in the past [2] and I would
>> like
>> > to complete the implementation of this blueprint/spec.
>> >
>> > My plan would be to have this script under the /tempest/cmd or
>> > /tempest/tools folder from tempest so it can be used to configure not
>> the
>> > tempest gate but any cloud we'd like to run tempest against.
>> >
>> > Adding the configuration script was discussed briefly at the Mitaka
>> summit
>> > in the QA Priorities meting [3]. I propose we use the existing etherpad
>> to
>> > continue the discussion around and tracking of implementing "tempest
>> > config-create" using the downstream config script as a starting point.
>> [4]
>> >
>> > If you have any questions, comments or opinion, please let me know.
>>
>> This topic have happened several times, and I also felt this kind of
>> tool was very useful for Tempest users, because Tempest contains 296
>> options($ grep cfg * -R | grep Opt | wc -l) now and it is difficult to
>> set the configuration up.
>> However, there is a big concern:
>> If the script contain a bug and creates the configuration which makes
>> most tests skipped, we cannot do enough tests on the gate.
>> Tempest contains 1432 tests and difficult to detect which tests are
>> skipped as unexpected.
>> Actually we faced unexpected skipped tests on the gate before due to
>> some bug, then the problem has been fixed.
>> But I can imagine this kind of problem happens after implementing this
>> kind of script.
>>
>> So now I am feeling Tempest users need to know what cloud they want to
>> test with Tempest, and need to know what tests run with Tempest.
>> Testers need to know what test target/items they are testing basically.
>>
>> Thanks
>> Ken Ohmichi
>>
>> ---
>>
>> > ---
>> > [1]
>> >
>> https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
>> > [2]
>> https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
>> > [3] https://etherpad.openstack.org/p/mitaka-qa-priorities
>> > [4] https://etherpad.openstack.org/p/tempest-cli-improvements
>> >
>> >
>> https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
>> >
>> >
>> __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Best regards,
> Andrey Kurilin.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>

[openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

[Daniel Mellado]

Hi All,

As you might already know, within Red Hat's tempest fork, we do have one
tempest configuration script which was built in the past by David Kranz
[1] and that's been actively used in our CI system. Regarding this
topic, I'm aware that quite some effort has been done in the past [2]
and I would like to complete the implementation of this blueprint/spec.

My plan would be to have this script under the /tempest/cmd or
/tempest/tools folder from tempest so it can be used to configure not
the tempest gate but any cloud we'd like to run tempest against.

Adding the configuration script was discussed briefly at the Mitaka
summit in the QA Priorities meting [3]. I propose we use the existing
etherpad to continue the discussion around and tracking of implementing
"tempest config-create" using the downstream config script as a starting
point. [4]

If you have any questions, comments or opinion, please let me know.

Best Regards

Daniel Mellado

---
[1]
https://github.com/redhat-openstack/tempest/blob/master/tools/config_tempest.py
[2] https://blueprints.launchpad.net/tempest/+spec/tempest-config-generator
[3] https://etherpad.openstack.org/p/mitaka-qa-priorities
[4] https://etherpad.openstack.org/p/tempest-cli-improvements

https://github.com/openstack/qa-specs/blob/master/specs/tempest/tempest-cli-improvements.rst
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev