Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements Looking at the tool, it seems to me that is servers a combination of functions: - provision test resources - support for distribution specific / cloud

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Looking at the tool, it seems to me that is servers a combination of functions: - provision test resources - support for distribution specific / cloud specific overrides to default - support for configuration override via CLI - discovery of configuration Test resource provisioning is something

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Hi, I agree with Jordan. We don't have to use the tool as part of the gate. It's target audience is people and not CI systems. More specifically - new users. However, we could add a gate (or a few) for the tool that makes sure a proper conf file is generated. It doesn't have to run the tests, just

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Hi, I agree with Ken'ichi's opinion, basically. Tempest users should know "what do we test for?" and we shouldn't discover values that we test for automatically. If users seems that "My current cloud is good. This is what I expect.", discovering function could work. But I suppose many of users

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Hi, I think this script is valuable to some users: Rally and Red Hat expressed their needs, they seem clear. This tool is far from bullet proof and if used blindly or in case of bugs, Tempest could be misconfigured. So, we could have this tool inside the Tempest repository (in the tools/) but not

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Sorry for wrong numbers. The bug-fix for issue with counters is merged. Correct numbers(latest result from rally's gate[1]): - total number of executed tests: 1689 - success: 1155 - skipped: 534 (neutron,heat,sahara,ceilometer are disabled. [2] should enable them) - failed: 0 [1] -

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

2015-11-27 15:40 GMT+09:00 Daniel Mellado : > I still do think that even if there are some issues addressed to the > feature, such as skipping tests in the gate, the feature itself it's still > good -we just won't use it for the gates- > Instead it'd be used as a

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

I still do think that even if there are some issues addressed to the feature, such as skipping tests in the gate, the feature itself it's still good -we just won't use it for the gates- Instead it'd be used as a wrapper for a user who would be interested on trying it against a real/reals clouds.

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Hi Daniel, Thanks for pointing this up. 2015-11-25 1:40 GMT+09:00 Daniel Mellado : > Hi All, > > As you might already know, within Red Hat's tempest fork, we do have one > tempest configuration script which was built in the past by David Kranz [1] > and that's been

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Hi! Boris P. and I tried to push a spec[1] for automation tempest config generator, but we did not succeed to merge it. Imo, qa-team doesn't want to have such tool:( >However, there is a big concern: >If the script contain a bug and creates the configuration which makes >most tests skipped, we

Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Hello everyone, Yes, I am working on this now. We have some success already, but there is a lot of work to do. Of course, some things don't work ideally. For example, in [2] from the previous letter we have not 24 skipped tests, actually much more. So we have a bug somewhere :) Regards, Yaroslav

[openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

Hi All, As you might already know, within Red Hat's tempest fork, we do have one tempest configuration script which was built in the past by David Kranz [1] and that's been actively used in our CI system. Regarding this topic, I'm aware that quite some effort has been done in the past [2] and I