Re: [openstack-dev] [Openstack-operators] [all][qa][gabbi][rally][tempest] Extend rally verfiy to unify work with Gabbi, Tempest and all in-tree functional tests

2015-03-19 Thread Andrey Kurilin
I have an alternative solution for `rally verify project start` command. What do you think about similar management stuff for verifiers as we have for deployments? It requires several changes: - `rally verifiers` command: Implementation of new command, which will manage(install, reinstall,

Re: [openstack-dev] [Openstack-operators] [all][qa][gabbi][rally][tempest] Extend rally verfiy to unify work with Gabbi, Tempest and all in-tree functional tests

2015-03-11 Thread Andrey Kurilin
$ rally verify https://github.com/openstack/nova start As one of end-users of Rally, I dislike such construction, because I don't want to remember links to repos, they are too long for me:) On Wed, Mar 11, 2015 at 12:49 PM, Aleksandr Maretskiy amarets...@mirantis.com wrote: The idea is

Re: [openstack-dev] [Openstack-operators] [all][qa][gabbi][rally][tempest] Extend rally verfiy to unify work with Gabbi, Tempest and all in-tree functional tests

2015-03-11 Thread Boris Pavlovic
Alex, * rally plugin should be a part of project (for example, located in functional tests directory) There are 2 issues with such solution: 1) If rally didn't load plugin, command rally verify project won't exist 2) Putting some strange Rally plugin to source of other projects will be

Re: [openstack-dev] [Openstack-operators] [all][qa][gabbi][rally][tempest] Extend rally verfiy to unify work with Gabbi, Tempest and all in-tree functional tests

2015-03-11 Thread Aleksandr Maretskiy
The idea is great, but IMHO we can move all project-specific code out of rally, so: * rally plugin should be a part of project (for example, located in functional tests directory) * use {project url} instead of {project name} in rally verify command, example: $ rally verify

Re: [openstack-dev] [Openstack-operators] [all][qa][gabbi][rally][tempest] Extend rally verfiy to unify work with Gabbi, Tempest and all in-tree functional tests

2015-03-10 Thread Timur Nurlygayanov
Hi, I like this idea, we use Rally for OpenStack clouds verification at scale and it is the real issue - how to run all functional tests from each project with the one script. If Rally will do this, I will use Rally to run these tests. Thank you! On Mon, Mar 9, 2015 at 6:04 PM, Chris Dent

Re: [openstack-dev] [Openstack-operators] [all][qa][gabbi][rally][tempest] Extend rally verfiy to unify work with Gabbi, Tempest and all in-tree functional tests

2015-03-09 Thread Davanum Srinivas
Boris, 1. Suppose a project say Nova wants to enable this Rally Integration for its functional tests, what does that project have to do? (other than the existing well defined tox targets) 2. Is there a test project with Gabbi based tests that you know of? 3. What changes if any are needed in

Re: [openstack-dev] [Openstack-operators] [all][qa][gabbi][rally][tempest] Extend rally verfiy to unify work with Gabbi, Tempest and all in-tree functional tests

2015-03-09 Thread Boris Pavlovic
Davanum, 1. Suppose a project say Nova wants to enable this Rally Integration for its functional tests, what does that project have to do? (other than the existing well defined tox targets) Actually project by it self shouldn't do anything. The whole work regarding to integration belongs to

Re: [openstack-dev] [Openstack-operators] [all][qa][gabbi][rally][tempest] Extend rally verfiy to unify work with Gabbi, Tempest and all in-tree functional tests

2015-03-09 Thread Chris Dent
On Mon, 9 Mar 2015, Davanum Srinivas wrote: 2. Is there a test project with Gabbi based tests that you know of? In addition to the ceilometer tests that Boris pointed out gnocchi is using it as well: https://github.com/stackforge/gnocchi/tree/master/gnocchi/tests/gabbi 3. What changes