Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Boris Pavlovic]

Hi,

It's still not clear for me, why we can't just add Rally jobs with
scenarios related to specific project.
It will work quite fast and it will cover CLI (instantly)  with good
integration/functional testing.


Best regards,
Boris Pavlovic

On Wed, Mar 2, 2016 at 4:52 AM, Sean Dague  wrote:

> On 03/02/2016 07:34 AM, Ivan Kolodyazhny wrote:
> > Sean,
> >
> > I've mentioned above, that current tempest job runs ~1429 tests and only
> > about 10 of them uses cinderclient. It tooks a lot of time without any
> > benefits for cinder, e.g.: tests like tempest.api.network.* verifies
> > Neutron, not python-cinderclient.
>
> We can say that about a lot of things in that stack. For better or
> worse, that's where our testing is. It's a full stack same set of tests
> against all these components which get used. The tempest.api.network
> tests are quite quick. The biggest time hitters in the runs are scenario
> tests, many of which are volumes driven.
>
> 2016-02-12 19:07:46.277 |
>
> tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot[compute,id-7b6860c2-afa3-4846-9522-adeb38dfbe08,network]
>  193.523
> 2016-02-12 19:07:46.277 |
>
> tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,smoke,volume]
> 150.766
> 2016-02-12 19:07:46.277 |
>
> tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,smoke,volume]
>   136.834
> 2016-02-12 19:07:46.278 |
>
> tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic[compute,id-e79f879e-debb-440c-a7e4-efeda05b6848,network]
>107.045
> 2016-02-12 19:07:46.278 |
>
> tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac[compute,id-9178ad42-10e4-47e9-8987-e02b170cc5cd,network]
> 101.252
> 2016-02-12 19:07:46.278 |
>
> tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless[compute,id-cf1c4425-766b-45b8-be35-e2959728eb00,network]
>   99.041
> 2016-02-12 19:07:46.278 |
>
> tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]
> 96.954
> 2016-02-12 19:07:46.278 |
>
> tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_volume_backed_instance[compute,id-c1b6318c-b9da-490b-9c67-9339b627271f,image,network,volume]
> 95.120
> 2016-02-12 19:07:46.278 |
>
> tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario[compute,id-bdbb5441-9204-419d-a225-b4fdbfb1a1a8,image,network,volume]
>86.165
> 2016-02-12 19:07:46.278 |
>
> tempest.scenario.test_snapshot_pattern.TestSnapshotPattern.test_snapshot_pattern[compute,id-608e604b-1d63-4a82-8e3e-91bc665c90b4,image,network]
>   85.422
>
>
> If you would like to pitch in on an optimization strategy for all the
> components, that would be great. But this needs to be thought about in
> those terms. It would be great to stop testing 2 versions of cinder API
> in every run, for instance. That would be super helpful to everyone as
> those Boot from volume tests take over 2 minutes each.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Sean Dague]

On 03/02/2016 07:34 AM, Ivan Kolodyazhny wrote:
> Sean,
> 
> I've mentioned above, that current tempest job runs ~1429 tests and only
> about 10 of them uses cinderclient. It tooks a lot of time without any
> benefits for cinder, e.g.: tests like tempest.api.network.* verifies
> Neutron, not python-cinderclient.

We can say that about a lot of things in that stack. For better or
worse, that's where our testing is. It's a full stack same set of tests
against all these components which get used. The tempest.api.network
tests are quite quick. The biggest time hitters in the runs are scenario
tests, many of which are volumes driven.

2016-02-12 19:07:46.277 |
tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot[compute,id-7b6860c2-afa3-4846-9522-adeb38dfbe08,network]
 193.523
2016-02-12 19:07:46.277 |
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,smoke,volume]
150.766
2016-02-12 19:07:46.277 |
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,smoke,volume]
  136.834
2016-02-12 19:07:46.278 |
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic[compute,id-e79f879e-debb-440c-a7e4-efeda05b6848,network]
   107.045
2016-02-12 19:07:46.278 |
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_slaac[compute,id-9178ad42-10e4-47e9-8987-e02b170cc5cd,network]
101.252
2016-02-12 19:07:46.278 |
tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_multi_prefix_dhcpv6_stateless[compute,id-cf1c4425-766b-45b8-be35-e2959728eb00,network]
  99.041
2016-02-12 19:07:46.278 |
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]
96.954
2016-02-12 19:07:46.278 |
tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_volume_backed_instance[compute,id-c1b6318c-b9da-490b-9c67-9339b627271f,image,network,volume]
95.120
2016-02-12 19:07:46.278 |
tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario[compute,id-bdbb5441-9204-419d-a225-b4fdbfb1a1a8,image,network,volume]
   86.165
2016-02-12 19:07:46.278 |
tempest.scenario.test_snapshot_pattern.TestSnapshotPattern.test_snapshot_pattern[compute,id-608e604b-1d63-4a82-8e3e-91bc665c90b4,image,network]
  85.422


If you would like to pitch in on an optimization strategy for all the
components, that would be great. But this needs to be thought about in
those terms. It would be great to stop testing 2 versions of cinder API
in every run, for instance. That would be super helpful to everyone as
those Boot from volume tests take over 2 minutes each.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Ivan Kolodyazhny]

Sean,

I've mentioned above, that current tempest job runs ~1429 tests and only
about 10 of them uses cinderclient. It tooks a lot of time without any
benefits for cinder, e.g.: tests like tempest.api.network.* verifies
Neutron, not python-cinderclient.

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 2:15 PM, Sean Dague  wrote:

> On 03/02/2016 06:58 AM, Ivan Kolodyazhny wrote:
> > Sean,
> >
> > In such case, can we configure which tests should be run for
> > the python-cinderclient src job?
>
> No, there really isn't a mechanism for this because getting it right is
> quite complicated and error prone.
>
> What is the motivation here? Maybe if we start with that I can point you
> in a fruitful direction.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Sean Dague]

On 03/02/2016 06:58 AM, Ivan Kolodyazhny wrote:
> Sean,
> 
> In such case, can we configure which tests should be run for
> the python-cinderclient src job?

No, there really isn't a mechanism for this because getting it right is
quite complicated and error prone.

What is the motivation here? Maybe if we start with that I can point you
in a fruitful direction.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Ivan Kolodyazhny]

Sean,

In such case, can we configure which tests should be run for the
python-cinderclient
src job?

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 1:45 PM, Sean Dague  wrote:

> On 03/02/2016 05:24 AM, Ivan Kolodyazhny wrote:
> > Sean,
> >
> > I do understand why we have tempest for python-cinderclient now.
> >
> > But my point is: tempest runs more than 200 tests per each cinderclient
> > change request which takes a lot of time. Why can't we just introduce
> > few integration tests which will tests nova<=>python-cinderclient API.
> >
> > Also, Nova is not only one consumer of cinderclient. What about Heat? We
> > don't want to break it too but to run all Heat-related Tempest tests is
> > not a good idea. We have to implement integration tests between Heat and
> > python-cinderclient too.
>
> You can add any new tests that you want in additional test jobs.
>
> Cinder can not remove the python-cinderclient src job anytime soon.
> That's table stakes for being part of openstack given how deeply that is
> coupled between Nova / Neutron / Cinder.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Sean Dague]

On 03/02/2016 05:24 AM, Ivan Kolodyazhny wrote:
> Sean, 
> 
> I do understand why we have tempest for python-cinderclient now. 
> 
> But my point is: tempest runs more than 200 tests per each cinderclient
> change request which takes a lot of time. Why can't we just introduce
> few integration tests which will tests nova<=>python-cinderclient API. 
> 
> Also, Nova is not only one consumer of cinderclient. What about Heat? We
> don't want to break it too but to run all Heat-related Tempest tests is
> not a good idea. We have to implement integration tests between Heat and
> python-cinderclient too.

You can add any new tests that you want in additional test jobs.

Cinder can not remove the python-cinderclient src job anytime soon.
That's table stakes for being part of openstack given how deeply that is
coupled between Nova / Neutron / Cinder.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Ivan Kolodyazhny]

Luigi,

>From my point of view, Tempest should verify Cinder  for DefCore
requirements [1] only. Such integration tests won't be just a
re-implementation if a Tempest tests. It will me more extended set of tests
for integration between cinderclient and other projects.


[1] https://github.com/openstack/defcore

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 1:20 PM, Luigi Toscano  wrote:

> On Wednesday 02 of March 2016 12:24:57 Ivan Kolodyazhny wrote:
> > Sean,
> >
> > I do understand why we have tempest for python-cinderclient now.
> >
> > But my point is: tempest runs more than 200 tests per each cinderclient
> > change request which takes a lot of time. Why can't we just introduce few
> > integration tests which will tests nova<=>python-cinderclient API.
> >
> > Also, Nova is not only one consumer of cinderclient. What about Heat? We
> > don't want to break it too but to run all Heat-related Tempest tests is
> not
> > a good idea. We have to implement integration tests between Heat and
> > python-cinderclient too.
>
> So you want to reimplement the tests you already have in Tempest? If the
> issue
> is the time, why not run a relevant subset of the tempests tests?
>
> Ciao
> --
> Luigi
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Luigi Toscano]

On Wednesday 02 of March 2016 12:24:57 Ivan Kolodyazhny wrote:
> Sean,
> 
> I do understand why we have tempest for python-cinderclient now.
> 
> But my point is: tempest runs more than 200 tests per each cinderclient
> change request which takes a lot of time. Why can't we just introduce few
> integration tests which will tests nova<=>python-cinderclient API.
> 
> Also, Nova is not only one consumer of cinderclient. What about Heat? We
> don't want to break it too but to run all Heat-related Tempest tests is not
> a good idea. We have to implement integration tests between Heat and
> python-cinderclient too.

So you want to reimplement the tests you already have in Tempest? If the issue 
is the time, why not run a relevant subset of the tempests tests?

Ciao
-- 
Luigi


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Ivan Kolodyazhny]

Sean,

I do understand why we have tempest for python-cinderclient now.

But my point is: tempest runs more than 200 tests per each cinderclient
change request which takes a lot of time. Why can't we just introduce few
integration tests which will tests nova<=>python-cinderclient API.

Also, Nova is not only one consumer of cinderclient. What about Heat? We
don't want to break it too but to run all Heat-related Tempest tests is not
a good idea. We have to implement integration tests between Heat and
python-cinderclient too.

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Tue, Feb 16, 2016 at 2:05 PM, Sean Dague  wrote:

> On 02/15/2016 02:48 PM, Ivan Kolodyazhny wrote:
> > Hi all,
> >
> > I'll talk mostly about python-cinderclient but the same question could
> > be related for other clients.
> >
> > Now, for python-cinderclient we've got to kinds for
> > functional/integrated jobs:
> >
> > 1) gate-cinderclient-dsvm-functional - a very limited (for now) set of
> > functional tests, most of them were part of tempest CLI tests in the
> past.
> >
> > 2) gate-tempest-dsvm-neutron-src-python-cinderclient - if I understand
> > right, the idea os this job was to have integrated tests to test
> > cinderclient with other projects to verify that new patch to
> > python-cinderclietn won't break any other project.
> > But it does *not* test cinderclient at all, except few attach-related
> > tests because Tempest doesn't use python-*client.
>
> This does test the real world usage of Nova consuming
> python-cinderclient. That's why it's still there. This ensures that a
> cinderclient upcoming release won't completely tank the integrated gate.
> All openstack libraries that get used by all servers in openstack have
> something equivalent.
>
> > The same job was added for python-heatclient but was removed because
> > devstack didn't install Heat for that job [1].
> >
> > We agreed [2] to remove this job from cinderclient gates too, once
> > functional or integration tests will be implemented.
>
> Um, what now?
>
> > There is a proposal to python-cinderclient tests to implement some
> > cross-project testing to make sure, that new python-cinderclient won't
> > break any of existing project who use it.
> >
> > After discussing in IRC with John Griffith (jgriffith) I'm realized that
> > it could be an cross-project initiative in such kind of integration
> > tests. OpenStack Client (OSC) could cover some part of such tests, but
> > does it mean that we'll run OSC tests on every patch to python-*client?
> > We can run only cinder-realated OSC tests on our gates to verify that it
> > doesn't breack OSC and, may be other project.
> >
> > The other option, is to implement tests like [3] per project basis and
> > call it "integration".  Such tests could cover more cases than OSC
> > functional tests and have more project-related test cases, e.g.: test
> > some python-cinderclient specific corner cases, which is not related to
> OSC.
> >
> > IMO, It would be good to have some cross-project decision on how will be
> > implement clients' integration tests per project.
> >
> >
> > [1] https://review.openstack.org/#/c/272411/
> > [2]
> >
> http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-12-16-16.00.log.html
> > [3] https://review.openstack.org/#/c/279432/8
> >
> > Regards,
> > Ivan Kolodyazhny,
> > http://blog.e0ne.info/
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Sean Dague]

On 02/15/2016 02:48 PM, Ivan Kolodyazhny wrote:
> Hi all,
> 
> I'll talk mostly about python-cinderclient but the same question could
> be related for other clients.
> 
> Now, for python-cinderclient we've got to kinds for
> functional/integrated jobs:
> 
> 1) gate-cinderclient-dsvm-functional - a very limited (for now) set of
> functional tests, most of them were part of tempest CLI tests in the past.
> 
> 2) gate-tempest-dsvm-neutron-src-python-cinderclient - if I understand
> right, the idea os this job was to have integrated tests to test
> cinderclient with other projects to verify that new patch to
> python-cinderclietn won't break any other project.
> But it does *not* test cinderclient at all, except few attach-related
> tests because Tempest doesn't use python-*client.

This does test the real world usage of Nova consuming
python-cinderclient. That's why it's still there. This ensures that a
cinderclient upcoming release won't completely tank the integrated gate.
All openstack libraries that get used by all servers in openstack have
something equivalent.

> The same job was added for python-heatclient but was removed because
> devstack didn't install Heat for that job [1].
> 
> We agreed [2] to remove this job from cinderclient gates too, once
> functional or integration tests will be implemented.

Um, what now?

> There is a proposal to python-cinderclient tests to implement some
> cross-project testing to make sure, that new python-cinderclient won't
> break any of existing project who use it. 
> 
> After discussing in IRC with John Griffith (jgriffith) I'm realized that
> it could be an cross-project initiative in such kind of integration
> tests. OpenStack Client (OSC) could cover some part of such tests, but
> does it mean that we'll run OSC tests on every patch to python-*client?
> We can run only cinder-realated OSC tests on our gates to verify that it
> doesn't breack OSC and, may be other project.
> 
> The other option, is to implement tests like [3] per project basis and
> call it "integration".  Such tests could cover more cases than OSC
> functional tests and have more project-related test cases, e.g.: test
> some python-cinderclient specific corner cases, which is not related to OSC.
> 
> IMO, It would be good to have some cross-project decision on how will be
> implement clients' integration tests per project.
> 
> 
> [1] https://review.openstack.org/#/c/272411/
> [2]
> http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-12-16-16.00.log.html
> [3] https://review.openstack.org/#/c/279432/8
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[John Griffith]

On Mon, Feb 15, 2016 at 1:02 PM, Clark Boylan  wrote:

> On Mon, Feb 15, 2016, at 11:48 AM, Ivan Kolodyazhny wrote:
> > Hi all,
> >
> > I'll talk mostly about python-cinderclient but the same question could be
> > related for other clients.
> >
> > Now, for python-cinderclient we've got to kinds for functional/integrated
> > jobs:
> >
> > 1) gate-cinderclient-dsvm-functional - a very limited (for now) set of
> > functional tests, most of them were part of tempest CLI tests in the
> > past.
> >
> > 2) gate-tempest-dsvm-neutron-src-python-cinderclient - if I understand
> > right, the idea os this job was to have integrated tests to test
> > cinderclient with other projects to verify that new patch to
> > python-cinderclietn won't break any other project.
> > But it does *not* test cinderclient at all, except few attach-related
> > tests
> > because Tempest doesn't use python-*client.
>
> Tempest doesn't use python-*client to talk to the APIs but the various
> OpenStack services do use python-*client to talk to the other services.
> Using cinderclient as an example, nova consumes cinderclient to perform
> volume operations in nova/volume/cinder.py. There is value in this
> existing test if those code paths are exercised. Basically ensuring the
> next release of cinderclient does not break nova. It may be the case
> that cinderclient is a bad example because tempest doesn't do volume
> operations through nova, but I am sure for many of the other clients
> these tests do provide value.
>
> >
> > The same job was added for python-heatclient but was removed because
> > devstack didn't install Heat for that job [1].
> >
> > We agreed [2] to remove this job from cinderclient gates too, once
> > functional or integration tests will be implemented.
>
> Just make sure that you don't lose exercising of the above code paths
> when this transition happens. If we don't currently test that code it
> would be a good goal for any new integration testing to do so.
>
> >
> >
> > There is a proposal to python-cinderclient tests to implement some
> > cross-project testing to make sure, that new python-cinderclient won't
> > break any of existing project who use it.
> >
> > After discussing in IRC with John Griffith (jgriffith) I'm realized that
> > it
> > could be an cross-project initiative in such kind of integration tests.
> > OpenStack Client (OSC) could cover some part of such tests, but does it
> > mean that we'll run OSC tests on every patch to python-*client? We can
> > run
> > only cinder-realated OSC tests on our gates to verify that it doesn't
> > breack OSC and, may be other project.
> >
> > The other option, is to implement tests like [3] per project basis and
> > call
> > it "integration".  Such tests could cover more cases than OSC functional
> > tests and have more project-related test cases, e.g.: test some
> > python-cinderclient specific corner cases, which is not related to OSC.
> >
> > IMO, It would be good to have some cross-project decision on how will be
> > implement clients' integration tests per project.
> >
> >
> > [1] https://review.openstack.org/#/c/272411/
> > [2]
> >
> http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-12-16-16.00.log.html
> > [3] https://review.openstack.org/#/c/279432/8
> >
> > Regards,
> > Ivan Kolodyazhny,
> > http://blog.e0ne.info/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Hey Everyone,

So this started after after I made some comments on a patch in CinderClient
that added attach/detach integration tests to cinder's local test repo.  My
first thought was that we should focus on just Cinder functional tests
first, and that maybe the integration tests (including with the clients)
should be centralized, or have a more standardized approach to them.

What I was getting at is that while the Tempest tests don't use the clients
directly, there are a number of places where tempest does end up calling
them indirectly. Volume attach in Nova is a good example of this, while we
don't call NovaClient to do this, the Nova API drills down into
volume/cinder.py which just loads and calls CinderClient in order to issue
the volume related calls that it does.  My thought was that maybe it would
be useful to have a more cross-project effort for things like this. There
are other places we do this in a few projects with Glance, Keystone and
Swift as I recall.

I was actually thinking of a common test-run (similar to dsvm-full) but
something that focuses exclusively on the cross-project calls that are made
via clients.  The idea being that any xxx-client change would have to load
this scenario up with current master and run successfully.  Maybe this is
overkill?  Maybe we don't do the "import xxxClient" in as many 

Re: [openstack-dev] [cinder][all] Integration python-*client tests on gates

[Clark Boylan]

On Mon, Feb 15, 2016, at 11:48 AM, Ivan Kolodyazhny wrote:
> Hi all,
> 
> I'll talk mostly about python-cinderclient but the same question could be
> related for other clients.
> 
> Now, for python-cinderclient we've got to kinds for functional/integrated
> jobs:
> 
> 1) gate-cinderclient-dsvm-functional - a very limited (for now) set of
> functional tests, most of them were part of tempest CLI tests in the
> past.
> 
> 2) gate-tempest-dsvm-neutron-src-python-cinderclient - if I understand
> right, the idea os this job was to have integrated tests to test
> cinderclient with other projects to verify that new patch to
> python-cinderclietn won't break any other project.
> But it does *not* test cinderclient at all, except few attach-related
> tests
> because Tempest doesn't use python-*client.

Tempest doesn't use python-*client to talk to the APIs but the various
OpenStack services do use python-*client to talk to the other services.
Using cinderclient as an example, nova consumes cinderclient to perform
volume operations in nova/volume/cinder.py. There is value in this
existing test if those code paths are exercised. Basically ensuring the
next release of cinderclient does not break nova. It may be the case
that cinderclient is a bad example because tempest doesn't do volume
operations through nova, but I am sure for many of the other clients
these tests do provide value.

> 
> The same job was added for python-heatclient but was removed because
> devstack didn't install Heat for that job [1].
> 
> We agreed [2] to remove this job from cinderclient gates too, once
> functional or integration tests will be implemented.

Just make sure that you don't lose exercising of the above code paths
when this transition happens. If we don't currently test that code it
would be a good goal for any new integration testing to do so.

> 
> 
> There is a proposal to python-cinderclient tests to implement some
> cross-project testing to make sure, that new python-cinderclient won't
> break any of existing project who use it.
> 
> After discussing in IRC with John Griffith (jgriffith) I'm realized that
> it
> could be an cross-project initiative in such kind of integration tests.
> OpenStack Client (OSC) could cover some part of such tests, but does it
> mean that we'll run OSC tests on every patch to python-*client? We can
> run
> only cinder-realated OSC tests on our gates to verify that it doesn't
> breack OSC and, may be other project.
> 
> The other option, is to implement tests like [3] per project basis and
> call
> it "integration".  Such tests could cover more cases than OSC functional
> tests and have more project-related test cases, e.g.: test some
> python-cinderclient specific corner cases, which is not related to OSC.
> 
> IMO, It would be good to have some cross-project decision on how will be
> implement clients' integration tests per project.
> 
> 
> [1] https://review.openstack.org/#/c/272411/
> [2]
> http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-12-16-16.00.log.html
> [3] https://review.openstack.org/#/c/279432/8
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev