Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-09 Thread Ivan Kolodyazhny
John, It's a good option. Let's try it! Also, we can try to find/implement something like [13] for ostestr. [13] https://github.com/mahmoudimus/nose-timer Regards, Ivan Kolodyazhny, http://blog.e0ne.info/ On Mon, Mar 7, 2016 at 7:16 PM, John Griffith wrote: > > > On Mon, Mar 7, 2016 at 8:57

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-07 Thread Jeremy Stanley
On 2016-03-07 23:54:49 +0800 (+0800), Duncan Thomas wrote: > Complexity can be tricky to spot by hand, and expecting reviewers to get it > right all of the time is not a reasonable expectation. > > My ideal would be something that processes the commit and the jenkins logs, > extracts the timing in

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-07 Thread John Griffith
On Mon, Mar 7, 2016 at 8:57 AM, Knight, Clinton wrote: > > > On 3/7/16, 10:45 AM, "Eric Harney" wrote: > > >On 03/06/2016 09:35 PM, John Griffith wrote: > >> On Sat, Mar 5, 2016 at 4:27 PM, Jay S. Bryant > >> >>> wrote: > >> > >>> Ivan, > >>> > >>> I agree that our testing needs improvement. Th

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-07 Thread Knight, Clinton
On 3/7/16, 10:45 AM, "Eric Harney" wrote: >On 03/06/2016 09:35 PM, John Griffith wrote: >> On Sat, Mar 5, 2016 at 4:27 PM, Jay S. Bryant wrote: >> >>> Ivan, >>> >>> I agree that our testing needs improvement. Thanks for starting this >>> thread. >>> >>> With regards to adding a hacking c

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-07 Thread Duncan Thomas
On 7 March 2016 at 23:45, Eric Harney wrote: > > > I'm not really sure that writing a "hacking" check for this is a > worthwhile investment. (It's not a hacking check really, but something > more like what you're describing, but that's beside the point.) > > We should just be looking for large,

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-07 Thread Eric Harney
On 03/06/2016 09:35 PM, John Griffith wrote: > On Sat, Mar 5, 2016 at 4:27 PM, Jay S. Bryant > wrote: > >> Ivan, >> >> I agree that our testing needs improvement. Thanks for starting this >> thread. >> >> With regards to adding a hacking check for tests that run too long ... are >> you thinking t

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-06 Thread John Griffith
On Sat, Mar 5, 2016 at 4:27 PM, Jay S. Bryant wrote: > Ivan, > > I agree that our testing needs improvement. Thanks for starting this > thread. > > With regards to adding a hacking check for tests that run too long ... are > you thinking that we would have a timer that checks or long running job

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-05 Thread Jay S. Bryant
Ivan, I agree that our testing needs improvement. Thanks for starting this thread. With regards to adding a hacking check for tests that run too long ... are you thinking that we would have a timer that checks or long running jobs or something that checks for long sleeps in the testing code

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-03 Thread Gorka Eguileor
On 02/03, Ivan Kolodyazhny wrote: > I'll try to implement such scenario and step-by-step guideline soon. > That would be fantastic!! Thank you very much Looking forward to it. :-) Cheers, Gorka. > Regards, > Ivan Kolodyazhny, > http://blog.e0ne.info/ > > On Wed, Mar 2, 2016 at 5:16 PM, Eric

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Boris Pavlovic
Hi, I will try to be short. - Voting unit test coverage job is ready, and you can just use it as is from rally source code: you need this file https://github.com/openstack/rally/blob/master/tests/ci/cover.sh and this change in tox: https://github.com/openstack/rally/blob/master/tox.ini#L51-

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread MichaƂ Dulko
On 03/02/2016 04:11 PM, Gorka Eguileor wrote: > On 02/03, Ivan Kolodyazhny wrote: >> Eric, >> >> There are Gorka's patches [10] to remove API Races >> >> >> [10] >> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified >> > I looked at Rally a long t

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Ivan Kolodyazhny
I'll try to implement such scenario and step-by-step guideline soon. Regards, Ivan Kolodyazhny, http://blog.e0ne.info/ On Wed, Mar 2, 2016 at 5:16 PM, Eric Harney wrote: > On 03/02/2016 10:07 AM, Ivan Kolodyazhny wrote: > > Eric, > > > > For now, we test Cinder API with some concurrency only wi

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Eric Harney
On 03/02/2016 10:07 AM, Ivan Kolodyazhny wrote: > Eric, > > For now, we test Cinder API with some concurrency only with Rally, so, IMO, > it's reasonable get more scenarios for API races fixes. > > It's not a hard task to implement new scenarios, they are pretty simple: > [11] and [12] > Sure,

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Gorka Eguileor
On 02/03, Ivan Kolodyazhny wrote: > Eric, > > There are Gorka's patches [10] to remove API Races > > > [10] > https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified > I looked at Rally a long time ago so apologies if I'm totally off base here, bu

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Ivan Kolodyazhny
Eric, For now, we test Cinder API with some concurrency only with Rally, so, IMO, it's reasonable get more scenarios for API races fixes. It's not a hard task to implement new scenarios, they are pretty simple: [11] and [12] [11] https://github.com/openstack/rally/blob/master/rally/plugins/opens

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Ivan Kolodyazhny
Mailing List (not for usage questions) < > openstack-dev@lists.openstack.org> > *Subject:* Re: [openstack-dev] [cinder] Proposal: changes to our current > testing process > > > > Eric, > > > > There are Gorka's patches [10] to remove API Races > >

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Arkady_Kanevsky
Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [cinder] Proposal: changes to our current testing process Eric, There are Gorka's patches [10] to remove API Races [10] https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Eric Harney
On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote: > Eric, > > There are Gorka's patches [10] to remove API Races > > > [10] > https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified > > Regards, > Ivan Kolodyazhny, > http://blog.e0ne.info/ > So the

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Ivan Kolodyazhny
Eric, There are Gorka's patches [10] to remove API Races [10] https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified Regards, Ivan Kolodyazhny, http://blog.e0ne.info/ On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney wrote: > On 03/02/2016 06:25 AM,

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Eric Harney
On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote: > Hi Team, > > Here are my thoughts and proposals how to make Cinder testing process > better. I won't cover "3rd party CI's" topic here. I will share my opinion > about current and feature jobs. > > > Unit-tests > >- Long-running tests. I hop

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Sean Dague
t; Scott D'Angelo (scottda) > > From: Ivan Kolodyazhny [e...@e0ne.info] > Sent: Wednesday, March 02, 2016 4:25 AM > To: OpenStack Development Mailing List > Subject: [openstack-dev] [cinder] Proposal: changes to our current testing > process > > Hi Te

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread D'Angelo, Scott
ngelo (scottda) From: Ivan Kolodyazhny [e...@e0ne.info] Sent: Wednesday, March 02, 2016 4:25 AM To: OpenStack Development Mailing List Subject: [openstack-dev] [cinder] Proposal: changes to our current testing process Hi Team, Here are my thoughts and proposals h

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Andrey Kurilin
Hi Ivan! On Wed, Mar 2, 2016 at 1:25 PM, Ivan Kolodyazhny wrote: > Hi Team, > > Here are my thoughts and proposals how to make Cinder testing process > better. I won't cover "3rd party CI's" topic here. I will share my opinion > about current and feature jobs. > > > Unit-tests > >- Long-runn

[openstack-dev] [cinder] Proposal: changes to our current testing process

2016-03-02 Thread Ivan Kolodyazhny
Hi Team, Here are my thoughts and proposals how to make Cinder testing process better. I won't cover "3rd party CI's" topic here. I will share my opinion about current and feature jobs. Unit-tests - Long-running tests. I hope, everybody will agree that unit-tests must be quite simple and