Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Ivan Kolodyazhny]

John,

It's a good option. Let's try it!

Also, we can try to find/implement something like [13] for ostestr.


[13] https://github.com/mahmoudimus/nose-timer

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Mon, Mar 7, 2016 at 7:16 PM, John Griffith 
wrote:

>
>
> On Mon, Mar 7, 2016 at 8:57 AM, Knight, Clinton  > wrote:
>
>>
>>
>> On 3/7/16, 10:45 AM, "Eric Harney"  wrote:
>>
>> >On 03/06/2016 09:35 PM, John Griffith wrote:
>> >> On Sat, Mar 5, 2016 at 4:27 PM, Jay S. Bryant
>> >>> >>> wrote:
>> >>
>> >>> Ivan,
>> >>>
>> >>> I agree that our testing needs improvement.  Thanks for starting this
>> >>> thread.
>> >>>
>> >>> With regards to adding a hacking check for tests that run too long ...
>> >>>are
>> >>> you thinking that we would have a timer that checks or long running
>> >>>jobs or
>> >>> something that checks for long sleeps in the testing code?  Just
>> >>>curious
>> >>> your ideas for tackling that situation.  Would be interested in
>> helping
>> >>> with that, perhaps.
>> >>>
>> >>> Thanks!
>> >>> Jay
>> >>>
>> >>>
>> >>> On 03/02/2016 05:25 AM, Ivan Kolodyazhny wrote:
>> >>>
>> >>> Hi Team,
>> >>>
>> >>> Here are my thoughts and proposals how to make Cinder testing process
>> >>> better. I won't cover "3rd party CI's" topic here. I will share my
>> >>>opinion
>> >>> about current and feature jobs.
>> >>>
>> >>>
>> >>> Unit-tests
>> >>>
>> >>>- Long-running tests. I hope, everybody will agree that unit-tests
>> >>>must be quite simple and very fast. Unit tests which takes more
>> >>>than 3-5
>> >>>seconds should be refactored and/or moved to 'integration' tests.
>> >>>Thanks to Tom Barron for several fixes like [1]. IMO, we it would
>> be
>> >>>good to have some hacking checks to prevent such issues in a
>> future.
>> >>>
>> >>>- Tests coverage. We don't check it in an automatic way on gates.
>> >>>Usually, we require to add some unit-tests during code review
>> >>>process. Why
>> >>>can't we add coverage job to our CI and do not merge new patches,
>> >>>with
>> >>>will decrease tests coverage rate? Maybe, such job could be voting
>> >>>in a
>> >>>future to not ignore it. For now, there is not simple way to check
>> >>>coverage
>> >>>because 'tox -e cover' output is not useful [2].
>>
>> The Manila project has a coverage job that may be of interest to Cinder.
>> It’s not perfect, because sometimes the periodic loopingcall routines run
>> during the test run and sometimes not, leading to false negatives.  But
>> most of the time it’s a handy confirmation that the unit test coverage
>> didn’t decline due to a patch.  Look at the manila-coverage job in this
>> example:  https://review.openstack.org/#/c/287575/
>>
>> >>>
>> >>>
>> >>> Functional tests for Cinder
>> >>>
>> >>> We introduced some functional tests last month [3]. Here is a patch to
>> >>> infra to add new job [4]. Because these tests were moved from
>> >>>unit-tests, I
>> >>> think we're OK to make this job voting. Such tests should not be a
>> >>> replacement for Tempest. They even could tests Cinder with Fake Driver
>> >>>to
>> >>> make it faster and not related on storage backends issues.
>> >>>
>> >>>
>> >>> Tempest in-tree tests
>> >>>
>> >>> Sean started work on it [5] and I think it's a good idea to get them
>> in
>> >>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
>> >>>real
>> >>> backend.
>> >>>
>> >>>
>> >>> Functional tests for python-brick-cinderclient-ext
>> >>>
>> >>> There are patches that introduces functional tests [6] and new job
>> [7].
>> >>>
>> >>>
>> >>> Functional tests for python-cinderclient
>> >>>
>> >>> We've got a very limited set of such tests and non-voting job. IMO, we
>> >>>can
>> >>> run them even with Cinder Fake Driver to make them not depended on a
>> >>> storage backend and make it faster. I believe, we can make this job
>> >>>voting
>> >>> soon. Also, we need more contributors to this kind of tests.
>> >>>
>> >>>
>> >>> Integrated tests for python-cinderclient
>> >>>
>> >>> We need such tests to make sure that we won't break Nova, Heat or
>> other
>> >>> python-cinderclient consumers with a next merged patch. There is a
>> >>>thread
>> >>> in openstack-dev ML about such tests [8] and proposal [9] to introduce
>> >>>them
>> >>> to python-cinderclient.
>> >>>
>> >>>
>> >>> Rally tests
>> >>>
>> >>> IMO, it would be good to have new Rally scenarios for every patches
>> >>>like
>> >>> 'improves performance', 'fixes concurrency issues', etc. Even if we as
>> >>>a
>> >>> Cinder community don't have enough time to implement them, we have to
>> >>>ask
>> >>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints
>> >>>if
>> >>> needed.
>> >>>
>> >>>
>> >>> [1] https://review.openstack.org/#/c/282861/
>> >>> [2] http://paste.openstack.org/show/488925/
>> >>> [3] https://review.openstack.org/#/c/267801/
>> >>> 

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Jeremy Stanley]

On 2016-03-07 23:54:49 +0800 (+0800), Duncan Thomas wrote:
> Complexity can be tricky to spot by hand, and expecting reviewers to get it
> right all of the time is not a reasonable expectation.
> 
> My ideal would be something that processes the commit and the jenkins logs,
> extracts the timing info of any new tests, and if they are outside some
> (fairly tight) window, then posts a comment to the review indicating that
> these tests should get closer scrutiny. This does not remove reviewer
> judgement from the equation, just provides a helpful prod that there's
> something to be considered.

Has any analysis been performed on the existing body of test timing
data so far (e.g., by querying the subunit2sql data used for
http://status.openstack.org/openstack-health/#/job/gate-cinder-python27?groupKey=project=hour
 >)?
I suspect you'll find that evaluating durations of tests on the
basis of individual runs is fraught with false positive results due
to the significant variability in our CI workers across different
service providers. If it really were predictably consistent, then
it seems like just adding a timeout fixture at whatever you
determine is a sane duration would be sufficient.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[John Griffith]

On Mon, Mar 7, 2016 at 8:57 AM, Knight, Clinton 
wrote:

>
>
> On 3/7/16, 10:45 AM, "Eric Harney"  wrote:
>
> >On 03/06/2016 09:35 PM, John Griffith wrote:
> >> On Sat, Mar 5, 2016 at 4:27 PM, Jay S. Bryant
> >> >>> wrote:
> >>
> >>> Ivan,
> >>>
> >>> I agree that our testing needs improvement.  Thanks for starting this
> >>> thread.
> >>>
> >>> With regards to adding a hacking check for tests that run too long ...
> >>>are
> >>> you thinking that we would have a timer that checks or long running
> >>>jobs or
> >>> something that checks for long sleeps in the testing code?  Just
> >>>curious
> >>> your ideas for tackling that situation.  Would be interested in helping
> >>> with that, perhaps.
> >>>
> >>> Thanks!
> >>> Jay
> >>>
> >>>
> >>> On 03/02/2016 05:25 AM, Ivan Kolodyazhny wrote:
> >>>
> >>> Hi Team,
> >>>
> >>> Here are my thoughts and proposals how to make Cinder testing process
> >>> better. I won't cover "3rd party CI's" topic here. I will share my
> >>>opinion
> >>> about current and feature jobs.
> >>>
> >>>
> >>> Unit-tests
> >>>
> >>>- Long-running tests. I hope, everybody will agree that unit-tests
> >>>must be quite simple and very fast. Unit tests which takes more
> >>>than 3-5
> >>>seconds should be refactored and/or moved to 'integration' tests.
> >>>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
> >>>good to have some hacking checks to prevent such issues in a future.
> >>>
> >>>- Tests coverage. We don't check it in an automatic way on gates.
> >>>Usually, we require to add some unit-tests during code review
> >>>process. Why
> >>>can't we add coverage job to our CI and do not merge new patches,
> >>>with
> >>>will decrease tests coverage rate? Maybe, such job could be voting
> >>>in a
> >>>future to not ignore it. For now, there is not simple way to check
> >>>coverage
> >>>because 'tox -e cover' output is not useful [2].
>
> The Manila project has a coverage job that may be of interest to Cinder.
> It’s not perfect, because sometimes the periodic loopingcall routines run
> during the test run and sometimes not, leading to false negatives.  But
> most of the time it’s a handy confirmation that the unit test coverage
> didn’t decline due to a patch.  Look at the manila-coverage job in this
> example:  https://review.openstack.org/#/c/287575/
>
> >>>
> >>>
> >>> Functional tests for Cinder
> >>>
> >>> We introduced some functional tests last month [3]. Here is a patch to
> >>> infra to add new job [4]. Because these tests were moved from
> >>>unit-tests, I
> >>> think we're OK to make this job voting. Such tests should not be a
> >>> replacement for Tempest. They even could tests Cinder with Fake Driver
> >>>to
> >>> make it faster and not related on storage backends issues.
> >>>
> >>>
> >>> Tempest in-tree tests
> >>>
> >>> Sean started work on it [5] and I think it's a good idea to get them in
> >>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
> >>>real
> >>> backend.
> >>>
> >>>
> >>> Functional tests for python-brick-cinderclient-ext
> >>>
> >>> There are patches that introduces functional tests [6] and new job [7].
> >>>
> >>>
> >>> Functional tests for python-cinderclient
> >>>
> >>> We've got a very limited set of such tests and non-voting job. IMO, we
> >>>can
> >>> run them even with Cinder Fake Driver to make them not depended on a
> >>> storage backend and make it faster. I believe, we can make this job
> >>>voting
> >>> soon. Also, we need more contributors to this kind of tests.
> >>>
> >>>
> >>> Integrated tests for python-cinderclient
> >>>
> >>> We need such tests to make sure that we won't break Nova, Heat or other
> >>> python-cinderclient consumers with a next merged patch. There is a
> >>>thread
> >>> in openstack-dev ML about such tests [8] and proposal [9] to introduce
> >>>them
> >>> to python-cinderclient.
> >>>
> >>>
> >>> Rally tests
> >>>
> >>> IMO, it would be good to have new Rally scenarios for every patches
> >>>like
> >>> 'improves performance', 'fixes concurrency issues', etc. Even if we as
> >>>a
> >>> Cinder community don't have enough time to implement them, we have to
> >>>ask
> >>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints
> >>>if
> >>> needed.
> >>>
> >>>
> >>> [1] https://review.openstack.org/#/c/282861/
> >>> [2] http://paste.openstack.org/show/488925/
> >>> [3] https://review.openstack.org/#/c/267801/
> >>> [4] https://review.openstack.org/#/c/287115/
> >>> [5] https://review.openstack.org/#/c/274471/
> >>> [6] https://review.openstack.org/#/c/265811/
> >>> [7] https://review.openstack.org/#/c/265925/
> >>> [8]
> >>>
> >>>
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.htm
> >>>l
> >>> [9] https://review.openstack.org/#/c/279432/
> >>>
> >>>
> >>> Regards,
> >>> Ivan Kolodyazhny,
> >>> http://blog.e0ne.info/
> >>>
> >>>
> 

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Knight, Clinton]

On 3/7/16, 10:45 AM, "Eric Harney"  wrote:

>On 03/06/2016 09:35 PM, John Griffith wrote:
>> On Sat, Mar 5, 2016 at 4:27 PM, Jay S. Bryant
>>>> wrote:
>> 
>>> Ivan,
>>>
>>> I agree that our testing needs improvement.  Thanks for starting this
>>> thread.
>>>
>>> With regards to adding a hacking check for tests that run too long ...
>>>are
>>> you thinking that we would have a timer that checks or long running
>>>jobs or
>>> something that checks for long sleeps in the testing code?  Just
>>>curious
>>> your ideas for tackling that situation.  Would be interested in helping
>>> with that, perhaps.
>>>
>>> Thanks!
>>> Jay
>>>
>>>
>>> On 03/02/2016 05:25 AM, Ivan Kolodyazhny wrote:
>>>
>>> Hi Team,
>>>
>>> Here are my thoughts and proposals how to make Cinder testing process
>>> better. I won't cover "3rd party CI's" topic here. I will share my
>>>opinion
>>> about current and feature jobs.
>>>
>>>
>>> Unit-tests
>>>
>>>- Long-running tests. I hope, everybody will agree that unit-tests
>>>must be quite simple and very fast. Unit tests which takes more
>>>than 3-5
>>>seconds should be refactored and/or moved to 'integration' tests.
>>>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>>>good to have some hacking checks to prevent such issues in a future.
>>>
>>>- Tests coverage. We don't check it in an automatic way on gates.
>>>Usually, we require to add some unit-tests during code review
>>>process. Why
>>>can't we add coverage job to our CI and do not merge new patches,
>>>with
>>>will decrease tests coverage rate? Maybe, such job could be voting
>>>in a
>>>future to not ignore it. For now, there is not simple way to check
>>>coverage
>>>because 'tox -e cover' output is not useful [2].

The Manila project has a coverage job that may be of interest to Cinder.
It’s not perfect, because sometimes the periodic loopingcall routines run
during the test run and sometimes not, leading to false negatives.  But
most of the time it’s a handy confirmation that the unit test coverage
didn’t decline due to a patch.  Look at the manila-coverage job in this
example:  https://review.openstack.org/#/c/287575/

>>>
>>>
>>> Functional tests for Cinder
>>>
>>> We introduced some functional tests last month [3]. Here is a patch to
>>> infra to add new job [4]. Because these tests were moved from
>>>unit-tests, I
>>> think we're OK to make this job voting. Such tests should not be a
>>> replacement for Tempest. They even could tests Cinder with Fake Driver
>>>to
>>> make it faster and not related on storage backends issues.
>>>
>>>
>>> Tempest in-tree tests
>>>
>>> Sean started work on it [5] and I think it's a good idea to get them in
>>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
>>>real
>>> backend.
>>>
>>>
>>> Functional tests for python-brick-cinderclient-ext
>>>
>>> There are patches that introduces functional tests [6] and new job [7].
>>>
>>>
>>> Functional tests for python-cinderclient
>>>
>>> We've got a very limited set of such tests and non-voting job. IMO, we
>>>can
>>> run them even with Cinder Fake Driver to make them not depended on a
>>> storage backend and make it faster. I believe, we can make this job
>>>voting
>>> soon. Also, we need more contributors to this kind of tests.
>>>
>>>
>>> Integrated tests for python-cinderclient
>>>
>>> We need such tests to make sure that we won't break Nova, Heat or other
>>> python-cinderclient consumers with a next merged patch. There is a
>>>thread
>>> in openstack-dev ML about such tests [8] and proposal [9] to introduce
>>>them
>>> to python-cinderclient.
>>>
>>>
>>> Rally tests
>>>
>>> IMO, it would be good to have new Rally scenarios for every patches
>>>like
>>> 'improves performance', 'fixes concurrency issues', etc. Even if we as
>>>a
>>> Cinder community don't have enough time to implement them, we have to
>>>ask
>>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints
>>>if
>>> needed.
>>>
>>>
>>> [1] https://review.openstack.org/#/c/282861/
>>> [2] http://paste.openstack.org/show/488925/
>>> [3] https://review.openstack.org/#/c/267801/
>>> [4] https://review.openstack.org/#/c/287115/
>>> [5] https://review.openstack.org/#/c/274471/
>>> [6] https://review.openstack.org/#/c/265811/
>>> [7] https://review.openstack.org/#/c/265925/
>>> [8]
>>> 
>>>http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.htm
>>>l
>>> [9] https://review.openstack.org/#/c/279432/
>>>
>>>
>>> Regards,
>>> Ivan Kolodyazhny,
>>> http://blog.e0ne.info/
>>>
>>>
>>>
>>> ​We could just parse out the tox slowest tests output we already have.
>>> Do
>> something like pylint where we look at existing/current slowest test and
>> balk if that's exceeded.
>> 
>> Thoughts?
>> 
>> John​
>> 
>
>I'm not really sure that writing a "hacking" check for this is a
>worthwhile investment.  (It's not a hacking check really, but something

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Duncan Thomas]

On 7 March 2016 at 23:45, Eric Harney  wrote:

>
>
> I'm not really sure that writing a "hacking" check for this is a
> worthwhile investment.  (It's not a hacking check really, but something
> more like what you're describing, but that's beside the point.)
>
> We should just be looking for large, complex unit tests in review, and
> the ones that we already have should be moving towards the functional
> test area anyway.
>
> So what would the objective here be exactly?
>
>
Complexity can be tricky to spot by hand, and expecting reviewers to get it
right all of the time is not a reasonable expectation.

My ideal would be something that processes the commit and the jenkins logs,
extracts the timing info of any new tests, and if they are outside some
(fairly tight) window, then posts a comment to the review indicating that
these tests should get closer scrutiny. This does not remove reviewer
judgement from the equation, just provides a helpful prod that there's
something to be considered.

-- 
Duncan Thomas
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Eric Harney]

On 03/06/2016 09:35 PM, John Griffith wrote:
> On Sat, Mar 5, 2016 at 4:27 PM, Jay S. Bryant > wrote:
> 
>> Ivan,
>>
>> I agree that our testing needs improvement.  Thanks for starting this
>> thread.
>>
>> With regards to adding a hacking check for tests that run too long ... are
>> you thinking that we would have a timer that checks or long running jobs or
>> something that checks for long sleeps in the testing code?  Just curious
>> your ideas for tackling that situation.  Would be interested in helping
>> with that, perhaps.
>>
>> Thanks!
>> Jay
>>
>>
>> On 03/02/2016 05:25 AM, Ivan Kolodyazhny wrote:
>>
>> Hi Team,
>>
>> Here are my thoughts and proposals how to make Cinder testing process
>> better. I won't cover "3rd party CI's" topic here. I will share my opinion
>> about current and feature jobs.
>>
>>
>> Unit-tests
>>
>>- Long-running tests. I hope, everybody will agree that unit-tests
>>must be quite simple and very fast. Unit tests which takes more than 3-5
>>seconds should be refactored and/or moved to 'integration' tests.
>>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>>good to have some hacking checks to prevent such issues in a future.
>>
>>- Tests coverage. We don't check it in an automatic way on gates.
>>Usually, we require to add some unit-tests during code review process. Why
>>can't we add coverage job to our CI and do not merge new patches, with
>>will decrease tests coverage rate? Maybe, such job could be voting in a
>>future to not ignore it. For now, there is not simple way to check 
>> coverage
>>because 'tox -e cover' output is not useful [2].
>>
>>
>> Functional tests for Cinder
>>
>> We introduced some functional tests last month [3]. Here is a patch to
>> infra to add new job [4]. Because these tests were moved from unit-tests, I
>> think we're OK to make this job voting. Such tests should not be a
>> replacement for Tempest. They even could tests Cinder with Fake Driver to
>> make it faster and not related on storage backends issues.
>>
>>
>> Tempest in-tree tests
>>
>> Sean started work on it [5] and I think it's a good idea to get them in
>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
>> backend.
>>
>>
>> Functional tests for python-brick-cinderclient-ext
>>
>> There are patches that introduces functional tests [6] and new job [7].
>>
>>
>> Functional tests for python-cinderclient
>>
>> We've got a very limited set of such tests and non-voting job. IMO, we can
>> run them even with Cinder Fake Driver to make them not depended on a
>> storage backend and make it faster. I believe, we can make this job voting
>> soon. Also, we need more contributors to this kind of tests.
>>
>>
>> Integrated tests for python-cinderclient
>>
>> We need such tests to make sure that we won't break Nova, Heat or other
>> python-cinderclient consumers with a next merged patch. There is a thread
>> in openstack-dev ML about such tests [8] and proposal [9] to introduce them
>> to python-cinderclient.
>>
>>
>> Rally tests
>>
>> IMO, it would be good to have new Rally scenarios for every patches like
>> 'improves performance', 'fixes concurrency issues', etc. Even if we as a
>> Cinder community don't have enough time to implement them, we have to ask
>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
>> needed.
>>
>>
>> [1] https://review.openstack.org/#/c/282861/
>> [2] http://paste.openstack.org/show/488925/
>> [3] https://review.openstack.org/#/c/267801/
>> [4] https://review.openstack.org/#/c/287115/
>> [5] https://review.openstack.org/#/c/274471/
>> [6] https://review.openstack.org/#/c/265811/
>> [7] https://review.openstack.org/#/c/265925/
>> [8]
>> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
>> [9] https://review.openstack.org/#/c/279432/
>>
>>
>> Regards,
>> Ivan Kolodyazhny,
>> http://blog.e0ne.info/
>>
>>
>>
>> ​We could just parse out the tox slowest tests output we already have.  Do
> something like pylint where we look at existing/current slowest test and
> balk if that's exceeded.
> 
> Thoughts?
> 
> John​
> 

I'm not really sure that writing a "hacking" check for this is a
worthwhile investment.  (It's not a hacking check really, but something
more like what you're describing, but that's beside the point.)

We should just be looking for large, complex unit tests in review, and
the ones that we already have should be moving towards the functional
test area anyway.

So what would the objective here be exactly?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[John Griffith]

On Sat, Mar 5, 2016 at 4:27 PM, Jay S. Bryant  wrote:

> Ivan,
>
> I agree that our testing needs improvement.  Thanks for starting this
> thread.
>
> With regards to adding a hacking check for tests that run too long ... are
> you thinking that we would have a timer that checks or long running jobs or
> something that checks for long sleeps in the testing code?  Just curious
> your ideas for tackling that situation.  Would be interested in helping
> with that, perhaps.
>
> Thanks!
> Jay
>
>
> On 03/02/2016 05:25 AM, Ivan Kolodyazhny wrote:
>
> Hi Team,
>
> Here are my thoughts and proposals how to make Cinder testing process
> better. I won't cover "3rd party CI's" topic here. I will share my opinion
> about current and feature jobs.
>
>
> Unit-tests
>
>- Long-running tests. I hope, everybody will agree that unit-tests
>must be quite simple and very fast. Unit tests which takes more than 3-5
>seconds should be refactored and/or moved to 'integration' tests.
>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>good to have some hacking checks to prevent such issues in a future.
>
>- Tests coverage. We don't check it in an automatic way on gates.
>Usually, we require to add some unit-tests during code review process. Why
>can't we add coverage job to our CI and do not merge new patches, with
>will decrease tests coverage rate? Maybe, such job could be voting in a
>future to not ignore it. For now, there is not simple way to check coverage
>because 'tox -e cover' output is not useful [2].
>
>
> Functional tests for Cinder
>
> We introduced some functional tests last month [3]. Here is a patch to
> infra to add new job [4]. Because these tests were moved from unit-tests, I
> think we're OK to make this job voting. Such tests should not be a
> replacement for Tempest. They even could tests Cinder with Fake Driver to
> make it faster and not related on storage backends issues.
>
>
> Tempest in-tree tests
>
> Sean started work on it [5] and I think it's a good idea to get them in
> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> backend.
>
>
> Functional tests for python-brick-cinderclient-ext
>
> There are patches that introduces functional tests [6] and new job [7].
>
>
> Functional tests for python-cinderclient
>
> We've got a very limited set of such tests and non-voting job. IMO, we can
> run them even with Cinder Fake Driver to make them not depended on a
> storage backend and make it faster. I believe, we can make this job voting
> soon. Also, we need more contributors to this kind of tests.
>
>
> Integrated tests for python-cinderclient
>
> We need such tests to make sure that we won't break Nova, Heat or other
> python-cinderclient consumers with a next merged patch. There is a thread
> in openstack-dev ML about such tests [8] and proposal [9] to introduce them
> to python-cinderclient.
>
>
> Rally tests
>
> IMO, it would be good to have new Rally scenarios for every patches like
> 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> Cinder community don't have enough time to implement them, we have to ask
> for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
> needed.
>
>
> [1] https://review.openstack.org/#/c/282861/
> [2] http://paste.openstack.org/show/488925/
> [3] https://review.openstack.org/#/c/267801/
> [4] https://review.openstack.org/#/c/287115/
> [5] https://review.openstack.org/#/c/274471/
> [6] https://review.openstack.org/#/c/265811/
> [7] https://review.openstack.org/#/c/265925/
> [8]
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> [9] https://review.openstack.org/#/c/279432/
>
>
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> ​We could just parse out the tox slowest tests output we already have.  Do
something like pylint where we look at existing/current slowest test and
balk if that's exceeded.

Thoughts?

John​
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Jay S. Bryant]

Ivan,

I agree that our testing needs improvement.  Thanks for starting this 
thread.


With regards to adding a hacking check for tests that run too long ... 
are you thinking that we would have a timer that checks or long running 
jobs or something that checks for long sleeps in the testing code?  Just 
curious your ideas for tackling that situation.  Would be interested in 
helping with that, perhaps.


Thanks!
Jay

On 03/02/2016 05:25 AM, Ivan Kolodyazhny wrote:

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process 
better. I won't cover "3rd party CI's" topic here. I will share my 
opinion about current and feature jobs.



Unit-tests

  * Long-running tests. I hope, everybody will agree that unit-tests
must be quite simple and very fast. Unit tests which takes more
than 3-5 seconds should be refactored and/or moved to
'integration' tests.
Thanks to Tom Barron for several fixes like [1]. IMO, we it would
be good to have some hacking checks to prevent such issues in a
future.

  * Tests coverage. We don't check it in an automatic way on gates.
Usually, we require to add some unit-tests during code review
process. Why can't we add coverage job to our CI and do not merge
new patches, with will decrease tests coverage rate? Maybe, such
job could be voting in a future to not ignore it. For now, there
is not simple way to check coverage because 'tox -e cover' output
is not useful [2].


Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to 
infra to add new job [4]. Because these tests were moved from 
unit-tests, I think we're OK to make this job voting. Such tests 
should not be a replacement for Tempest. They even could tests Cinder 
with Fake Driver to make it faster and not related on storage backends 
issues.



Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them 
in Cinder repo to run them on Tempest jobs and 3-rd party CIs against 
a real backend.



Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].


Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we 
can run them even with Cinder Fake Driver to make them not depended on 
a storage backend and make it faster. I believe, we can make this job 
voting soon. Also, we need more contributors to this kind of tests.



Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or 
other python-cinderclient consumers with a next merged patch. There is 
a thread in openstack-dev ML about such tests [8] and proposal [9] to 
introduce them to python-cinderclient.



Rally tests

IMO, it would be good to have new Rally scenarios for every patches 
like 'improves performance', 'fixes concurrency issues', etc. Even if 
we as a Cinder community don't have enough time to implement them, we 
have to ask for them in reviews, openstack-dev ML, file Rally bugs and 
blueprints if needed.



[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8] 
http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html

[9] https://review.openstack.org/#/c/279432/


Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Gorka Eguileor]

On 02/03, Ivan Kolodyazhny wrote:
> I'll try to implement such scenario and step-by-step guideline soon.
> 

That would be fantastic!! Thank you very much

Looking forward to it.  :-)

Cheers,
Gorka.

> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 
> On Wed, Mar 2, 2016 at 5:16 PM, Eric Harney  wrote:
> 
> > On 03/02/2016 10:07 AM, Ivan Kolodyazhny wrote:
> > > Eric,
> > >
> > > For now, we test Cinder API with some concurrency only with Rally, so,
> > IMO,
> > > it's reasonable get more scenarios for API races fixes.
> > >
> > > It's not a hard task to implement new scenarios, they are pretty simple:
> > > [11] and [12]
> > >
> >
> > Sure, these are simple, but I think it's nowhere near that simple to
> > write a scenario which will prove that "remove API races" works correctly.
> >
> > > [11]
> > >
> > https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535
> > > [12]
> > >
> > https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516
> > >
> > > Regards,
> > > Ivan Kolodyazhny,
> > > http://blog.e0ne.info/
> > >
> > > On Wed, Mar 2, 2016 at 4:50 PM, Eric Harney  wrote:
> > >
> > >> On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:
> > >>> Eric,
> > >>>
> > >>> There are Gorka's patches [10] to remove API Races
> > >>>
> > >>>
> > >>> [10]
> > >>>
> > >>
> > https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> > >>>
> > >>> Regards,
> > >>> Ivan Kolodyazhny,
> > >>> http://blog.e0ne.info/
> > >>>
> > >>
> > >> So the second part of my question is, is writing a Rally job to prove
> > >> out that code a reasonable task?
> > >>
> > >> How hard is that to do and what does it look like?
> > >>
> > >>> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney 
> > wrote:
> > >>>
> >  On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
> > > Hi Team,
> > >
> > > Here are my thoughts and proposals how to make Cinder testing process
> > > better. I won't cover "3rd party CI's" topic here. I will share my
> >  opinion
> > > about current and feature jobs.
> > >
> > >
> > > Unit-tests
> > >
> > >- Long-running tests. I hope, everybody will agree that unit-tests
> >  must
> > >be quite simple and very fast. Unit tests which takes more than
> > 3-5
> >  seconds
> > >should be refactored and/or moved to 'integration' tests.
> > >Thanks to Tom Barron for several fixes like [1]. IMO, we it would
> > be
> > >good to have some hacking checks to prevent such issues in a
> > future.
> > >
> > >- Tests coverage. We don't check it in an automatic way on gates.
> > >Usually, we require to add some unit-tests during code review
> >  process. Why
> > >can't we add coverage job to our CI and do not merge new patches,
> > >> with
> > >will decrease tests coverage rate? Maybe, such job could be voting
> > >> in
> >  a
> > >future to not ignore it. For now, there is not simple way to check
> >  coverage
> > >because 'tox -e cover' output is not useful [2].
> > >
> > >
> > > Functional tests for Cinder
> > >
> > > We introduced some functional tests last month [3]. Here is a patch
> > to
> > > infra to add new job [4]. Because these tests were moved from
> >  unit-tests, I
> > > think we're OK to make this job voting. Such tests should not be a
> > > replacement for Tempest. They even could tests Cinder with Fake
> > Driver
> > >> to
> > > make it faster and not related on storage backends issues.
> > >
> > >
> > > Tempest in-tree tests
> > >
> > > Sean started work on it [5] and I think it's a good idea to get them
> > in
> > > Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
> > >> real
> > > backend.
> > >
> > >
> > > Functional tests for python-brick-cinderclient-ext
> > >
> > > There are patches that introduces functional tests [6] and new job
> > [7].
> > >
> > >
> > > Functional tests for python-cinderclient
> > >
> > > We've got a very limited set of such tests and non-voting job. IMO,
> > we
> >  can
> > > run them even with Cinder Fake Driver to make them not depended on a
> > > storage backend and make it faster. I believe, we can make this job
> >  voting
> > > soon. Also, we need more contributors to this kind of tests.
> > >
> > >
> > > Integrated tests for python-cinderclient
> > >
> > > We need such tests to make sure that we won't break Nova, Heat or
> > other
> > > python-cinderclient consumers with a next merged patch. There is a
> > >> thread
> > > in openstack-dev ML about such tests [8] and proposal [9] to
> > introduce
> >  them
> > > to python-cinderclient.
> > >
> > >
> > > Rally tests
> > >
> 

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Boris Pavlovic]

Hi,

I will try to be short.

- Voting unit test coverage job is ready, and you can just use it as is
from rally source code:
   you need this file
https://github.com/openstack/rally/blob/master/tests/ci/cover.sh
   and this change in tox:
https://github.com/openstack/rally/blob/master/tox.ini#L51-L52

- Rally is in gates, and it's easy to add jobs in any project. If you have
any problems with this
  just ping me or someone from Rally team (or just write comment in
openstack-rally IRC)

- Rally was a performance tool, however that change  and now we are more
like common testing
  framework, that allows to do various kinds of testing (perf, volume,
stress, functional, ...)

- In Rally we were testing all plugins with relative small concurrency
(already for more then 1.5 year),
  and I can say that we faced a lot of issues with concurrency (and we are
still facing).
  However I can't give guarantee that we are facing 100% of cases
  (however facing most of issues is better then nothing)



Best regards,
Boris Pavlovic

On Wed, Mar 2, 2016 at 7:30 AM, Michał Dulko  wrote:

> On 03/02/2016 04:11 PM, Gorka Eguileor wrote:
> > On 02/03, Ivan Kolodyazhny wrote:
> >> Eric,
> >>
> >> There are Gorka's patches [10] to remove API Races
> >>
> >>
> >> [10]
> >>
> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> >>
> > I looked at Rally a long time ago so apologies if I'm totally off base
> > here, but it looked like it was a performance evaluation tool, which
> > means that it probably won't help to check for API Races (at least I
> > didn't see how when I looked).
> >
> > Many of the API races only happen if you simultaneously try the same
> > operation multiple times against the same resource or if there are
> > different operations that are trying to operate on the same resource.
> >
> > On the first case if Rally allowed it we could test it because we know
> > only 1 of the operations should succeed, but on the second case when we
> > are talking about preventing races from different operations there is no
> > way to know what the result should be, since the order in which those
> > operations are executed on each test run will determine which one will
> > fail and which one will succeed.
> >
> > I'm not trying to go against the general idea of adding rally tests, I
> > just think that they won't help in the case of the API races.
>
> You're probably right - Rally would need to cache API responses to
> parallel runs, predict the result of accepted requests (these which
> haven't received VolumeIsBusy) and then verify it. In case of API race
> conditions things explode inside the stack, and not on the API response
> level. The issue is that two requests, that shouldn't ever be accepted
> together, get positive API response.
>
> I cannot say it's impossible to implement a situation like that as Rally
> resource, but definitely it seems non-trivial to verify if result is
> correct.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Michał Dulko]

On 03/02/2016 04:11 PM, Gorka Eguileor wrote:
> On 02/03, Ivan Kolodyazhny wrote:
>> Eric,
>>
>> There are Gorka's patches [10] to remove API Races
>>
>>
>> [10]
>> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
>>
> I looked at Rally a long time ago so apologies if I'm totally off base
> here, but it looked like it was a performance evaluation tool, which
> means that it probably won't help to check for API Races (at least I
> didn't see how when I looked).
>
> Many of the API races only happen if you simultaneously try the same
> operation multiple times against the same resource or if there are
> different operations that are trying to operate on the same resource.
>
> On the first case if Rally allowed it we could test it because we know
> only 1 of the operations should succeed, but on the second case when we
> are talking about preventing races from different operations there is no
> way to know what the result should be, since the order in which those
> operations are executed on each test run will determine which one will
> fail and which one will succeed.
>
> I'm not trying to go against the general idea of adding rally tests, I
> just think that they won't help in the case of the API races.

You're probably right - Rally would need to cache API responses to
parallel runs, predict the result of accepted requests (these which
haven't received VolumeIsBusy) and then verify it. In case of API race
conditions things explode inside the stack, and not on the API response
level. The issue is that two requests, that shouldn't ever be accepted
together, get positive API response.

I cannot say it's impossible to implement a situation like that as Rally
resource, but definitely it seems non-trivial to verify if result is
correct.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Ivan Kolodyazhny]

I'll try to implement such scenario and step-by-step guideline soon.

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 5:16 PM, Eric Harney  wrote:

> On 03/02/2016 10:07 AM, Ivan Kolodyazhny wrote:
> > Eric,
> >
> > For now, we test Cinder API with some concurrency only with Rally, so,
> IMO,
> > it's reasonable get more scenarios for API races fixes.
> >
> > It's not a hard task to implement new scenarios, they are pretty simple:
> > [11] and [12]
> >
>
> Sure, these are simple, but I think it's nowhere near that simple to
> write a scenario which will prove that "remove API races" works correctly.
>
> > [11]
> >
> https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535
> > [12]
> >
> https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516
> >
> > Regards,
> > Ivan Kolodyazhny,
> > http://blog.e0ne.info/
> >
> > On Wed, Mar 2, 2016 at 4:50 PM, Eric Harney  wrote:
> >
> >> On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:
> >>> Eric,
> >>>
> >>> There are Gorka's patches [10] to remove API Races
> >>>
> >>>
> >>> [10]
> >>>
> >>
> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> >>>
> >>> Regards,
> >>> Ivan Kolodyazhny,
> >>> http://blog.e0ne.info/
> >>>
> >>
> >> So the second part of my question is, is writing a Rally job to prove
> >> out that code a reasonable task?
> >>
> >> How hard is that to do and what does it look like?
> >>
> >>> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney 
> wrote:
> >>>
>  On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
> > Hi Team,
> >
> > Here are my thoughts and proposals how to make Cinder testing process
> > better. I won't cover "3rd party CI's" topic here. I will share my
>  opinion
> > about current and feature jobs.
> >
> >
> > Unit-tests
> >
> >- Long-running tests. I hope, everybody will agree that unit-tests
>  must
> >be quite simple and very fast. Unit tests which takes more than
> 3-5
>  seconds
> >should be refactored and/or moved to 'integration' tests.
> >Thanks to Tom Barron for several fixes like [1]. IMO, we it would
> be
> >good to have some hacking checks to prevent such issues in a
> future.
> >
> >- Tests coverage. We don't check it in an automatic way on gates.
> >Usually, we require to add some unit-tests during code review
>  process. Why
> >can't we add coverage job to our CI and do not merge new patches,
> >> with
> >will decrease tests coverage rate? Maybe, such job could be voting
> >> in
>  a
> >future to not ignore it. For now, there is not simple way to check
>  coverage
> >because 'tox -e cover' output is not useful [2].
> >
> >
> > Functional tests for Cinder
> >
> > We introduced some functional tests last month [3]. Here is a patch
> to
> > infra to add new job [4]. Because these tests were moved from
>  unit-tests, I
> > think we're OK to make this job voting. Such tests should not be a
> > replacement for Tempest. They even could tests Cinder with Fake
> Driver
> >> to
> > make it faster and not related on storage backends issues.
> >
> >
> > Tempest in-tree tests
> >
> > Sean started work on it [5] and I think it's a good idea to get them
> in
> > Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
> >> real
> > backend.
> >
> >
> > Functional tests for python-brick-cinderclient-ext
> >
> > There are patches that introduces functional tests [6] and new job
> [7].
> >
> >
> > Functional tests for python-cinderclient
> >
> > We've got a very limited set of such tests and non-voting job. IMO,
> we
>  can
> > run them even with Cinder Fake Driver to make them not depended on a
> > storage backend and make it faster. I believe, we can make this job
>  voting
> > soon. Also, we need more contributors to this kind of tests.
> >
> >
> > Integrated tests for python-cinderclient
> >
> > We need such tests to make sure that we won't break Nova, Heat or
> other
> > python-cinderclient consumers with a next merged patch. There is a
> >> thread
> > in openstack-dev ML about such tests [8] and proposal [9] to
> introduce
>  them
> > to python-cinderclient.
> >
> >
> > Rally tests
> >
> > IMO, it would be good to have new Rally scenarios for every patches
> >> like
> > 'improves performance', 'fixes concurrency issues', etc. Even if we
> as
> >> a
> > Cinder community don't have enough time to implement them, we have to
> >> ask
> > for them in reviews, openstack-dev ML, file Rally bugs and blueprints
> >> if
> > needed.
> >
> 
>  Are there any recent examples of a fix 

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Eric Harney]

On 03/02/2016 10:07 AM, Ivan Kolodyazhny wrote:
> Eric,
> 
> For now, we test Cinder API with some concurrency only with Rally, so, IMO,
> it's reasonable get more scenarios for API races fixes.
> 
> It's not a hard task to implement new scenarios, they are pretty simple:
> [11] and [12]
> 

Sure, these are simple, but I think it's nowhere near that simple to
write a scenario which will prove that "remove API races" works correctly.

> [11]
> https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535
> [12]
> https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 
> On Wed, Mar 2, 2016 at 4:50 PM, Eric Harney  wrote:
> 
>> On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:
>>> Eric,
>>>
>>> There are Gorka's patches [10] to remove API Races
>>>
>>>
>>> [10]
>>>
>> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
>>>
>>> Regards,
>>> Ivan Kolodyazhny,
>>> http://blog.e0ne.info/
>>>
>>
>> So the second part of my question is, is writing a Rally job to prove
>> out that code a reasonable task?
>>
>> How hard is that to do and what does it look like?
>>
>>> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney  wrote:
>>>
 On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
> Hi Team,
>
> Here are my thoughts and proposals how to make Cinder testing process
> better. I won't cover "3rd party CI's" topic here. I will share my
 opinion
> about current and feature jobs.
>
>
> Unit-tests
>
>- Long-running tests. I hope, everybody will agree that unit-tests
 must
>be quite simple and very fast. Unit tests which takes more than 3-5
 seconds
>should be refactored and/or moved to 'integration' tests.
>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>good to have some hacking checks to prevent such issues in a future.
>
>- Tests coverage. We don't check it in an automatic way on gates.
>Usually, we require to add some unit-tests during code review
 process. Why
>can't we add coverage job to our CI and do not merge new patches,
>> with
>will decrease tests coverage rate? Maybe, such job could be voting
>> in
 a
>future to not ignore it. For now, there is not simple way to check
 coverage
>because 'tox -e cover' output is not useful [2].
>
>
> Functional tests for Cinder
>
> We introduced some functional tests last month [3]. Here is a patch to
> infra to add new job [4]. Because these tests were moved from
 unit-tests, I
> think we're OK to make this job voting. Such tests should not be a
> replacement for Tempest. They even could tests Cinder with Fake Driver
>> to
> make it faster and not related on storage backends issues.
>
>
> Tempest in-tree tests
>
> Sean started work on it [5] and I think it's a good idea to get them in
> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
>> real
> backend.
>
>
> Functional tests for python-brick-cinderclient-ext
>
> There are patches that introduces functional tests [6] and new job [7].
>
>
> Functional tests for python-cinderclient
>
> We've got a very limited set of such tests and non-voting job. IMO, we
 can
> run them even with Cinder Fake Driver to make them not depended on a
> storage backend and make it faster. I believe, we can make this job
 voting
> soon. Also, we need more contributors to this kind of tests.
>
>
> Integrated tests for python-cinderclient
>
> We need such tests to make sure that we won't break Nova, Heat or other
> python-cinderclient consumers with a next merged patch. There is a
>> thread
> in openstack-dev ML about such tests [8] and proposal [9] to introduce
 them
> to python-cinderclient.
>
>
> Rally tests
>
> IMO, it would be good to have new Rally scenarios for every patches
>> like
> 'improves performance', 'fixes concurrency issues', etc. Even if we as
>> a
> Cinder community don't have enough time to implement them, we have to
>> ask
> for them in reviews, openstack-dev ML, file Rally bugs and blueprints
>> if
> needed.
>

 Are there any recent examples of a fix like this recently where it would
 seem like a reasonable task to write a Rally scenario along with the
>> patch?

 Not being very familiar with Rally (as I think most of us aren't), I'm
 having a hard time picturing this.

>
> [1] https://review.openstack.org/#/c/282861/
> [2] http://paste.openstack.org/show/488925/
> [3] https://review.openstack.org/#/c/267801/
> [4] https://review.openstack.org/#/c/287115/
> [5] 

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Gorka Eguileor]

On 02/03, Ivan Kolodyazhny wrote:
> Eric,
> 
> There are Gorka's patches [10] to remove API Races
> 
> 
> [10]
> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> 

I looked at Rally a long time ago so apologies if I'm totally off base
here, but it looked like it was a performance evaluation tool, which
means that it probably won't help to check for API Races (at least I
didn't see how when I looked).

Many of the API races only happen if you simultaneously try the same
operation multiple times against the same resource or if there are
different operations that are trying to operate on the same resource.

On the first case if Rally allowed it we could test it because we know
only 1 of the operations should succeed, but on the second case when we
are talking about preventing races from different operations there is no
way to know what the result should be, since the order in which those
operations are executed on each test run will determine which one will
fail and which one will succeed.

I'm not trying to go against the general idea of adding rally tests, I
just think that they won't help in the case of the API races.

> > > [ ... ]
> > >
> > >- Tests coverage. We don't check it in an automatic way on gates.
> > >Usually, we require to add some unit-tests during code review
> > process. Why
> > >can't we add coverage job to our CI and do not merge new patches, with
> > >will decrease tests coverage rate? Maybe, such job could be voting in
> > a
> > >future to not ignore it. For now, there is not simple way to check
> > coverage
> > >because 'tox -e cover' output is not useful [2].

In my experience preventing patches that reduce test coverage can be
sometimes problematic because it's a percentage, and many times code
refactoring gives false positives.  :-(


Cheers,
Gorka.


> > >
> > >
> > > Functional tests for Cinder
> > >
> > > We introduced some functional tests last month [3]. Here is a patch to
> > > infra to add new job [4]. Because these tests were moved from
> > unit-tests, I
> > > think we're OK to make this job voting. Such tests should not be a
> > > replacement for Tempest. They even could tests Cinder with Fake Driver to
> > > make it faster and not related on storage backends issues.
> > >
> > >
> > > Tempest in-tree tests
> > >
> > > Sean started work on it [5] and I think it's a good idea to get them in
> > > Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> > > backend.
> > >
> > >
> > > Functional tests for python-brick-cinderclient-ext
> > >
> > > There are patches that introduces functional tests [6] and new job [7].
> > >
> > >
> > > Functional tests for python-cinderclient
> > >
> > > We've got a very limited set of such tests and non-voting job. IMO, we
> > can
> > > run them even with Cinder Fake Driver to make them not depended on a
> > > storage backend and make it faster. I believe, we can make this job
> > voting
> > > soon. Also, we need more contributors to this kind of tests.
> > >
> > >
> > > Integrated tests for python-cinderclient
> > >
> > > We need such tests to make sure that we won't break Nova, Heat or other
> > > python-cinderclient consumers with a next merged patch. There is a thread
> > > in openstack-dev ML about such tests [8] and proposal [9] to introduce
> > them
> > > to python-cinderclient.
> > >
> > >
> > > Rally tests
> > >
> > > IMO, it would be good to have new Rally scenarios for every patches like
> > > 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> > > Cinder community don't have enough time to implement them, we have to ask
> > > for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
> > > needed.
> > >
> >
> > Are there any recent examples of a fix like this recently where it would
> > seem like a reasonable task to write a Rally scenario along with the patch?
> >
> > Not being very familiar with Rally (as I think most of us aren't), I'm
> > having a hard time picturing this.
> >
> > >
> > > [1] https://review.openstack.org/#/c/282861/
> > > [2] http://paste.openstack.org/show/488925/
> > > [3] https://review.openstack.org/#/c/267801/
> > > [4] https://review.openstack.org/#/c/287115/
> > > [5] https://review.openstack.org/#/c/274471/
> > > [6] https://review.openstack.org/#/c/265811/
> > > [7] https://review.openstack.org/#/c/265925/
> > > [8]
> > >
> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> > > [9] https://review.openstack.org/#/c/279432/
> > >
> > >
> > > Regards,
> > > Ivan Kolodyazhny,
> > > http://blog.e0ne.info/
> > >
> > >
> > >
> > >
> > __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> >
> > 

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Ivan Kolodyazhny]

Eric,

For now, we test Cinder API with some concurrency only with Rally, so, IMO,
it's reasonable get more scenarios for API races fixes.

It's not a hard task to implement new scenarios, they are pretty simple:
[11] and [12]

[11]
https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535
[12]
https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 4:50 PM, Eric Harney  wrote:

> On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:
> > Eric,
> >
> > There are Gorka's patches [10] to remove API Races
> >
> >
> > [10]
> >
> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> >
> > Regards,
> > Ivan Kolodyazhny,
> > http://blog.e0ne.info/
> >
>
> So the second part of my question is, is writing a Rally job to prove
> out that code a reasonable task?
>
> How hard is that to do and what does it look like?
>
> > On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney  wrote:
> >
> >> On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
> >>> Hi Team,
> >>>
> >>> Here are my thoughts and proposals how to make Cinder testing process
> >>> better. I won't cover "3rd party CI's" topic here. I will share my
> >> opinion
> >>> about current and feature jobs.
> >>>
> >>>
> >>> Unit-tests
> >>>
> >>>- Long-running tests. I hope, everybody will agree that unit-tests
> >> must
> >>>be quite simple and very fast. Unit tests which takes more than 3-5
> >> seconds
> >>>should be refactored and/or moved to 'integration' tests.
> >>>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
> >>>good to have some hacking checks to prevent such issues in a future.
> >>>
> >>>- Tests coverage. We don't check it in an automatic way on gates.
> >>>Usually, we require to add some unit-tests during code review
> >> process. Why
> >>>can't we add coverage job to our CI and do not merge new patches,
> with
> >>>will decrease tests coverage rate? Maybe, such job could be voting
> in
> >> a
> >>>future to not ignore it. For now, there is not simple way to check
> >> coverage
> >>>because 'tox -e cover' output is not useful [2].
> >>>
> >>>
> >>> Functional tests for Cinder
> >>>
> >>> We introduced some functional tests last month [3]. Here is a patch to
> >>> infra to add new job [4]. Because these tests were moved from
> >> unit-tests, I
> >>> think we're OK to make this job voting. Such tests should not be a
> >>> replacement for Tempest. They even could tests Cinder with Fake Driver
> to
> >>> make it faster and not related on storage backends issues.
> >>>
> >>>
> >>> Tempest in-tree tests
> >>>
> >>> Sean started work on it [5] and I think it's a good idea to get them in
> >>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
> real
> >>> backend.
> >>>
> >>>
> >>> Functional tests for python-brick-cinderclient-ext
> >>>
> >>> There are patches that introduces functional tests [6] and new job [7].
> >>>
> >>>
> >>> Functional tests for python-cinderclient
> >>>
> >>> We've got a very limited set of such tests and non-voting job. IMO, we
> >> can
> >>> run them even with Cinder Fake Driver to make them not depended on a
> >>> storage backend and make it faster. I believe, we can make this job
> >> voting
> >>> soon. Also, we need more contributors to this kind of tests.
> >>>
> >>>
> >>> Integrated tests for python-cinderclient
> >>>
> >>> We need such tests to make sure that we won't break Nova, Heat or other
> >>> python-cinderclient consumers with a next merged patch. There is a
> thread
> >>> in openstack-dev ML about such tests [8] and proposal [9] to introduce
> >> them
> >>> to python-cinderclient.
> >>>
> >>>
> >>> Rally tests
> >>>
> >>> IMO, it would be good to have new Rally scenarios for every patches
> like
> >>> 'improves performance', 'fixes concurrency issues', etc. Even if we as
> a
> >>> Cinder community don't have enough time to implement them, we have to
> ask
> >>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints
> if
> >>> needed.
> >>>
> >>
> >> Are there any recent examples of a fix like this recently where it would
> >> seem like a reasonable task to write a Rally scenario along with the
> patch?
> >>
> >> Not being very familiar with Rally (as I think most of us aren't), I'm
> >> having a hard time picturing this.
> >>
> >>>
> >>> [1] https://review.openstack.org/#/c/282861/
> >>> [2] http://paste.openstack.org/show/488925/
> >>> [3] https://review.openstack.org/#/c/267801/
> >>> [4] https://review.openstack.org/#/c/287115/
> >>> [5] https://review.openstack.org/#/c/274471/
> >>> [6] https://review.openstack.org/#/c/265811/
> >>> [7] https://review.openstack.org/#/c/265925/
> >>> [8]
> >>>
> >>
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> >>> [9] 

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Ivan Kolodyazhny]

Arkady,

It's not true. We've got non-voting Rally job on Cinder gates called
"gate-rally-dsvm-cinder".

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 4:52 PM, <arkady_kanev...@dell.com> wrote:

> Rally is not part of the gate.
>
> Also making performance test without 3rd party CI will not be very useful.
>
> It is a good idea to run Rally performance and scenario testing but
> outside gate process.
>
>
>
> *From:* Ivan Kolodyazhny [mailto:e...@e0ne.info]
> *Sent:* Wednesday, March 02, 2016 8:36 AM
> *To:* OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>
> *Subject:* Re: [openstack-dev] [cinder] Proposal: changes to our current
> testing process
>
>
>
> Eric,
>
>
>
> There are Gorka's patches [10] to remove API Races
>
>
>
>
>
> [10]
> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
>
>
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
>
>
>
> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney <ehar...@redhat.com> wrote:
>
> On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
> > Hi Team,
> >
> > Here are my thoughts and proposals how to make Cinder testing process
> > better. I won't cover "3rd party CI's" topic here. I will share my
> opinion
> > about current and feature jobs.
> >
> >
> > Unit-tests
> >
> >- Long-running tests. I hope, everybody will agree that unit-tests
> must
> >be quite simple and very fast. Unit tests which takes more than 3-5
> seconds
> >should be refactored and/or moved to 'integration' tests.
> >Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
> >good to have some hacking checks to prevent such issues in a future.
> >
> >- Tests coverage. We don't check it in an automatic way on gates.
>
> >Usually, we require to add some unit-tests during code review
> process. Why
> >can't we add coverage job to our CI and do not merge new patches, with
> >will decrease tests coverage rate? Maybe, such job could be voting in
> a
> >future to not ignore it. For now, there is not simple way to check
> coverage
> >because 'tox -e cover' output is not useful [2].
> >
> >
> > Functional tests for Cinder
> >
> > We introduced some functional tests last month [3]. Here is a patch to
> > infra to add new job [4]. Because these tests were moved from
> unit-tests, I
> > think we're OK to make this job voting. Such tests should not be a
> > replacement for Tempest. They even could tests Cinder with Fake Driver to
> > make it faster and not related on storage backends issues.
> >
> >
> > Tempest in-tree tests
> >
> > Sean started work on it [5] and I think it's a good idea to get them in
> > Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> > backend.
> >
> >
> > Functional tests for python-brick-cinderclient-ext
> >
> > There are patches that introduces functional tests [6] and new job [7].
> >
> >
> > Functional tests for python-cinderclient
> >
> > We've got a very limited set of such tests and non-voting job. IMO, we
> can
> > run them even with Cinder Fake Driver to make them not depended on a
> > storage backend and make it faster. I believe, we can make this job
> voting
> > soon. Also, we need more contributors to this kind of tests.
> >
> >
> > Integrated tests for python-cinderclient
> >
> > We need such tests to make sure that we won't break Nova, Heat or other
> > python-cinderclient consumers with a next merged patch. There is a thread
> > in openstack-dev ML about such tests [8] and proposal [9] to introduce
> them
> > to python-cinderclient.
> >
> >
> > Rally tests
> >
> > IMO, it would be good to have new Rally scenarios for every patches like
> > 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> > Cinder community don't have enough time to implement them, we have to ask
> > for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
> > needed.
> >
>
> Are there any recent examples of a fix like this recently where it would
> seem like a reasonable task to write a Rally scenario along with the patch?
>
> Not being very familiar with Rally (as I think most of us aren't), I'm
> having a hard time picturing this.
>
> >
> > [1] https://review.openstack.org/#/c/282861/
> > [2] http://paste.openstack.org

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Arkady_Kanevsky]

Rally is not part of the gate.
Also making performance test without 3rd party CI will not be very useful.
It is a good idea to run Rally performance and scenario testing but outside 
gate process.

From: Ivan Kolodyazhny [mailto:e...@e0ne.info]
Sent: Wednesday, March 02, 2016 8:36 AM
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [cinder] Proposal: changes to our current testing 
process

Eric,

There are Gorka's patches [10] to remove API Races


[10] 
https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney 
<ehar...@redhat.com<mailto:ehar...@redhat.com>> wrote:
On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
> Hi Team,
>
> Here are my thoughts and proposals how to make Cinder testing process
> better. I won't cover "3rd party CI's" topic here. I will share my opinion
> about current and feature jobs.
>
>
> Unit-tests
>
>- Long-running tests. I hope, everybody will agree that unit-tests must
>be quite simple and very fast. Unit tests which takes more than 3-5 seconds
>should be refactored and/or moved to 'integration' tests.
>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>good to have some hacking checks to prevent such issues in a future.
>
>- Tests coverage. We don't check it in an automatic way on gates.
>Usually, we require to add some unit-tests during code review process. Why
>can't we add coverage job to our CI and do not merge new patches, with
>will decrease tests coverage rate? Maybe, such job could be voting in a
>future to not ignore it. For now, there is not simple way to check coverage
>because 'tox -e cover' output is not useful [2].
>
>
> Functional tests for Cinder
>
> We introduced some functional tests last month [3]. Here is a patch to
> infra to add new job [4]. Because these tests were moved from unit-tests, I
> think we're OK to make this job voting. Such tests should not be a
> replacement for Tempest. They even could tests Cinder with Fake Driver to
> make it faster and not related on storage backends issues.
>
>
> Tempest in-tree tests
>
> Sean started work on it [5] and I think it's a good idea to get them in
> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> backend.
>
>
> Functional tests for python-brick-cinderclient-ext
>
> There are patches that introduces functional tests [6] and new job [7].
>
>
> Functional tests for python-cinderclient
>
> We've got a very limited set of such tests and non-voting job. IMO, we can
> run them even with Cinder Fake Driver to make them not depended on a
> storage backend and make it faster. I believe, we can make this job voting
> soon. Also, we need more contributors to this kind of tests.
>
>
> Integrated tests for python-cinderclient
>
> We need such tests to make sure that we won't break Nova, Heat or other
> python-cinderclient consumers with a next merged patch. There is a thread
> in openstack-dev ML about such tests [8] and proposal [9] to introduce them
> to python-cinderclient.
>
>
> Rally tests
>
> IMO, it would be good to have new Rally scenarios for every patches like
> 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> Cinder community don't have enough time to implement them, we have to ask
> for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
> needed.
>
Are there any recent examples of a fix like this recently where it would
seem like a reasonable task to write a Rally scenario along with the patch?

Not being very familiar with Rally (as I think most of us aren't), I'm
having a hard time picturing this.

>
> [1] https://review.openstack.org/#/c/282861/
> [2] http://paste.openstack.org/show/488925/
> [3] https://review.openstack.org/#/c/267801/
> [4] https://review.openstack.org/#/c/287115/
> [5] https://review.openstack.org/#/c/274471/
> [6] https://review.openstack.org/#/c/265811/
> [7] https://review.openstack.org/#/c/265925/
> [8]
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> [9] https://review.openstack.org/#/c/279432/
>
>
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/lis

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Eric Harney]

On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:
> Eric,
> 
> There are Gorka's patches [10] to remove API Races
> 
> 
> [10]
> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 

So the second part of my question is, is writing a Rally job to prove
out that code a reasonable task?

How hard is that to do and what does it look like?

> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney  wrote:
> 
>> On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
>>> Hi Team,
>>>
>>> Here are my thoughts and proposals how to make Cinder testing process
>>> better. I won't cover "3rd party CI's" topic here. I will share my
>> opinion
>>> about current and feature jobs.
>>>
>>>
>>> Unit-tests
>>>
>>>- Long-running tests. I hope, everybody will agree that unit-tests
>> must
>>>be quite simple and very fast. Unit tests which takes more than 3-5
>> seconds
>>>should be refactored and/or moved to 'integration' tests.
>>>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>>>good to have some hacking checks to prevent such issues in a future.
>>>
>>>- Tests coverage. We don't check it in an automatic way on gates.
>>>Usually, we require to add some unit-tests during code review
>> process. Why
>>>can't we add coverage job to our CI and do not merge new patches, with
>>>will decrease tests coverage rate? Maybe, such job could be voting in
>> a
>>>future to not ignore it. For now, there is not simple way to check
>> coverage
>>>because 'tox -e cover' output is not useful [2].
>>>
>>>
>>> Functional tests for Cinder
>>>
>>> We introduced some functional tests last month [3]. Here is a patch to
>>> infra to add new job [4]. Because these tests were moved from
>> unit-tests, I
>>> think we're OK to make this job voting. Such tests should not be a
>>> replacement for Tempest. They even could tests Cinder with Fake Driver to
>>> make it faster and not related on storage backends issues.
>>>
>>>
>>> Tempest in-tree tests
>>>
>>> Sean started work on it [5] and I think it's a good idea to get them in
>>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
>>> backend.
>>>
>>>
>>> Functional tests for python-brick-cinderclient-ext
>>>
>>> There are patches that introduces functional tests [6] and new job [7].
>>>
>>>
>>> Functional tests for python-cinderclient
>>>
>>> We've got a very limited set of such tests and non-voting job. IMO, we
>> can
>>> run them even with Cinder Fake Driver to make them not depended on a
>>> storage backend and make it faster. I believe, we can make this job
>> voting
>>> soon. Also, we need more contributors to this kind of tests.
>>>
>>>
>>> Integrated tests for python-cinderclient
>>>
>>> We need such tests to make sure that we won't break Nova, Heat or other
>>> python-cinderclient consumers with a next merged patch. There is a thread
>>> in openstack-dev ML about such tests [8] and proposal [9] to introduce
>> them
>>> to python-cinderclient.
>>>
>>>
>>> Rally tests
>>>
>>> IMO, it would be good to have new Rally scenarios for every patches like
>>> 'improves performance', 'fixes concurrency issues', etc. Even if we as a
>>> Cinder community don't have enough time to implement them, we have to ask
>>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
>>> needed.
>>>
>>
>> Are there any recent examples of a fix like this recently where it would
>> seem like a reasonable task to write a Rally scenario along with the patch?
>>
>> Not being very familiar with Rally (as I think most of us aren't), I'm
>> having a hard time picturing this.
>>
>>>
>>> [1] https://review.openstack.org/#/c/282861/
>>> [2] http://paste.openstack.org/show/488925/
>>> [3] https://review.openstack.org/#/c/267801/
>>> [4] https://review.openstack.org/#/c/287115/
>>> [5] https://review.openstack.org/#/c/274471/
>>> [6] https://review.openstack.org/#/c/265811/
>>> [7] https://review.openstack.org/#/c/265925/
>>> [8]
>>>
>> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
>>> [9] https://review.openstack.org/#/c/279432/
>>>
>>>
>>> Regards,
>>> Ivan Kolodyazhny,
>>> http://blog.e0ne.info/
>>>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Ivan Kolodyazhny]

Eric,

There are Gorka's patches [10] to remove API Races


[10]
https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney  wrote:

> On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
> > Hi Team,
> >
> > Here are my thoughts and proposals how to make Cinder testing process
> > better. I won't cover "3rd party CI's" topic here. I will share my
> opinion
> > about current and feature jobs.
> >
> >
> > Unit-tests
> >
> >- Long-running tests. I hope, everybody will agree that unit-tests
> must
> >be quite simple and very fast. Unit tests which takes more than 3-5
> seconds
> >should be refactored and/or moved to 'integration' tests.
> >Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
> >good to have some hacking checks to prevent such issues in a future.
> >
> >- Tests coverage. We don't check it in an automatic way on gates.
> >Usually, we require to add some unit-tests during code review
> process. Why
> >can't we add coverage job to our CI and do not merge new patches, with
> >will decrease tests coverage rate? Maybe, such job could be voting in
> a
> >future to not ignore it. For now, there is not simple way to check
> coverage
> >because 'tox -e cover' output is not useful [2].
> >
> >
> > Functional tests for Cinder
> >
> > We introduced some functional tests last month [3]. Here is a patch to
> > infra to add new job [4]. Because these tests were moved from
> unit-tests, I
> > think we're OK to make this job voting. Such tests should not be a
> > replacement for Tempest. They even could tests Cinder with Fake Driver to
> > make it faster and not related on storage backends issues.
> >
> >
> > Tempest in-tree tests
> >
> > Sean started work on it [5] and I think it's a good idea to get them in
> > Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> > backend.
> >
> >
> > Functional tests for python-brick-cinderclient-ext
> >
> > There are patches that introduces functional tests [6] and new job [7].
> >
> >
> > Functional tests for python-cinderclient
> >
> > We've got a very limited set of such tests and non-voting job. IMO, we
> can
> > run them even with Cinder Fake Driver to make them not depended on a
> > storage backend and make it faster. I believe, we can make this job
> voting
> > soon. Also, we need more contributors to this kind of tests.
> >
> >
> > Integrated tests for python-cinderclient
> >
> > We need such tests to make sure that we won't break Nova, Heat or other
> > python-cinderclient consumers with a next merged patch. There is a thread
> > in openstack-dev ML about such tests [8] and proposal [9] to introduce
> them
> > to python-cinderclient.
> >
> >
> > Rally tests
> >
> > IMO, it would be good to have new Rally scenarios for every patches like
> > 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> > Cinder community don't have enough time to implement them, we have to ask
> > for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
> > needed.
> >
>
> Are there any recent examples of a fix like this recently where it would
> seem like a reasonable task to write a Rally scenario along with the patch?
>
> Not being very familiar with Rally (as I think most of us aren't), I'm
> having a hard time picturing this.
>
> >
> > [1] https://review.openstack.org/#/c/282861/
> > [2] http://paste.openstack.org/show/488925/
> > [3] https://review.openstack.org/#/c/267801/
> > [4] https://review.openstack.org/#/c/287115/
> > [5] https://review.openstack.org/#/c/274471/
> > [6] https://review.openstack.org/#/c/265811/
> > [7] https://review.openstack.org/#/c/265925/
> > [8]
> >
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> > [9] https://review.openstack.org/#/c/279432/
> >
> >
> > Regards,
> > Ivan Kolodyazhny,
> > http://blog.e0ne.info/
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Eric Harney]

On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
> Hi Team,
> 
> Here are my thoughts and proposals how to make Cinder testing process
> better. I won't cover "3rd party CI's" topic here. I will share my opinion
> about current and feature jobs.
> 
> 
> Unit-tests
> 
>- Long-running tests. I hope, everybody will agree that unit-tests must
>be quite simple and very fast. Unit tests which takes more than 3-5 seconds
>should be refactored and/or moved to 'integration' tests.
>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>good to have some hacking checks to prevent such issues in a future.
> 
>- Tests coverage. We don't check it in an automatic way on gates.
>Usually, we require to add some unit-tests during code review process. Why
>can't we add coverage job to our CI and do not merge new patches, with
>will decrease tests coverage rate? Maybe, such job could be voting in a
>future to not ignore it. For now, there is not simple way to check coverage
>because 'tox -e cover' output is not useful [2].
> 
> 
> Functional tests for Cinder
> 
> We introduced some functional tests last month [3]. Here is a patch to
> infra to add new job [4]. Because these tests were moved from unit-tests, I
> think we're OK to make this job voting. Such tests should not be a
> replacement for Tempest. They even could tests Cinder with Fake Driver to
> make it faster and not related on storage backends issues.
> 
> 
> Tempest in-tree tests
> 
> Sean started work on it [5] and I think it's a good idea to get them in
> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> backend.
> 
> 
> Functional tests for python-brick-cinderclient-ext
> 
> There are patches that introduces functional tests [6] and new job [7].
> 
> 
> Functional tests for python-cinderclient
> 
> We've got a very limited set of such tests and non-voting job. IMO, we can
> run them even with Cinder Fake Driver to make them not depended on a
> storage backend and make it faster. I believe, we can make this job voting
> soon. Also, we need more contributors to this kind of tests.
> 
> 
> Integrated tests for python-cinderclient
> 
> We need such tests to make sure that we won't break Nova, Heat or other
> python-cinderclient consumers with a next merged patch. There is a thread
> in openstack-dev ML about such tests [8] and proposal [9] to introduce them
> to python-cinderclient.
> 
> 
> Rally tests
> 
> IMO, it would be good to have new Rally scenarios for every patches like
> 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> Cinder community don't have enough time to implement them, we have to ask
> for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
> needed.
> 

Are there any recent examples of a fix like this recently where it would
seem like a reasonable task to write a Rally scenario along with the patch?

Not being very familiar with Rally (as I think most of us aren't), I'm
having a hard time picturing this.

> 
> [1] https://review.openstack.org/#/c/282861/
> [2] http://paste.openstack.org/show/488925/
> [3] https://review.openstack.org/#/c/267801/
> [4] https://review.openstack.org/#/c/287115/
> [5] https://review.openstack.org/#/c/274471/
> [6] https://review.openstack.org/#/c/265811/
> [7] https://review.openstack.org/#/c/265925/
> [8]
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> [9] https://review.openstack.org/#/c/279432/
> 
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Sean Dague]

That would be great. The boot from volume test remains one of the top
failure scenarios, both in the gate, and for end users. It would be
really good to get a handle on why and get that shored up.

-Sean

On 03/02/2016 08:18 AM, D'Angelo, Scott wrote:
> +1 to making the testing process better.
> It has been discussed that services could/should consider devoting some or 
> all of a release cycle to stability and/or quality.
> I propose the Cinder team makes improving and fixing the tests and test 
> process a priority for the Newton cycle.
> 
> Scott D'Angelo (scottda)
> 
> From: Ivan Kolodyazhny [e...@e0ne.info]
> Sent: Wednesday, March 02, 2016 4:25 AM
> To: OpenStack Development Mailing List
> Subject: [openstack-dev] [cinder] Proposal: changes to our current testing 
> process
> 
> Hi Team,
> 
> Here are my thoughts and proposals how to make Cinder testing process better. 
> I won't cover "3rd party CI's" topic here. I will share my opinion about 
> current and feature jobs.
> 
> 
> Unit-tests
> 
>   *   Long-running tests. I hope, everybody will agree that unit-tests must 
> be quite simple and very fast. Unit tests which takes more than 3-5 seconds 
> should be refactored and/or moved to 'integration' tests.
> Thanks to Tom Barron for several fixes like [1]. IMO, we it would be good to 
> have some hacking checks to prevent such issues in a future.
> 
>   *   Tests coverage. We don't check it in an automatic way on gates. 
> Usually, we require to add some unit-tests during code review process. Why 
> can't we add coverage job to our CI and do not merge new patches, with will 
> decrease tests coverage rate? Maybe, such job could be voting in a future to 
> not ignore it. For now, there is not simple way to check coverage because 
> 'tox -e cover' output is not useful [2].
> 
> Functional tests for Cinder
> 
> We introduced some functional tests last month [3]. Here is a patch to infra 
> to add new job [4]. Because these tests were moved from unit-tests, I think 
> we're OK to make this job voting. Such tests should not be a replacement for 
> Tempest. They even could tests Cinder with Fake Driver to make it faster and 
> not related on storage backends issues.
> 
> 
> Tempest in-tree tests
> 
> Sean started work on it [5] and I think it's a good idea to get them in 
> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real 
> backend.
> 
> 
> Functional tests for python-brick-cinderclient-ext
> 
> There are patches that introduces functional tests [6] and new job [7].
> 
> 
> Functional tests for python-cinderclient
> 
> We've got a very limited set of such tests and non-voting job. IMO, we can 
> run them even with Cinder Fake Driver to make them not depended on a storage 
> backend and make it faster. I believe, we can make this job voting soon. 
> Also, we need more contributors to this kind of tests.
> 
> 
> Integrated tests for python-cinderclient
> 
> We need such tests to make sure that we won't break Nova, Heat or other 
> python-cinderclient consumers with a next merged patch. There is a thread in 
> openstack-dev ML about such tests [8] and proposal [9] to introduce them to 
> python-cinderclient.
> 
> 
> Rally tests
> 
> IMO, it would be good to have new Rally scenarios for every patches like 
> 'improves performance', 'fixes concurrency issues', etc. Even if we as a 
> Cinder community don't have enough time to implement them, we have to ask for 
> them in reviews, openstack-dev ML, file Rally bugs and blueprints if needed.
> 
> 
> [1] https://review.openstack.org/#/c/282861/
> [2] http://paste.openstack.org/show/488925/
> [3] https://review.openstack.org/#/c/267801/
> [4] https://review.openstack.org/#/c/287115/
> [5] https://review.openstack.org/#/c/274471/
> [6] https://review.openstack.org/#/c/265811/
> [7] https://review.openstack.org/#/c/265925/
> [8] http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> [9] https://review.openstack.org/#/c/279432/
> 
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[D'Angelo, Scott]

+1 to making the testing process better.
It has been discussed that services could/should consider devoting some or all 
of a release cycle to stability and/or quality.
I propose the Cinder team makes improving and fixing the tests and test process 
a priority for the Newton cycle.

Scott D'Angelo (scottda)

From: Ivan Kolodyazhny [e...@e0ne.info]
Sent: Wednesday, March 02, 2016 4:25 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [cinder] Proposal: changes to our current testing 
process

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process better. I 
won't cover "3rd party CI's" topic here. I will share my opinion about current 
and feature jobs.


Unit-tests

  *   Long-running tests. I hope, everybody will agree that unit-tests must be 
quite simple and very fast. Unit tests which takes more than 3-5 seconds should 
be refactored and/or moved to 'integration' tests.
Thanks to Tom Barron for several fixes like [1]. IMO, we it would be good to 
have some hacking checks to prevent such issues in a future.

  *   Tests coverage. We don't check it in an automatic way on gates. Usually, 
we require to add some unit-tests during code review process. Why can't we add 
coverage job to our CI and do not merge new patches, with will decrease tests 
coverage rate? Maybe, such job could be voting in a future to not ignore it. 
For now, there is not simple way to check coverage because 'tox -e cover' 
output is not useful [2].

Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to infra to 
add new job [4]. Because these tests were moved from unit-tests, I think we're 
OK to make this job voting. Such tests should not be a replacement for Tempest. 
They even could tests Cinder with Fake Driver to make it faster and not related 
on storage backends issues.


Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in Cinder 
repo to run them on Tempest jobs and 3-rd party CIs against a real backend.


Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].


Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we can run 
them even with Cinder Fake Driver to make them not depended on a storage 
backend and make it faster. I believe, we can make this job voting soon. Also, 
we need more contributors to this kind of tests.


Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other 
python-cinderclient consumers with a next merged patch. There is a thread in 
openstack-dev ML about such tests [8] and proposal [9] to introduce them to 
python-cinderclient.


Rally tests

IMO, it would be good to have new Rally scenarios for every patches like 
'improves performance', 'fixes concurrency issues', etc. Even if we as a Cinder 
community don't have enough time to implement them, we have to ask for them in 
reviews, openstack-dev ML, file Rally bugs and blueprints if needed.


[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8] http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
[9] https://review.openstack.org/#/c/279432/


Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder] Proposal: changes to our current testing process

[Andrey Kurilin]

Hi Ivan!

On Wed, Mar 2, 2016 at 1:25 PM, Ivan Kolodyazhny  wrote:

> Hi Team,
>
> Here are my thoughts and proposals how to make Cinder testing process
> better. I won't cover "3rd party CI's" topic here. I will share my opinion
> about current and feature jobs.
>
>
> Unit-tests
>
>- Long-running tests. I hope, everybody will agree that unit-tests
>must be quite simple and very fast. Unit tests which takes more than 3-5
>seconds should be refactored and/or moved to 'integration' tests.
>Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>good to have some hacking checks to prevent such issues in a future.
>
>- Tests coverage. We don't check it in an automatic way on gates.
>Usually, we require to add some unit-tests during code review process. Why
>can't we add coverage job to our CI and do not merge new patches, with
>will decrease tests coverage rate? Maybe, such job could be voting in a
>future to not ignore it. For now, there is not simple way to check coverage
>because 'tox -e cover' output is not useful [2].
>
> There is a script in Rally which can help you to check coverage rate -
https://github.com/openstack/rally/blob/master/tests/ci/cover.sh

Functional tests for Cinder
>
> We introduced some functional tests last month [3]. Here is a patch to
> infra to add new job [4]. Because these tests were moved from unit-tests, I
> think we're OK to make this job voting. Such tests should not be a
> replacement for Tempest. They even could tests Cinder with Fake Driver to
> make it faster and not related on storage backends issues.
>
>
> Tempest in-tree tests
>
> Sean started work on it [5] and I think it's a good idea to get them in
> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> backend.
>
>
> Functional tests for python-brick-cinderclient-ext
>
> There are patches that introduces functional tests [6] and new job [7].
>
>
> Functional tests for python-cinderclient
>
> We've got a very limited set of such tests and non-voting job. IMO, we can
> run them even with Cinder Fake Driver to make them not depended on a
> storage backend and make it faster. I believe, we can make this job voting
> soon. Also, we need more contributors to this kind of tests.
>
>
> Integrated tests for python-cinderclient
>
> We need such tests to make sure that we won't break Nova, Heat or other
> python-cinderclient consumers with a next merged patch. There is a thread
> in openstack-dev ML about such tests [8] and proposal [9] to introduce them
> to python-cinderclient.
>
>
> Rally tests
>
> IMO, it would be good to have new Rally scenarios for every patches like
> 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> Cinder community don't have enough time to implement them, we have to ask
> for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
> needed.
>
>
> [1] https://review.openstack.org/#/c/282861/
> [2] http://paste.openstack.org/show/488925/
> [3] https://review.openstack.org/#/c/267801/
> [4] https://review.openstack.org/#/c/287115/
> [5] https://review.openstack.org/#/c/274471/
> [6] https://review.openstack.org/#/c/265811/
> [7] https://review.openstack.org/#/c/265925/
> [8]
> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> [9] https://review.openstack.org/#/c/279432/
>
>
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Best regards,
Andrey Kurilin.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder] Proposal: changes to our current testing process

[Ivan Kolodyazhny]

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my opinion
about current and feature jobs.


Unit-tests

   - Long-running tests. I hope, everybody will agree that unit-tests must
   be quite simple and very fast. Unit tests which takes more than 3-5 seconds
   should be refactored and/or moved to 'integration' tests.
   Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
   good to have some hacking checks to prevent such issues in a future.

   - Tests coverage. We don't check it in an automatic way on gates.
   Usually, we require to add some unit-tests during code review process. Why
   can't we add coverage job to our CI and do not merge new patches, with
   will decrease tests coverage rate? Maybe, such job could be voting in a
   future to not ignore it. For now, there is not simple way to check coverage
   because 'tox -e cover' output is not useful [2].


Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.


Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.


Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].


Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job voting
soon. Also, we need more contributors to this kind of tests.


Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce them
to python-cinderclient.


Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.


[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8]
http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
[9] https://review.openstack.org/#/c/279432/


Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev