Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[John Garbutt]

On 21 December 2013 13:24, Matt Riedemann  wrote:
>
>
> On 12/19/2013 8:51 AM, John Garbutt wrote:
>>
>> On 4 December 2013 17:10, Russell Bryant  wrote:
>>>
>>> I think option 3 makes the most sense here (pending anyone saying we
>>> should run away screaming from mox3 for some reason).  It's actually
>>> what I had been assuming since this thread a while back.
>>>
>>> This means that we don't need to *require* that tests get converted if
>>> you're changing one.  It just gets you bonus imaginary internet points.
>>>
>>> Requiring mock for new tests seems fine.  We can grant exceptions in
>>> specific cases if necessary.  In general, we should be using mock for
>>> new tests.
>>
>>
>> I have lost track a bit here.
>>
>> The above seems like a sane approach. Do we all agree on that now?
>>
>> Can we add the above text into here:
>> https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist
>>
>> John
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> Yeah, at some point I wanted to cleanup the various testing guides but until
> then I like the idea of just putting something simple into the nova review
> checklist. Basically use mock for new tests, mox can be used in exceptional
> cases. What I've considered exceptional so far includes changes that will be
> backported to a stable release where mock isn't being used and cases where
> you basically have to bend over backwards to work new mock tests into an
> existing test class that has lots of existing setUp with mox. However, even
> in the latter case you can usually use mock after resetting the mox setup
> via self.mox.ResetAll() in the new test case(s).

The wiki looks good to me, thank you!

John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Ben Nemec]

 

I added it to the page John linked earlier:
https://wiki.openstack.org/wiki/ReviewChecklist 

-Ben 

On 2013-12-23 17:08, Shawn Hartsock wrote: 

> Where in the wiki is this written down? Maybe I should read some of these 
> entries. I have looked but I can't find it. 
> On Dec 23, 2013 11:56 AM, "Ben Nemec"  wrote:
> On 2013-12-21 07:24, Matt Riedemann wrote:
> On 12/19/2013 8:51 AM, John Garbutt wrote:
> On 4 December 2013 17:10, Russell Bryant  wrote:
> I think option 3 makes the most sense here (pending anyone saying we
> should run away screaming from mox3 for some reason). It's actually
> what I had been assuming since this thread a while back.
> 
> This means that we don't need to *require* that tests get converted if
> you're changing one. It just gets you bonus imaginary internet points.
> 
> Requiring mock for new tests seems fine. We can grant exceptions in
> specific cases if necessary. In general, we should be using mock for
> new tests. 
> I have lost track a bit here.
> 
> The above seems like a sane approach. Do we all agree on that now?
> 
> Can we add the above text into here:
> https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist [1]
> 
> John
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev [2]

 Yeah, at some point I wanted to cleanup the various testing guides but
 until then I like the idea of just putting something simple into the
 nova review checklist. Basically use mock for new tests, mox can be
 used in exceptional cases. What I've considered exceptional so far
 includes changes that will be backported to a stable release where
 mock isn't being used and cases where you basically have to bend over
 backwards to work new mock tests into an existing test class that has
 lots of existing setUp with mox. However, even in the latter case you
 can usually use mock after resetting the mox setup via
 self.mox.ResetAll() in the new test case(s). 
 I went ahead and added this to the wiki, so it's now an absolutely
inviolate policy. Unless, ya know, someone edits the wiki after me. ;-)

 Also, I put it in the common section because this doesn't seem like
something we should be doing differently per-project. If anyone objects,
feel free to add to the discussion here as to why.

 Thanks.

 -Ben

 ___
 OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev [2] 

 

Links:
--
[1]
https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist
[2] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Shawn Hartsock]

Where in the wiki is this written down? Maybe I should read some of these
entries. I have looked but I can't find it.
On Dec 23, 2013 11:56 AM, "Ben Nemec"  wrote:

> On 2013-12-21 07:24, Matt Riedemann wrote:
>
>> On 12/19/2013 8:51 AM, John Garbutt wrote:
>>
>>> On 4 December 2013 17:10, Russell Bryant  wrote:
>>>
 I think option 3 makes the most sense here (pending anyone saying we
 should run away screaming from mox3 for some reason).  It's actually
 what I had been assuming since this thread a while back.

 This means that we don't need to *require* that tests get converted if
 you're changing one.  It just gets you bonus imaginary internet points.

 Requiring mock for new tests seems fine.  We can grant exceptions in
 specific cases if necessary.  In general, we should be using mock for
 new tests.

>>>
>>> I have lost track a bit here.
>>>
>>> The above seems like a sane approach. Do we all agree on that now?
>>>
>>> Can we add the above text into here:
>>> https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist
>>>
>>> John
>>>
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>> Yeah, at some point I wanted to cleanup the various testing guides but
>> until then I like the idea of just putting something simple into the
>> nova review checklist. Basically use mock for new tests, mox can be
>> used in exceptional cases. What I've considered exceptional so far
>> includes changes that will be backported to a stable release where
>> mock isn't being used and cases where you basically have to bend over
>> backwards to work new mock tests into an existing test class that has
>> lots of existing setUp with mox. However, even in the latter case you
>> can usually use mock after resetting the mox setup via
>> self.mox.ResetAll() in the new test case(s).
>>
>
> I went ahead and added this to the wiki, so it's now an absolutely
> inviolate policy.  Unless, ya know, someone edits the wiki after me. ;-)
>
> Also, I put it in the common section because this doesn't seem like
> something we should be doing differently per-project.  If anyone objects,
> feel free to add to the discussion here as to why.
>
> Thanks.
>
> -Ben
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[John Griffith]

On Thu, Dec 5, 2013 at 8:38 PM, Jay Pipes  wrote:
> On 12/04/2013 12:10 PM, Russell Bryant wrote:
>>
>> On 12/04/2013 11:16 AM, Nikola Đipanov wrote:
>>>
>>> Resurrecting this thread because of an interesting review that came up
>>> yesterday [1].
>>>
>>> It seems that our lack of a firm decision on what to do with the mocking
>>> framework has left people confused. In hope to help - I'll give my view
>>> of where things are now and what we should do going forward, and
>>> hopefully we'll reach some consensus on this.
>>>
>>> Here's the breakdown:
>>>
>>> We should abandon mox:
>>> * It has not had a release in over 3 years [2] and a patch upstream for 2
>>> * There are bugs that are impacting the project with it (see above)
>>> * It will not be ported to python 3
>>>
>>> Proposed path forward options:
>>> 1) Port nova to mock now:
>>>* Literally unmanageable - huge review overhead and regression risk
>>> for not so much gain (maybe) [1]
>>>
>>> 2) Opportunistically port nova (write new tests using mock, when fixing
>>> tests, move them to mock):
>>>   * Will take a really long time to move to mock, and is not really a
>>> solution since we are stuck with mock for an undetermined period of time
>>> - it's what we are doing now (kind of).
>>>
>>> 3) Same as 2) but move current codebase to mox3
>>>   * Buys us py3k compat, and fresher code
>>>   * Mox3 and mox have diverged and we would need to backport mox fixes
>>> onto the mox3 three and become de-facto active maintainers (as per Peter
>>> Feiner's last email - that may not be so easy).
>>>
>>> I think we should follow path 3) if we can, but we need to:
>>>
>>> 1) Figure out what is the deal with mox3 and decide if owning it will
>>> really be less trouble than porting nova. To be hones - I was unable to
>>> even find the code repo for it, only [3]. If anyone has more info -
>>> please weigh in. We'll also need volunteers
>>>
>>> 2) Make better testing guidelines when using mock, and maybe add some
>>> testing helpers (like we do already have for mox) that will make porting
>>> existing tests easier. mreidem already put this on this weeks nova
>>> meeting agenda - so that might be a good place to discuss all the issues
>>> mentioned here as well.
>>>
>>> We should really take a stronger stance on this soon IMHO, as this comes
>>> up with literally every commit.
>>
>>
>> I think option 3 makes the most sense here (pending anyone saying we
>> should run away screaming from mox3 for some reason).  It's actually
>> what I had been assuming since this thread a while back.
>
>
> What precisely is the benefit of moving the existing code to mox3 versus
> moving the existing code to mock? Is mox3 so similar to mox that the
> transition would be minimal?
>
>
>> This means that we don't need to *require* that tests get converted if
>> you're changing one.  It just gets you bonus imaginary internet points.
>>
>> Requiring mock for new tests seems fine.  We can grant exceptions in
>> specific cases if necessary.  In general, we should be using mock for
>> new tests.
>
>
> My vote would be to use mock for everything new (no brainer), keep old mox
> stuff around and slowly port it to mock. I see little value in bringing in
> another mox3 library, especially if we'd end up having to maintain it.

FWIW this is exactly what the Cinder team agreed upon a while back and
the direction we've been going.  There hasn't really been any
push-back on this and in most cases the response from people has been
"Wow, using mock was so much easier/straight forward".

>
> Best,
> -jay
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Ben Nemec]

On 2013-12-21 07:24, Matt Riedemann wrote:

On 12/19/2013 8:51 AM, John Garbutt wrote:

On 4 December 2013 17:10, Russell Bryant  wrote:

I think option 3 makes the most sense here (pending anyone saying we
should run away screaming from mox3 for some reason).  It's actually
what I had been assuming since this thread a while back.

This means that we don't need to *require* that tests get converted 
if
you're changing one.  It just gets you bonus imaginary internet 
points.


Requiring mock for new tests seems fine.  We can grant exceptions in
specific cases if necessary.  In general, we should be using mock for
new tests.


I have lost track a bit here.

The above seems like a sane approach. Do we all agree on that now?

Can we add the above text into here:
https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist

John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Yeah, at some point I wanted to cleanup the various testing guides but
until then I like the idea of just putting something simple into the
nova review checklist. Basically use mock for new tests, mox can be
used in exceptional cases. What I've considered exceptional so far
includes changes that will be backported to a stable release where
mock isn't being used and cases where you basically have to bend over
backwards to work new mock tests into an existing test class that has
lots of existing setUp with mox. However, even in the latter case you
can usually use mock after resetting the mox setup via
self.mox.ResetAll() in the new test case(s).


I went ahead and added this to the wiki, so it's now an absolutely 
inviolate policy.  Unless, ya know, someone edits the wiki after me. ;-)


Also, I put it in the common section because this doesn't seem like 
something we should be doing differently per-project.  If anyone 
objects, feel free to add to the discussion here as to why.


Thanks.

-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Matt Riedemann]

On 12/19/2013 8:51 AM, John Garbutt wrote:

On 4 December 2013 17:10, Russell Bryant  wrote:

I think option 3 makes the most sense here (pending anyone saying we
should run away screaming from mox3 for some reason).  It's actually
what I had been assuming since this thread a while back.

This means that we don't need to *require* that tests get converted if
you're changing one.  It just gets you bonus imaginary internet points.

Requiring mock for new tests seems fine.  We can grant exceptions in
specific cases if necessary.  In general, we should be using mock for
new tests.


I have lost track a bit here.

The above seems like a sane approach. Do we all agree on that now?

Can we add the above text into here:
https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist

John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Yeah, at some point I wanted to cleanup the various testing guides but 
until then I like the idea of just putting something simple into the 
nova review checklist. Basically use mock for new tests, mox can be used 
in exceptional cases. What I've considered exceptional so far includes 
changes that will be backported to a stable release where mock isn't 
being used and cases where you basically have to bend over backwards to 
work new mock tests into an existing test class that has lots of 
existing setUp with mox. However, even in the latter case you can 
usually use mock after resetting the mox setup via self.mox.ResetAll() 
in the new test case(s).


--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[John Garbutt]

On 4 December 2013 17:10, Russell Bryant  wrote:
> I think option 3 makes the most sense here (pending anyone saying we
> should run away screaming from mox3 for some reason).  It's actually
> what I had been assuming since this thread a while back.
>
> This means that we don't need to *require* that tests get converted if
> you're changing one.  It just gets you bonus imaginary internet points.
>
> Requiring mock for new tests seems fine.  We can grant exceptions in
> specific cases if necessary.  In general, we should be using mock for
> new tests.

I have lost track a bit here.

The above seems like a sane approach. Do we all agree on that now?

Can we add the above text into here:
https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist

John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Ben Nemec]

On 2013-12-05 21:38, Jay Pipes wrote:

On 12/04/2013 12:10 PM, Russell Bryant wrote:

On 12/04/2013 11:16 AM, Nikola Đipanov wrote:
Resurrecting this thread because of an interesting review that came 
up

yesterday [1].

It seems that our lack of a firm decision on what to do with the 
mocking
framework has left people confused. In hope to help - I'll give my 
view

of where things are now and what we should do going forward, and
hopefully we'll reach some consensus on this.

Here's the breakdown:

We should abandon mox:
* It has not had a release in over 3 years [2] and a patch upstream 
for 2

* There are bugs that are impacting the project with it (see above)
* It will not be ported to python 3

Proposed path forward options:
1) Port nova to mock now:
   * Literally unmanageable - huge review overhead and regression 
risk

for not so much gain (maybe) [1]

2) Opportunistically port nova (write new tests using mock, when 
fixing

tests, move them to mock):
  * Will take a really long time to move to mock, and is not really a
solution since we are stuck with mock for an undetermined period of 
time

- it's what we are doing now (kind of).

3) Same as 2) but move current codebase to mox3
  * Buys us py3k compat, and fresher code
  * Mox3 and mox have diverged and we would need to backport mox 
fixes
onto the mox3 three and become de-facto active maintainers (as per 
Peter

Feiner's last email - that may not be so easy).

I think we should follow path 3) if we can, but we need to:

1) Figure out what is the deal with mox3 and decide if owning it will
really be less trouble than porting nova. To be hones - I was unable 
to

even find the code repo for it, only [3]. If anyone has more info -
please weigh in. We'll also need volunteers

2) Make better testing guidelines when using mock, and maybe add some
testing helpers (like we do already have for mox) that will make 
porting

existing tests easier. mreidem already put this on this weeks nova
meeting agenda - so that might be a good place to discuss all the 
issues

mentioned here as well.

We should really take a stronger stance on this soon IMHO, as this 
comes

up with literally every commit.


I think option 3 makes the most sense here (pending anyone saying we
should run away screaming from mox3 for some reason).  It's actually
what I had been assuming since this thread a while back.


What precisely is the benefit of moving the existing code to mox3
versus moving the existing code to mock? Is mox3 so similar to mox
that the transition would be minimal?


This means that we don't need to *require* that tests get converted if
you're changing one.  It just gets you bonus imaginary internet 
points.


Requiring mock for new tests seems fine.  We can grant exceptions in
specific cases if necessary.  In general, we should be using mock for
new tests.


My vote would be to use mock for everything new (no brainer), keep old
mox stuff around and slowly port it to mock. I see little value in
bringing in another mox3 library, especially if we'd end up having to
maintain it.


My understanding is that mox3 is a drop-in, Python 3 compatible version 
of mox.


I agree that spending any significant time maintaining mox3 is a bad 
thing at this point.  Mock is part of the stdlib in Python 3 and I don't 
think we should put a lot of time into reinventing the wheel.  That 
said, as long as mox3 works right now I don't think we should be 
rewriting mox test cases just to move them to mock either.  That's a 
whole lot of code churn for basically no benefit.


So my preference would be to:
1) Use mock for new test cases, with possible exceptions for adding to 
test classes that already use mox
2) Leave the existing mox test cases alone as long as they work fine 
with mox3.

3) If any test cases don't work in mox3, rewrite them in mock

-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Jay Pipes]

On 12/04/2013 12:10 PM, Russell Bryant wrote:

On 12/04/2013 11:16 AM, Nikola Đipanov wrote:

Resurrecting this thread because of an interesting review that came up
yesterday [1].

It seems that our lack of a firm decision on what to do with the mocking
framework has left people confused. In hope to help - I'll give my view
of where things are now and what we should do going forward, and
hopefully we'll reach some consensus on this.

Here's the breakdown:

We should abandon mox:
* It has not had a release in over 3 years [2] and a patch upstream for 2
* There are bugs that are impacting the project with it (see above)
* It will not be ported to python 3

Proposed path forward options:
1) Port nova to mock now:
   * Literally unmanageable - huge review overhead and regression risk
for not so much gain (maybe) [1]

2) Opportunistically port nova (write new tests using mock, when fixing
tests, move them to mock):
  * Will take a really long time to move to mock, and is not really a
solution since we are stuck with mock for an undetermined period of time
- it's what we are doing now (kind of).

3) Same as 2) but move current codebase to mox3
  * Buys us py3k compat, and fresher code
  * Mox3 and mox have diverged and we would need to backport mox fixes
onto the mox3 three and become de-facto active maintainers (as per Peter
Feiner's last email - that may not be so easy).

I think we should follow path 3) if we can, but we need to:

1) Figure out what is the deal with mox3 and decide if owning it will
really be less trouble than porting nova. To be hones - I was unable to
even find the code repo for it, only [3]. If anyone has more info -
please weigh in. We'll also need volunteers

2) Make better testing guidelines when using mock, and maybe add some
testing helpers (like we do already have for mox) that will make porting
existing tests easier. mreidem already put this on this weeks nova
meeting agenda - so that might be a good place to discuss all the issues
mentioned here as well.

We should really take a stronger stance on this soon IMHO, as this comes
up with literally every commit.


I think option 3 makes the most sense here (pending anyone saying we
should run away screaming from mox3 for some reason).  It's actually
what I had been assuming since this thread a while back.


What precisely is the benefit of moving the existing code to mox3 versus 
moving the existing code to mock? Is mox3 so similar to mox that the 
transition would be minimal?



This means that we don't need to *require* that tests get converted if
you're changing one.  It just gets you bonus imaginary internet points.

Requiring mock for new tests seems fine.  We can grant exceptions in
specific cases if necessary.  In general, we should be using mock for
new tests.


My vote would be to use mock for everything new (no brainer), keep old 
mox stuff around and slowly port it to mock. I see little value in 
bringing in another mox3 library, especially if we'd end up having to 
maintain it.


Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Matt Riedemann]

On 12/4/2013 10:16 AM, Nikola Đipanov wrote:

On 11/19/2013 05:52 PM, Peter Feiner wrote:

On Tue, Nov 19, 2013 at 11:19 AM, Chuck Short  wrote:

Hi


On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner  wrote:


A substantive reason for switching from mox to mock is the derelict
state of mox releases. There hasn't been a release of mox in three
years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover,
in the past 3 years, substantial bugs have been fixed in upstream mox.
For example, with the year-old fix to
https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug
in nova would have been caught by an existing test [3].

Alternatively, a copy of the upstream mox code could be added in-tree.


Please no, I think we are in an agreement with mox3 and mock.


That's cool. As long as the mox* is phased out, the false-positive
test results will be fixed.

Of course, there's _another_ alternative, which is to retrofit mox3
with the upstream mox fixes (e.g., the bug I cited above exists in
mox3). However, the delta between mox3 and upstream mox is pretty huge
(I just checked), so effort is probably better spent switching to
mock. To that end, I plan on changing the tests I cited above.



Resurrecting this thread because of an interesting review that came up
yesterday [1].

It seems that our lack of a firm decision on what to do with the mocking
framework has left people confused. In hope to help - I'll give my view
of where things are now and what we should do going forward, and
hopefully we'll reach some consensus on this.

Here's the breakdown:

We should abandon mox:
* It has not had a release in over 3 years [2] and a patch upstream for 2
* There are bugs that are impacting the project with it (see above)
* It will not be ported to python 3

Proposed path forward options:
1) Port nova to mock now:
   * Literally unmanageable - huge review overhead and regression risk
for not so much gain (maybe) [1]

2) Opportunistically port nova (write new tests using mock, when fixing
tests, move them to mock):
  * Will take a really long time to move to mock, and is not really a
solution since we are stuck with mock for an undetermined period of time
- it's what we are doing now (kind of).

3) Same as 2) but move current codebase to mox3
  * Buys us py3k compat, and fresher code
  * Mox3 and mox have diverged and we would need to backport mox fixes
onto the mox3 three and become de-facto active maintainers (as per Peter
Feiner's last email - that may not be so easy).

I think we should follow path 3) if we can, but we need to:

1) Figure out what is the deal with mox3 and decide if owning it will
really be less trouble than porting nova. To be hones - I was unable to
even find the code repo for it, only [3]. If anyone has more info -
please weigh in. We'll also need volunteers

2) Make better testing guidelines when using mock, and maybe add some
testing helpers (like we do already have for mox) that will make porting
existing tests easier. mreidem already put this on this weeks nova
meeting agenda - so that might be a good place to discuss all the issues
mentioned here as well.


For anyone interested in seeing the nova meeting agenda notes before the 
actual meeting (since it covers more than just mock/mox), here you go:


https://wiki.openstack.org/wiki/Meetings/Nova

Copying here for reference later if necessary:

- Testing Guides - we need some guide on using mox vs mock; previously 
we were forcing all new tests to use mock but then mox3 was pointed out 
and that seemed to go away, but is anyone using mox3 yet 
(ceilomerclient, heatclient and ironicclient are all moving to mox3).
  - ML on stubs vs mox vs mock: 
http://lists.openstack.org/pipermail/openstack-dev/2013-November/018501.html
  - nova/tests/README.rst is outdated; lots of TBD and last update was 
on 2013/05/16: 
https://github.com/openstack/nova/blob/master/nova/tests/README.rst
- Note keystone bug 1252454 where the keystone docs were 
referencing the nova testing README
  - Ultimately it seems the best guide is Horizon's: 
http://docs.openstack.org/developer/horizon/topics/testing.html
  - We should have something like the Horizon guide in the overall 
OpenStack hacking guide so the projects can point to a single location, 
http://docs.openstack.org/developer/hacking/, but even that is sparse on 
details for writing unit tests (should almost merge or point to the 
horizon guide), and then the single guide should have notes on stubs vs 
mox vs mock with pros/cons and best practices / pitfalls.


===

The net problem is we have a lot of various testing guides throughout 
the projects, including hacking guides, readmes, wikis, and devref 
pages, we should really consolidate that into the universal hacking 
guide (IMO) and then integrate Horizon's excellent guide into that along 
with whatever rules we have for mox/mox3/mock.




We should really take a stronger stance on this soon IMHO, as this comes
up with literally 

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Chuck Short]

On Wed, Dec 4, 2013 at 11:16 AM, Nikola Đipanov  wrote:

> On 11/19/2013 05:52 PM, Peter Feiner wrote:
> > On Tue, Nov 19, 2013 at 11:19 AM, Chuck Short 
> wrote:
> >> Hi
> >>
> >>
> >> On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner 
> wrote:
> >>>
> >>> A substantive reason for switching from mox to mock is the derelict
> >>> state of mox releases. There hasn't been a release of mox in three
> >>> years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover,
> >>> in the past 3 years, substantial bugs have been fixed in upstream mox.
> >>> For example, with the year-old fix to
> >>> https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug
> >>> in nova would have been caught by an existing test [3].
> >>>
> >>> Alternatively, a copy of the upstream mox code could be added in-tree.
> >>>
> >> Please no, I think we are in an agreement with mox3 and mock.
> >
> > That's cool. As long as the mox* is phased out, the false-positive
> > test results will be fixed.
> >
> > Of course, there's _another_ alternative, which is to retrofit mox3
> > with the upstream mox fixes (e.g., the bug I cited above exists in
> > mox3). However, the delta between mox3 and upstream mox is pretty huge
> > (I just checked), so effort is probably better spent switching to
> > mock. To that end, I plan on changing the tests I cited above.
> >
>
> Resurrecting this thread because of an interesting review that came up
> yesterday [1].
>
> It seems that our lack of a firm decision on what to do with the mocking
> framework has left people confused. In hope to help - I'll give my view
> of where things are now and what we should do going forward, and
> hopefully we'll reach some consensus on this.
>
> Here's the breakdown:
>
> We should abandon mox:
> * It has not had a release in over 3 years [2] and a patch upstream for 2
> * There are bugs that are impacting the project with it (see above)
> * It will not be ported to python 3
>
> Proposed path forward options:
> 1) Port nova to mock now:
>   * Literally unmanageable - huge review overhead and regression risk
> for not so much gain (maybe) [1]
>
> 2) Opportunistically port nova (write new tests using mock, when fixing
> tests, move them to mock):
>  * Will take a really long time to move to mock, and is not really a
> solution since we are stuck with mock for an undetermined period of time
> - it's what we are doing now (kind of).
>
> 3) Same as 2) but move current codebase to mox3
>  * Buys us py3k compat, and fresher code
>  * Mox3 and mox have diverged and we would need to backport mox fixes
> onto the mox3 three and become de-facto active maintainers (as per Peter
> Feiner's last email - that may not be so easy).
>
>
So I thought we cleared this up already. We convert the current codebase
over to mox3, new tests should be done in mock. Eventually we start
converting over code to use mock.



> I think we should follow path 3) if we can, but we need to:
>
> 1) Figure out what is the deal with mox3 and decide if owning it will
> really be less trouble than porting nova. To be hones - I was unable to
> even find the code repo for it, only [3]. If anyone has more info -
> please weigh in. We'll also need volunteers
>
>
Monty and I did this last cycle, its apart of the openstack project,
although its not available in gerrit. Which should be fixed so we can start
getting bug fixes in for it.


> 2) Make better testing guidelines when using mock, and maybe add some
> testing helpers (like we do already have for mox) that will make porting
> existing tests easier. mreidem already put this on this weeks nova
> meeting agenda - so that might be a good place to discuss all the issues
> mentioned here as well.
>
> We should really take a stronger stance on this soon IMHO, as this comes
> up with literally every commit.
>

totally


>
> Cheers,
>
> Nikola
>
> [1] https://review.openstack.org/#/c/59694/
> [2] https://pypi.python.org/pypi/mox
> [3] https://pypi.python.org/pypi/mox3/0.7.0
>
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Nikola Đipanov]

On 12/04/2013 06:15 PM, Peter Feiner wrote:
> On Wed, Dec 4, 2013 at 11:16 AM, Nikola Đipanov  wrote:
>> 1) Figure out what is the deal with mox3 and decide if owning it will
>> really be less trouble than porting nova. To be hones - I was unable to
>> even find the code repo for it, only [3]. If anyone has more info -
>> please weigh in. We'll also need volunteers
>>
>> [3] https://pypi.python.org/pypi/mox3/0.7.0
> 
> That's all I was able to find.
> 

The package seems to be owned by people from the community - so maybe
someone will respond on this thread. Or if anyone knows who is behind
the package - let us know - it would be good to start figuring this out
sooner rather than later.

N.

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Peter Feiner]

On Wed, Dec 4, 2013 at 11:16 AM, Nikola Đipanov  wrote:
> 1) Figure out what is the deal with mox3 and decide if owning it will
> really be less trouble than porting nova. To be hones - I was unable to
> even find the code repo for it, only [3]. If anyone has more info -
> please weigh in. We'll also need volunteers
>
> [3] https://pypi.python.org/pypi/mox3/0.7.0

That's all I was able to find.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Russell Bryant]

On 12/04/2013 11:16 AM, Nikola Đipanov wrote:
> Resurrecting this thread because of an interesting review that came up
> yesterday [1].
> 
> It seems that our lack of a firm decision on what to do with the mocking
> framework has left people confused. In hope to help - I'll give my view
> of where things are now and what we should do going forward, and
> hopefully we'll reach some consensus on this.
> 
> Here's the breakdown:
> 
> We should abandon mox:
> * It has not had a release in over 3 years [2] and a patch upstream for 2
> * There are bugs that are impacting the project with it (see above)
> * It will not be ported to python 3
> 
> Proposed path forward options:
> 1) Port nova to mock now:
>   * Literally unmanageable - huge review overhead and regression risk
> for not so much gain (maybe) [1]
> 
> 2) Opportunistically port nova (write new tests using mock, when fixing
> tests, move them to mock):
>  * Will take a really long time to move to mock, and is not really a
> solution since we are stuck with mock for an undetermined period of time
> - it's what we are doing now (kind of).
> 
> 3) Same as 2) but move current codebase to mox3
>  * Buys us py3k compat, and fresher code
>  * Mox3 and mox have diverged and we would need to backport mox fixes
> onto the mox3 three and become de-facto active maintainers (as per Peter
> Feiner's last email - that may not be so easy).
> 
> I think we should follow path 3) if we can, but we need to:
> 
> 1) Figure out what is the deal with mox3 and decide if owning it will
> really be less trouble than porting nova. To be hones - I was unable to
> even find the code repo for it, only [3]. If anyone has more info -
> please weigh in. We'll also need volunteers
> 
> 2) Make better testing guidelines when using mock, and maybe add some
> testing helpers (like we do already have for mox) that will make porting
> existing tests easier. mreidem already put this on this weeks nova
> meeting agenda - so that might be a good place to discuss all the issues
> mentioned here as well.
> 
> We should really take a stronger stance on this soon IMHO, as this comes
> up with literally every commit.

I think option 3 makes the most sense here (pending anyone saying we
should run away screaming from mox3 for some reason).  It's actually
what I had been assuming since this thread a while back.

This means that we don't need to *require* that tests get converted if
you're changing one.  It just gets you bonus imaginary internet points.

Requiring mock for new tests seems fine.  We can grant exceptions in
specific cases if necessary.  In general, we should be using mock for
new tests.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Nikola Đipanov]

On 11/19/2013 05:52 PM, Peter Feiner wrote:
> On Tue, Nov 19, 2013 at 11:19 AM, Chuck Short  
> wrote:
>> Hi
>>
>>
>> On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner  wrote:
>>>
>>> A substantive reason for switching from mox to mock is the derelict
>>> state of mox releases. There hasn't been a release of mox in three
>>> years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover,
>>> in the past 3 years, substantial bugs have been fixed in upstream mox.
>>> For example, with the year-old fix to
>>> https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug
>>> in nova would have been caught by an existing test [3].
>>>
>>> Alternatively, a copy of the upstream mox code could be added in-tree.
>>>
>> Please no, I think we are in an agreement with mox3 and mock.
> 
> That's cool. As long as the mox* is phased out, the false-positive
> test results will be fixed.
> 
> Of course, there's _another_ alternative, which is to retrofit mox3
> with the upstream mox fixes (e.g., the bug I cited above exists in
> mox3). However, the delta between mox3 and upstream mox is pretty huge
> (I just checked), so effort is probably better spent switching to
> mock. To that end, I plan on changing the tests I cited above.
> 

Resurrecting this thread because of an interesting review that came up
yesterday [1].

It seems that our lack of a firm decision on what to do with the mocking
framework has left people confused. In hope to help - I'll give my view
of where things are now and what we should do going forward, and
hopefully we'll reach some consensus on this.

Here's the breakdown:

We should abandon mox:
* It has not had a release in over 3 years [2] and a patch upstream for 2
* There are bugs that are impacting the project with it (see above)
* It will not be ported to python 3

Proposed path forward options:
1) Port nova to mock now:
  * Literally unmanageable - huge review overhead and regression risk
for not so much gain (maybe) [1]

2) Opportunistically port nova (write new tests using mock, when fixing
tests, move them to mock):
 * Will take a really long time to move to mock, and is not really a
solution since we are stuck with mock for an undetermined period of time
- it's what we are doing now (kind of).

3) Same as 2) but move current codebase to mox3
 * Buys us py3k compat, and fresher code
 * Mox3 and mox have diverged and we would need to backport mox fixes
onto the mox3 three and become de-facto active maintainers (as per Peter
Feiner's last email - that may not be so easy).

I think we should follow path 3) if we can, but we need to:

1) Figure out what is the deal with mox3 and decide if owning it will
really be less trouble than porting nova. To be hones - I was unable to
even find the code repo for it, only [3]. If anyone has more info -
please weigh in. We'll also need volunteers

2) Make better testing guidelines when using mock, and maybe add some
testing helpers (like we do already have for mox) that will make porting
existing tests easier. mreidem already put this on this weeks nova
meeting agenda - so that might be a good place to discuss all the issues
mentioned here as well.

We should really take a stronger stance on this soon IMHO, as this comes
up with literally every commit.

Cheers,

Nikola

[1] https://review.openstack.org/#/c/59694/
[2] https://pypi.python.org/pypi/mox
[3] https://pypi.python.org/pypi/mox3/0.7.0

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Peter Feiner]

On Tue, Nov 19, 2013 at 11:19 AM, Chuck Short  wrote:
> Hi
>
>
> On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner  wrote:
>>
>> A substantive reason for switching from mox to mock is the derelict
>> state of mox releases. There hasn't been a release of mox in three
>> years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover,
>> in the past 3 years, substantial bugs have been fixed in upstream mox.
>> For example, with the year-old fix to
>> https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug
>> in nova would have been caught by an existing test [3].
>>
>> Alternatively, a copy of the upstream mox code could be added in-tree.
>>
> Please no, I think we are in an agreement with mox3 and mock.

That's cool. As long as the mox* is phased out, the false-positive
test results will be fixed.

Of course, there's _another_ alternative, which is to retrofit mox3
with the upstream mox fixes (e.g., the bug I cited above exists in
mox3). However, the delta between mox3 and upstream mox is pretty huge
(I just checked), so effort is probably better spent switching to
mock. To that end, I plan on changing the tests I cited above.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Chuck Short]

Hi


On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner  wrote:

> A substantive reason for switching from mox to mock is the derelict
> state of mox releases. There hasn't been a release of mox in three
> years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover,
> in the past 3 years, substantial bugs have been fixed in upstream mox.
> For example, with the year-old fix to
> https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug
> in nova would have been caught by an existing test [3].
>
> Alternatively, a copy of the upstream mox code could be added in-tree.
>
> Please no, I think we are in an agreement with mox3 and mock.



> [1] mox releases: https://code.google.com/p/pymox/downloads/list
> [2] mox on pypi: https://pypi.python.org/pypi/mox
> [3] see comments 5 and 6 in https://bugs.launchpad.net/nova/+bug/1251792
>
> On Wed, Nov 13, 2013 at 2:24 PM, Matt Riedemann
>  wrote:
> >
> >
> > On 11/12/2013 5:04 PM, Chuck Short wrote:
> >>
> >>
> >>
> >>
> >> On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin  >> > wrote:
> >>
> >> On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote:
> >>  >
> >>  > Hi
> >>  >
> >>  >
> >>  > On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin
> >> mailto:mar...@redhat.com>>
> >>
> >>  > wrote:
> >>  > On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
> >>  > > Maybe we should have some 60% rule... that is: If you
> >> change
> >>  > more than
> >>  > > half of a test... you should *probably* rewrite the
> test
> >> in
> >>  > Mock.
> >>  >
> >>  >
> >>  > A rule needs a reasoning attached to it :)
> >>  >
> >>  > Why do we want people to use mock?
> >>  >
> >>  > Is it really for Python3? If so, I assume that means
> we've
> >>  > ruled out the
> >>  > python3 port of mox? (Ok by me, but would be good to hear
> >> why)
> >>  > And, if
> >>  > that's the case, then we should encourage whoever wants
> to
> >>  > port mox
> >>  > based tests to mock.
> >>  >
> >>  >
> >>  >
> >>  > The upstream maintainer is not going to port mox to python3 so we
> >> have
> >>  > a fork of mox called mox3. Ideally, we would drop the usage of
> mox
> >> in
> >>  > favour of mock so we don't have to carry a forked mox.
> >>
> >> Isn't that the opposite conclusion you came to here:
> >>
> >>
> >>
> http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html
> >>
> >> i.e. using mox3 results in less code churn?
> >>
> >> Mark.
> >>
> >>
> >>
> >> Yes that was my original position but I though we agreed in thread
> >> (further on) that we would use mox3 and then migrate to mock further on.
> >>
> >> Regards
> >> chuck
> >>
> >>
> >> ___
> >> OpenStack-dev mailing list
> >> OpenStack-dev@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> > So it sounds like we're good with using mox for new tests again? Given
> Chuck
> > got it into global-requirements here:
> >
> >
> https://github.com/openstack/requirements/commit/998dda263d7c7881070e3f16e4523ddcd23fc36d
> >
> > We can stave off the need to transition everything from mox to mock?
> >
> > I can't seem to find the nova blueprint to convert everything from mox to
> > mock, maybe it was obsoleted already.
> >
> > Anyway, if mox(3) is OK and we don't need to use mock, it seems like we
> > could add something to the developer guide here because I think this
> > question comes up frequently:
> >
> > http://docs.openstack.org/developer/nova/devref/unit_tests.html
> >
> > Does anyone disagree?
> >
> > BTW, I care about this because I've been keeping in mind the mox/mock
> > transition when doing code reviews and giving a -1 when new tests are
> using
> > mox (since I thought that was a no-no now).
> > --
> >
> > Thanks,
> >
> > Matt Riedemann
> >
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Peter Feiner]

A substantive reason for switching from mox to mock is the derelict
state of mox releases. There hasn't been a release of mox in three
years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover,
in the past 3 years, substantial bugs have been fixed in upstream mox.
For example, with the year-old fix to
https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug
in nova would have been caught by an existing test [3].

Alternatively, a copy of the upstream mox code could be added in-tree.

[1] mox releases: https://code.google.com/p/pymox/downloads/list
[2] mox on pypi: https://pypi.python.org/pypi/mox
[3] see comments 5 and 6 in https://bugs.launchpad.net/nova/+bug/1251792

On Wed, Nov 13, 2013 at 2:24 PM, Matt Riedemann
 wrote:
>
>
> On 11/12/2013 5:04 PM, Chuck Short wrote:
>>
>>
>>
>>
>> On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin > > wrote:
>>
>> On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote:
>>  >
>>  > Hi
>>  >
>>  >
>>  > On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin
>> mailto:mar...@redhat.com>>
>>
>>  > wrote:
>>  > On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
>>  > > Maybe we should have some 60% rule... that is: If you
>> change
>>  > more than
>>  > > half of a test... you should *probably* rewrite the test
>> in
>>  > Mock.
>>  >
>>  >
>>  > A rule needs a reasoning attached to it :)
>>  >
>>  > Why do we want people to use mock?
>>  >
>>  > Is it really for Python3? If so, I assume that means we've
>>  > ruled out the
>>  > python3 port of mox? (Ok by me, but would be good to hear
>> why)
>>  > And, if
>>  > that's the case, then we should encourage whoever wants to
>>  > port mox
>>  > based tests to mock.
>>  >
>>  >
>>  >
>>  > The upstream maintainer is not going to port mox to python3 so we
>> have
>>  > a fork of mox called mox3. Ideally, we would drop the usage of mox
>> in
>>  > favour of mock so we don't have to carry a forked mox.
>>
>> Isn't that the opposite conclusion you came to here:
>>
>>
>> http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html
>>
>> i.e. using mox3 results in less code churn?
>>
>> Mark.
>>
>>
>>
>> Yes that was my original position but I though we agreed in thread
>> (further on) that we would use mox3 and then migrate to mock further on.
>>
>> Regards
>> chuck
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> So it sounds like we're good with using mox for new tests again? Given Chuck
> got it into global-requirements here:
>
> https://github.com/openstack/requirements/commit/998dda263d7c7881070e3f16e4523ddcd23fc36d
>
> We can stave off the need to transition everything from mox to mock?
>
> I can't seem to find the nova blueprint to convert everything from mox to
> mock, maybe it was obsoleted already.
>
> Anyway, if mox(3) is OK and we don't need to use mock, it seems like we
> could add something to the developer guide here because I think this
> question comes up frequently:
>
> http://docs.openstack.org/developer/nova/devref/unit_tests.html
>
> Does anyone disagree?
>
> BTW, I care about this because I've been keeping in mind the mox/mock
> transition when doing code reviews and giving a -1 when new tests are using
> mox (since I thought that was a no-no now).
> --
>
> Thanks,
>
> Matt Riedemann
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Matt Riedemann]

On 11/12/2013 5:04 PM, Chuck Short wrote:




On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin mailto:mar...@redhat.com>> wrote:

On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote:
 >
 > Hi
 >
 >
 > On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin
mailto:mar...@redhat.com>>
 > wrote:
 > On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
 > > Maybe we should have some 60% rule... that is: If you
change
 > more than
 > > half of a test... you should *probably* rewrite the test in
 > Mock.
 >
 >
 > A rule needs a reasoning attached to it :)
 >
 > Why do we want people to use mock?
 >
 > Is it really for Python3? If so, I assume that means we've
 > ruled out the
 > python3 port of mox? (Ok by me, but would be good to hear
why)
 > And, if
 > that's the case, then we should encourage whoever wants to
 > port mox
 > based tests to mock.
 >
 >
 >
 > The upstream maintainer is not going to port mox to python3 so we
have
 > a fork of mox called mox3. Ideally, we would drop the usage of mox in
 > favour of mock so we don't have to carry a forked mox.

Isn't that the opposite conclusion you came to here:

http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html

i.e. using mox3 results in less code churn?

Mark.



Yes that was my original position but I though we agreed in thread
(further on) that we would use mox3 and then migrate to mock further on.

Regards
chuck


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



So it sounds like we're good with using mox for new tests again? Given 
Chuck got it into global-requirements here:


https://github.com/openstack/requirements/commit/998dda263d7c7881070e3f16e4523ddcd23fc36d

We can stave off the need to transition everything from mox to mock?

I can't seem to find the nova blueprint to convert everything from mox 
to mock, maybe it was obsoleted already.


Anyway, if mox(3) is OK and we don't need to use mock, it seems like we 
could add something to the developer guide here because I think this 
question comes up frequently:


http://docs.openstack.org/developer/nova/devref/unit_tests.html

Does anyone disagree?

BTW, I care about this because I've been keeping in mind the mox/mock 
transition when doing code reviews and giving a -1 when new tests are 
using mox (since I thought that was a no-no now).

--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Chuck Short]

On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin  wrote:

> On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote:
> >
> > Hi
> >
> >
> > On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin 
> > wrote:
> > On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
> > > Maybe we should have some 60% rule... that is: If you change
> > more than
> > > half of a test... you should *probably* rewrite the test in
> > Mock.
> >
> >
> > A rule needs a reasoning attached to it :)
> >
> > Why do we want people to use mock?
> >
> > Is it really for Python3? If so, I assume that means we've
> > ruled out the
> > python3 port of mox? (Ok by me, but would be good to hear why)
> > And, if
> > that's the case, then we should encourage whoever wants to
> > port mox
> > based tests to mock.
> >
> >
> >
> > The upstream maintainer is not going to port mox to python3 so we have
> > a fork of mox called mox3. Ideally, we would drop the usage of mox in
> > favour of mock so we don't have to carry a forked mox.
>
> Isn't that the opposite conclusion you came to here:
>
> http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html
>
> i.e. using mox3 results in less code churn?
>
> Mark.
>
>
>
Yes that was my original position but I though we agreed in thread (further
on) that we would use mox3 and then migrate to mock further on.

Regards
chuck
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Angus Salkeld]

On 12/11/13 21:49 +, Mark McLoughlin wrote:

On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote:


Hi


On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin 
wrote:
On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
> Maybe we should have some 60% rule... that is: If you change
more than
> half of a test... you should *probably* rewrite the test in
Mock.


A rule needs a reasoning attached to it :)

Why do we want people to use mock?

Is it really for Python3? If so, I assume that means we've
ruled out the
python3 port of mox? (Ok by me, but would be good to hear why)
And, if
that's the case, then we should encourage whoever wants to
port mox
based tests to mock.



The upstream maintainer is not going to port mox to python3 so we have
a fork of mox called mox3. Ideally, we would drop the usage of mox in
favour of mock so we don't have to carry a forked mox.


Isn't that the opposite conclusion you came to here:

http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html

i.e. using mox3 results in less code churn?


Can't we at least seperate the two concerns?

Move everything to mox3 for python3 - should just be mostly
import changes.

Then decide if we want to migrate to mock (I like mock, but it can be
lots of work changing it). In Heat new tests are done in mock, but
we are not wholesale migrating to mock - 'cos well there is lots of
better things to do.

-Angus



Mark.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Shawn Hartsock]

So the real argument for Mock over Mox is that Mock is part of the stdlib in 
Python 3.3 ... that seems like a pretty sound argument to me.

# Shawn Hartsock


- Original Message -
> From: "Ben Nemec" 
> To: openstack-dev@lists.openstack.org
> Sent: Tuesday, November 12, 2013 4:41:47 PM
> Subject: Re: [openstack-dev]  [nova] Do we have some guidelines for mock,  
> stub, mox when writing unit test?
> 
> On 2013-11-12 15:27, Shawn Hartsock wrote:
> > Good point.
> > 
> > I assume someone made a comparison similar to this:
> > * http://garybernhardt.github.io/python-mock-comparison/
> > 
> > ... and evangelized a choice. I had assumed that Mock vs mox was not
> > merely based on Python3 support but had something to do with Mock
> > versus Mox. Does anyone have that context in their head? Would they
> > mind sharing it?
> 
> Not in my head, but the internet remembers everything :-)
> 
> http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html
> 
> In short, the majority of respondents liked Mock better, it seems to be
> more widely used, and Mock is part of the stdlib in Python 3.3.
> 
> -Ben
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Ben Nemec]

On 2013-11-12 15:27, Shawn Hartsock wrote:

Good point.

I assume someone made a comparison similar to this:
* http://garybernhardt.github.io/python-mock-comparison/

... and evangelized a choice. I had assumed that Mock vs mox was not
merely based on Python3 support but had something to do with Mock
versus Mox. Does anyone have that context in their head? Would they
mind sharing it?


Not in my head, but the internet remembers everything :-)

http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html

In short, the majority of respondents liked Mock better, it seems to be 
more widely used, and Mock is part of the stdlib in Python 3.3.


-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Mark McLoughlin]

On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote:
> 
> Hi
> 
> 
> On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin 
> wrote:
> On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
> > Maybe we should have some 60% rule... that is: If you change
> more than
> > half of a test... you should *probably* rewrite the test in
> Mock.
> 
> 
> A rule needs a reasoning attached to it :)
> 
> Why do we want people to use mock?
> 
> Is it really for Python3? If so, I assume that means we've
> ruled out the
> python3 port of mox? (Ok by me, but would be good to hear why)
> And, if
> that's the case, then we should encourage whoever wants to
> port mox
> based tests to mock.
> 
> 
> 
> The upstream maintainer is not going to port mox to python3 so we have
> a fork of mox called mox3. Ideally, we would drop the usage of mox in
> favour of mock so we don't have to carry a forked mox.

Isn't that the opposite conclusion you came to here:

http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html

i.e. using mox3 results in less code churn?

Mark.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Chuck Short]

http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html


On Tue, Nov 12, 2013 at 4:27 PM, Shawn Hartsock wrote:

> Good point.
>
> I assume someone made a comparison similar to this:
> * http://garybernhardt.github.io/python-mock-comparison/
>
> ... and evangelized a choice. I had assumed that Mock vs mox was not
> merely based on Python3 support but had something to do with Mock versus
> Mox. Does anyone have that context in their head? Would they mind sharing
> it?
>
> # Shawn Hartsock
>
>
> - Original Message -
> > From: "Mark McLoughlin" 
> > To: "Shawn Hartsock" 
> > Cc: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> > Sent: Tuesday, November 12, 2013 4:24:08 PM
> > Subject: Re: [openstack-dev] [nova] Do we have some guidelines for mock,
> stub, mox when writing unit test?
> >
> > On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
> > > Maybe we should have some 60% rule... that is: If you change more than
> > > half of a test... you should *probably* rewrite the test in Mock.
> >
> > A rule needs a reasoning attached to it :)
> >
> > Why do we want people to use mock?
> >
> > Is it really for Python3? If so, I assume that means we've ruled out the
> > python3 port of mox? (Ok by me, but would be good to hear why) And, if
> > that's the case, then we should encourage whoever wants to port mox
> > based tests to mock.
> >
> > Or maybe it has nothing to do with Python3 at all? Maybe we just like
> > mock more? But do we like it enough to have a mixture of mock and mox
> > across the codebase?
> >
> > Mark.
> >
> > > ----- Original Message -----
> > > > From: "John Garbutt" 
> > > > To: "Mark McLoughlin" , "OpenStack Development
> Mailing
> > > > List (not for usage questions)"
> > > > 
> > > > Sent: Tuesday, November 12, 2013 9:31:25 AM
> > > > Subject: Re: [openstack-dev] [nova] Do we have some guidelines for
> mock,
> > > > stub, mox when writing unit test?
> > > >
> > > > On 11 November 2013 23:18, Mark McLoughlin 
> wrote:
> > > > > On Mon, 2013-11-11 at 12:07 +, John Garbutt wrote:
> > > > >> On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services)
> > > > >>  wrote:
> > > > >> > Hi
> > > > >> >
> > > > >> >>Generally mock is supposed to be used over mox now for python 3
> > > > >> >>support.
> > > > >> >
> > > > >> > That is my understanding too
> > > > >>
> > > > >> +1
> > > > >>
> > > > >> But I don't think we should waste all our time re-writing all our
> mox
> > > > >> and stub tests. Lets just leave this to happen organically for
> now as
> > > > >> we add and refactor tests. We probably need to take the hit at
> some
> > > > >> point, but that doesn't feel like we should do that right now.
> > > > >
> > > > > Hmm, I don't follow this stance.
> > > > >
> > > > > Adding Python3 support is a goal we all share.
> > > > >
> > > > > If we're saying that the use of mox stands in the way of that
> goal, but
> > > > > that we'd really prefer if people didn't work on porting tests
> from mox
> > > > > to mock yet ... then are we saying we don't value people working on
> > > > > porting to Python3?
> > > > >
> > > > > And if we plan to use a Python3 compatible version of mox[1], then
> > > > > isn't
> > > > > the Python3 argument irrelevant and saying "use mock for new tests"
> > > > > just
> > > > > means we'll end up with a mixture of mox and mock?
> > > >
> > > > Good point, I forgot about the port of mox to python3.
> > > >
> > > > I liked the idea of "prefer mock", with a view that at some point in
> > > > the future there is only a small amount of mox related code left,
> that
> > > > can easily get moved to mock.
> > > >
> > > > I guess its a trade off between review capacity, risk of breaking
> > > > existing tests, and risk of never reaching that end goal.
> > > >
> > > > We already have stubs and mox, which do tend to fight each other,
> > > > adding a third does seem like a bad plan, unless there is a very good
> > > > reason, which I always had in my head as python3 support. Hmm... I do
> > > > prefer mock to mox, but not that strongly.
> > > >
> > > > John
> > > >
> > > > ___
> > > > OpenStack-dev mailing list
> > > > OpenStack-dev@lists.openstack.org
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > >
> >
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Chuck Short]

Hi

On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin  wrote:

> On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
> > Maybe we should have some 60% rule... that is: If you change more than
> > half of a test... you should *probably* rewrite the test in Mock.
>
> A rule needs a reasoning attached to it :)
>
> Why do we want people to use mock?
>
> Is it really for Python3? If so, I assume that means we've ruled out the
> python3 port of mox? (Ok by me, but would be good to hear why) And, if
> that's the case, then we should encourage whoever wants to port mox
> based tests to mock.
>
>
The upstream maintainer is not going to port mox to python3 so we have a
fork of mox called mox3. Ideally, we would drop the usage of mox in favour
of mock so we don't have to carry a forked mox.


> Or maybe it has nothing to do with Python3 at all? Maybe we just like
> mock more? But do we like it enough to have a mixture of mock and mox
> across the codebase?
>
> Mark.
>
> > - Original Message -
> > > From: "John Garbutt" 
> > > To: "Mark McLoughlin" , "OpenStack Development
> Mailing List (not for usage questions)"
> > > 
> > > Sent: Tuesday, November 12, 2013 9:31:25 AM
> > > Subject: Re: [openstack-dev] [nova] Do we have some guidelines for
> mock, stub, mox when writing unit test?
> > >
> > > On 11 November 2013 23:18, Mark McLoughlin  wrote:
> > > > On Mon, 2013-11-11 at 12:07 +, John Garbutt wrote:
> > > >> On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services)
> > > >>  wrote:
> > > >> > Hi
> > > >> >
> > > >> >>Generally mock is supposed to be used over mox now for python 3
> support.
> > > >> >
> > > >> > That is my understanding too
> > > >>
> > > >> +1
> > > >>
> > > >> But I don't think we should waste all our time re-writing all our
> mox
> > > >> and stub tests. Lets just leave this to happen organically for now
> as
> > > >> we add and refactor tests. We probably need to take the hit at some
> > > >> point, but that doesn't feel like we should do that right now.
> > > >
> > > > Hmm, I don't follow this stance.
> > > >
> > > > Adding Python3 support is a goal we all share.
> > > >
> > > > If we're saying that the use of mox stands in the way of that goal,
> but
> > > > that we'd really prefer if people didn't work on porting tests from
> mox
> > > > to mock yet ... then are we saying we don't value people working on
> > > > porting to Python3?
> > > >
> > > > And if we plan to use a Python3 compatible version of mox[1], then
> isn't
> > > > the Python3 argument irrelevant and saying "use mock for new tests"
> just
> > > > means we'll end up with a mixture of mox and mock?
> > >
> > > Good point, I forgot about the port of mox to python3.
> > >
> > > I liked the idea of "prefer mock", with a view that at some point in
> > > the future there is only a small amount of mox related code left, that
> > > can easily get moved to mock.
> > >
> > > I guess its a trade off between review capacity, risk of breaking
> > > existing tests, and risk of never reaching that end goal.
> > >
> > > We already have stubs and mox, which do tend to fight each other,
> > > adding a third does seem like a bad plan, unless there is a very good
> > > reason, which I always had in my head as python3 support. Hmm... I do
> > > prefer mock to mox, but not that strongly.
> > >
> > > John
> > >
> > > ___
> > > OpenStack-dev mailing list
> > > OpenStack-dev@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Shawn Hartsock]

Good point.

I assume someone made a comparison similar to this:
* http://garybernhardt.github.io/python-mock-comparison/

... and evangelized a choice. I had assumed that Mock vs mox was not merely 
based on Python3 support but had something to do with Mock versus Mox. Does 
anyone have that context in their head? Would they mind sharing it?

# Shawn Hartsock


- Original Message -
> From: "Mark McLoughlin" 
> To: "Shawn Hartsock" 
> Cc: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Tuesday, November 12, 2013 4:24:08 PM
> Subject: Re: [openstack-dev] [nova] Do we have some guidelines for mock, 
> stub, mox when writing unit test?
> 
> On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
> > Maybe we should have some 60% rule... that is: If you change more than
> > half of a test... you should *probably* rewrite the test in Mock.
> 
> A rule needs a reasoning attached to it :)
> 
> Why do we want people to use mock?
> 
> Is it really for Python3? If so, I assume that means we've ruled out the
> python3 port of mox? (Ok by me, but would be good to hear why) And, if
> that's the case, then we should encourage whoever wants to port mox
> based tests to mock.
> 
> Or maybe it has nothing to do with Python3 at all? Maybe we just like
> mock more? But do we like it enough to have a mixture of mock and mox
> across the codebase?
> 
> Mark.
> 
> > - Original Message -
> > > From: "John Garbutt" 
> > > To: "Mark McLoughlin" , "OpenStack Development Mailing
> > > List (not for usage questions)"
> > > 
> > > Sent: Tuesday, November 12, 2013 9:31:25 AM
> > > Subject: Re: [openstack-dev] [nova] Do we have some guidelines for mock,
> > > stub, mox when writing unit test?
> > > 
> > > On 11 November 2013 23:18, Mark McLoughlin  wrote:
> > > > On Mon, 2013-11-11 at 12:07 +, John Garbutt wrote:
> > > >> On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services)
> > > >>  wrote:
> > > >> > Hi
> > > >> >
> > > >> >>Generally mock is supposed to be used over mox now for python 3
> > > >> >>support.
> > > >> >
> > > >> > That is my understanding too
> > > >>
> > > >> +1
> > > >>
> > > >> But I don't think we should waste all our time re-writing all our mox
> > > >> and stub tests. Lets just leave this to happen organically for now as
> > > >> we add and refactor tests. We probably need to take the hit at some
> > > >> point, but that doesn't feel like we should do that right now.
> > > >
> > > > Hmm, I don't follow this stance.
> > > >
> > > > Adding Python3 support is a goal we all share.
> > > >
> > > > If we're saying that the use of mox stands in the way of that goal, but
> > > > that we'd really prefer if people didn't work on porting tests from mox
> > > > to mock yet ... then are we saying we don't value people working on
> > > > porting to Python3?
> > > >
> > > > And if we plan to use a Python3 compatible version of mox[1], then
> > > > isn't
> > > > the Python3 argument irrelevant and saying "use mock for new tests"
> > > > just
> > > > means we'll end up with a mixture of mox and mock?
> > > 
> > > Good point, I forgot about the port of mox to python3.
> > > 
> > > I liked the idea of "prefer mock", with a view that at some point in
> > > the future there is only a small amount of mox related code left, that
> > > can easily get moved to mock.
> > > 
> > > I guess its a trade off between review capacity, risk of breaking
> > > existing tests, and risk of never reaching that end goal.
> > > 
> > > We already have stubs and mox, which do tend to fight each other,
> > > adding a third does seem like a bad plan, unless there is a very good
> > > reason, which I always had in my head as python3 support. Hmm... I do
> > > prefer mock to mox, but not that strongly.
> > > 
> > > John
> > > 
> > > ___
> > > OpenStack-dev mailing list
> > > OpenStack-dev@lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > 
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Mark McLoughlin]

On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote:
> Maybe we should have some 60% rule... that is: If you change more than
> half of a test... you should *probably* rewrite the test in Mock.

A rule needs a reasoning attached to it :)

Why do we want people to use mock?

Is it really for Python3? If so, I assume that means we've ruled out the
python3 port of mox? (Ok by me, but would be good to hear why) And, if
that's the case, then we should encourage whoever wants to port mox
based tests to mock.

Or maybe it has nothing to do with Python3 at all? Maybe we just like
mock more? But do we like it enough to have a mixture of mock and mox
across the codebase?

Mark.

> - Original Message -
> > From: "John Garbutt" 
> > To: "Mark McLoughlin" , "OpenStack Development Mailing 
> > List (not for usage questions)"
> > 
> > Sent: Tuesday, November 12, 2013 9:31:25 AM
> > Subject: Re: [openstack-dev] [nova] Do we have some guidelines for mock, 
> > stub, mox when writing unit test?
> > 
> > On 11 November 2013 23:18, Mark McLoughlin  wrote:
> > > On Mon, 2013-11-11 at 12:07 +, John Garbutt wrote:
> > >> On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services)
> > >>  wrote:
> > >> > Hi
> > >> >
> > >> >>Generally mock is supposed to be used over mox now for python 3 
> > >> >>support.
> > >> >
> > >> > That is my understanding too
> > >>
> > >> +1
> > >>
> > >> But I don't think we should waste all our time re-writing all our mox
> > >> and stub tests. Lets just leave this to happen organically for now as
> > >> we add and refactor tests. We probably need to take the hit at some
> > >> point, but that doesn't feel like we should do that right now.
> > >
> > > Hmm, I don't follow this stance.
> > >
> > > Adding Python3 support is a goal we all share.
> > >
> > > If we're saying that the use of mox stands in the way of that goal, but
> > > that we'd really prefer if people didn't work on porting tests from mox
> > > to mock yet ... then are we saying we don't value people working on
> > > porting to Python3?
> > >
> > > And if we plan to use a Python3 compatible version of mox[1], then isn't
> > > the Python3 argument irrelevant and saying "use mock for new tests" just
> > > means we'll end up with a mixture of mox and mock?
> > 
> > Good point, I forgot about the port of mox to python3.
> > 
> > I liked the idea of "prefer mock", with a view that at some point in
> > the future there is only a small amount of mox related code left, that
> > can easily get moved to mock.
> > 
> > I guess its a trade off between review capacity, risk of breaking
> > existing tests, and risk of never reaching that end goal.
> > 
> > We already have stubs and mox, which do tend to fight each other,
> > adding a third does seem like a bad plan, unless there is a very good
> > reason, which I always had in my head as python3 support. Hmm... I do
> > prefer mock to mox, but not that strongly.
> > 
> > John
> > 
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Shawn Hartsock]

Maybe we should have some 60% rule... that is: If you change more than half of 
a test... you should *probably* rewrite the test in Mock.

# Shawn Hartsock


- Original Message -
> From: "John Garbutt" 
> To: "Mark McLoughlin" , "OpenStack Development Mailing 
> List (not for usage questions)"
> 
> Sent: Tuesday, November 12, 2013 9:31:25 AM
> Subject: Re: [openstack-dev] [nova] Do we have some guidelines for mock, 
> stub, mox when writing unit test?
> 
> On 11 November 2013 23:18, Mark McLoughlin  wrote:
> > On Mon, 2013-11-11 at 12:07 +, John Garbutt wrote:
> >> On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services)
> >>  wrote:
> >> > Hi
> >> >
> >> >>Generally mock is supposed to be used over mox now for python 3 support.
> >> >
> >> > That is my understanding too
> >>
> >> +1
> >>
> >> But I don't think we should waste all our time re-writing all our mox
> >> and stub tests. Lets just leave this to happen organically for now as
> >> we add and refactor tests. We probably need to take the hit at some
> >> point, but that doesn't feel like we should do that right now.
> >
> > Hmm, I don't follow this stance.
> >
> > Adding Python3 support is a goal we all share.
> >
> > If we're saying that the use of mox stands in the way of that goal, but
> > that we'd really prefer if people didn't work on porting tests from mox
> > to mock yet ... then are we saying we don't value people working on
> > porting to Python3?
> >
> > And if we plan to use a Python3 compatible version of mox[1], then isn't
> > the Python3 argument irrelevant and saying "use mock for new tests" just
> > means we'll end up with a mixture of mox and mock?
> 
> Good point, I forgot about the port of mox to python3.
> 
> I liked the idea of "prefer mock", with a view that at some point in
> the future there is only a small amount of mox related code left, that
> can easily get moved to mock.
> 
> I guess its a trade off between review capacity, risk of breaking
> existing tests, and risk of never reaching that end goal.
> 
> We already have stubs and mox, which do tend to fight each other,
> adding a third does seem like a bad plan, unless there is a very good
> reason, which I always had in my head as python3 support. Hmm... I do
> prefer mock to mox, but not that strongly.
> 
> John
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[John Garbutt]

On 11 November 2013 23:18, Mark McLoughlin  wrote:
> On Mon, 2013-11-11 at 12:07 +, John Garbutt wrote:
>> On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services)
>>  wrote:
>> > Hi
>> >
>> >>Generally mock is supposed to be used over mox now for python 3 support.
>> >
>> > That is my understanding too
>>
>> +1
>>
>> But I don't think we should waste all our time re-writing all our mox
>> and stub tests. Lets just leave this to happen organically for now as
>> we add and refactor tests. We probably need to take the hit at some
>> point, but that doesn't feel like we should do that right now.
>
> Hmm, I don't follow this stance.
>
> Adding Python3 support is a goal we all share.
>
> If we're saying that the use of mox stands in the way of that goal, but
> that we'd really prefer if people didn't work on porting tests from mox
> to mock yet ... then are we saying we don't value people working on
> porting to Python3?
>
> And if we plan to use a Python3 compatible version of mox[1], then isn't
> the Python3 argument irrelevant and saying "use mock for new tests" just
> means we'll end up with a mixture of mox and mock?

Good point, I forgot about the port of mox to python3.

I liked the idea of "prefer mock", with a view that at some point in
the future there is only a small amount of mox related code left, that
can easily get moved to mock.

I guess its a trade off between review capacity, risk of breaking
existing tests, and risk of never reaching that end goal.

We already have stubs and mox, which do tend to fight each other,
adding a third does seem like a bad plan, unless there is a very good
reason, which I always had in my head as python3 support. Hmm... I do
prefer mock to mox, but not that strongly.

John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Mark McLoughlin]

On Mon, 2013-11-11 at 12:07 +, John Garbutt wrote:
> On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services)
>  wrote:
> > Hi
> >
> >>Generally mock is supposed to be used over mox now for python 3 support.
> >
> > That is my understanding too
> 
> +1
> 
> But I don't think we should waste all our time re-writing all our mox
> and stub tests. Lets just leave this to happen organically for now as
> we add and refactor tests. We probably need to take the hit at some
> point, but that doesn't feel like we should do that right now.

Hmm, I don't follow this stance.

Adding Python3 support is a goal we all share.

If we're saying that the use of mox stands in the way of that goal, but
that we'd really prefer if people didn't work on porting tests from mox
to mock yet ... then are we saying we don't value people working on
porting to Python3?

And if we plan to use a Python3 compatible version of mox[1], then isn't
the Python3 argument irrelevant and saying "use mock for new tests" just
means we'll end up with a mixture of mox and mock?

Thanks,
Mark.

[1] - https://github.com/emonty/pymox


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Jay Lau]

Got it. Thanks.

Jay


2013/11/11 Davanum Srinivas 

> Feedback from some of the Nova sessions were,
>
> "If you are writing new tests, try to use mock."
> "Writing new tests to cover more code (esp drivers) is more preferable
> to any effort that just converts from mox to mock"
>
> -- dims
>
> On Sun, Nov 10, 2013 at 11:25 PM, Noorul Islam K M 
> wrote:
> > Jay Lau  writes:
> >
> >> Hi,
> >>
> >> I noticed that we are now using mock, mox and stub for unit test, just
> >> curious do we have any guidelines for this, in which condition shall we
> use
> >> mock, mox or stub?
> >>
> >
> > There is already a blueprint [1] in Nova project to replace Mox with
> mock.
> >
> > Also it has a link to ML thread [2].
> >
> > Regards,
> > Noorul
> >
> > [1] https://blueprints.launchpad.net/nova/+spec/mox-to-mock-conversion
> > [2]
> http://lists.openstack.org/pipermail/openstack-dev/2013-July/012484.html
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Davanum Srinivas :: http://davanum.wordpress.com
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[John Garbutt]

On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services)
 wrote:
> Hi
>
>>Generally mock is supposed to be used over mox now for python 3 support.
>
> That is my understanding too

+1

But I don't think we should waste all our time re-writing all our mox
and stub tests. Lets just leave this to happen organically for now as
we add and refactor tests. We probably need to take the hit at some
point, but that doesn't feel like we should do that right now.

>>As for when to use mock vs stubs, I think you'll get different opinions from
>>different people. Stubs are quick and easy and that's what I used early when I
>>started contributing to the project, but since then have preferred mox/mock
>>since they validate that methods are actually called with specific parameters,
>>which can get lost when simply stubbing a method call out. In other words, if
>>I'm stubbing a method and doing assertions within it (which you'll usually 
>>see),
>>if that method is never called (maybe the code changed since the test was
>>written), the assertions are lost and the test is essentially broken.
>>
>>So I think in general it's best to use mock now unless you have a good reason
>>not to.

+1

All new tests should use mock

> AFAIK We all agree that for mocking we move to mock but there is no 
> indication about using mock or stub, and that's good to me.
> Mocking and stubbing are two different philosophies, I don't think one is 
> better than the other.
> I found this article [1] really useful.
> HTH
> Regards
>
> [1] http://martinfowler.com/articles/mocksArentStubs.html

In mock things are stubs by default, which really handy. I like how
the behaviour verification in mock sits in the same place as the state
verification. In terms of books, I am quite a fan of the book by Roy
Osherove.

John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Rosa, Andrea (HP Cloud Services)]

Hi 

>Generally mock is supposed to be used over mox now for python 3 support.

That is my understanding too

>As for when to use mock vs stubs, I think you'll get different opinions from
>different people. Stubs are quick and easy and that's what I used early when I
>started contributing to the project, but since then have preferred mox/mock
>since they validate that methods are actually called with specific parameters,
>which can get lost when simply stubbing a method call out. In other words, if
>I'm stubbing a method and doing assertions within it (which you'll usually 
>see),
>if that method is never called (maybe the code changed since the test was
>written), the assertions are lost and the test is essentially broken.
>
>So I think in general it's best to use mock now unless you have a good reason
>not to.


AFAIK We all agree that for mocking we move to mock but there is no indication 
about using mock or stub, and that's good to me.
Mocking and stubbing are two different philosophies, I don't think one is 
better than the other.
I found this article [1] really useful.
HTH
Regards

[1] http://martinfowler.com/articles/mocksArentStubs.html
--
Andrea Rosa



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Davanum Srinivas]

Feedback from some of the Nova sessions were,

"If you are writing new tests, try to use mock."
"Writing new tests to cover more code (esp drivers) is more preferable
to any effort that just converts from mox to mock"

-- dims

On Sun, Nov 10, 2013 at 11:25 PM, Noorul Islam K M  wrote:
> Jay Lau  writes:
>
>> Hi,
>>
>> I noticed that we are now using mock, mox and stub for unit test, just
>> curious do we have any guidelines for this, in which condition shall we use
>> mock, mox or stub?
>>
>
> There is already a blueprint [1] in Nova project to replace Mox with mock.
>
> Also it has a link to ML thread [2].
>
> Regards,
> Noorul
>
> [1] https://blueprints.launchpad.net/nova/+spec/mox-to-mock-conversion
> [2] http://lists.openstack.org/pipermail/openstack-dev/2013-July/012484.html
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: http://davanum.wordpress.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Noorul Islam K M]

Jay Lau  writes:

> Hi,
>
> I noticed that we are now using mock, mox and stub for unit test, just
> curious do we have any guidelines for this, in which condition shall we use
> mock, mox or stub?
>

There is already a blueprint [1] in Nova project to replace Mox with mock.

Also it has a link to ML thread [2].

Regards,
Noorul

[1] https://blueprints.launchpad.net/nova/+spec/mox-to-mock-conversion
[2] http://lists.openstack.org/pipermail/openstack-dev/2013-July/012484.html

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Matt Riedemann]

I don't see anything explicit in the wiki and hacking guides, they 
mainly just say to have unit tests for everything and tell you how to 
run/debug them.


Generally mock is supposed to be used over mox now for python 3 support.

There is also a blueprint to remove the usage of mox in neutron:

https://blueprints.launchpad.net/neutron/+spec/remove-mox

For all new patches, we should be using mock over mox because of the 
python 3 support of mock (and lack thereof for mox).


As for when to use mock vs stubs, I think you'll get different opinions 
from different people. Stubs are quick and easy and that's what I used 
early when I started contributing to the project, but since then have 
preferred mox/mock since they validate that methods are actually called 
with specific parameters, which can get lost when simply stubbing a 
method call out. In other words, if I'm stubbing a method and doing 
assertions within it (which you'll usually see), if that method is never 
called (maybe the code changed since the test was written), the 
assertions are lost and the test is essentially broken.


So I think in general it's best to use mock now unless you have a good 
reason not to.


On 11/10/2013 7:40 AM, Jay Lau wrote:

Hi,

I noticed that we are now using mock, mox and stub for unit test, just
curious do we have any guidelines for this, in which condition shall we
use mock, mox or stub?

Thanks,

Jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--

Thanks,

Matt Riedemann


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Boris Pavlovic]

Jay,


I am not 100% sure. But personally for me mock are the best choice. 1) py3
support  2) simple for usage.


Best regards,
Boris Pavlovic
---
Mirantis Inc.



On Sun, Nov 10, 2013 at 5:40 PM, Jay Lau  wrote:

> Hi,
>
> I noticed that we are now using mock, mox and stub for unit test, just
> curious do we have any guidelines for this, in which condition shall we use
> mock, mox or stub?
>
> Thanks,
>
> Jay
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?

[Jay Lau]

Hi,

I noticed that we are now using mock, mox and stub for unit test, just
curious do we have any guidelines for this, in which condition shall we use
mock, mox or stub?

Thanks,

Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev