Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 2013-12-21 07:24, Matt Riedemann wrote: On 12/19/2013 8:51 AM, John Garbutt wrote: On 4 December 2013 17:10, Russell Bryant rbry...@redhat.com wrote: I think option 3 makes the most sense here (pending anyone saying we should run away screaming from mox3 for some reason). It's actually what I had been assuming since this thread a while back. This means that we don't need to *require* that tests get converted if you're changing one. It just gets you bonus imaginary internet points. Requiring mock for new tests seems fine. We can grant exceptions in specific cases if necessary. In general, we should be using mock for new tests. I have lost track a bit here. The above seems like a sane approach. Do we all agree on that now? Can we add the above text into here: https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Yeah, at some point I wanted to cleanup the various testing guides but until then I like the idea of just putting something simple into the nova review checklist. Basically use mock for new tests, mox can be used in exceptional cases. What I've considered exceptional so far includes changes that will be backported to a stable release where mock isn't being used and cases where you basically have to bend over backwards to work new mock tests into an existing test class that has lots of existing setUp with mox. However, even in the latter case you can usually use mock after resetting the mox setup via self.mox.ResetAll() in the new test case(s). I went ahead and added this to the wiki, so it's now an absolutely inviolate policy. Unless, ya know, someone edits the wiki after me. ;-) Also, I put it in the common section because this doesn't seem like something we should be doing differently per-project. If anyone objects, feel free to add to the discussion here as to why. Thanks. -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On Thu, Dec 5, 2013 at 8:38 PM, Jay Pipes jaypi...@gmail.com wrote: On 12/04/2013 12:10 PM, Russell Bryant wrote: On 12/04/2013 11:16 AM, Nikola Đipanov wrote: Resurrecting this thread because of an interesting review that came up yesterday [1]. It seems that our lack of a firm decision on what to do with the mocking framework has left people confused. In hope to help - I'll give my view of where things are now and what we should do going forward, and hopefully we'll reach some consensus on this. Here's the breakdown: We should abandon mox: * It has not had a release in over 3 years [2] and a patch upstream for 2 * There are bugs that are impacting the project with it (see above) * It will not be ported to python 3 Proposed path forward options: 1) Port nova to mock now: * Literally unmanageable - huge review overhead and regression risk for not so much gain (maybe) [1] 2) Opportunistically port nova (write new tests using mock, when fixing tests, move them to mock): * Will take a really long time to move to mock, and is not really a solution since we are stuck with mock for an undetermined period of time - it's what we are doing now (kind of). 3) Same as 2) but move current codebase to mox3 * Buys us py3k compat, and fresher code * Mox3 and mox have diverged and we would need to backport mox fixes onto the mox3 three and become de-facto active maintainers (as per Peter Feiner's last email - that may not be so easy). I think we should follow path 3) if we can, but we need to: 1) Figure out what is the deal with mox3 and decide if owning it will really be less trouble than porting nova. To be hones - I was unable to even find the code repo for it, only [3]. If anyone has more info - please weigh in. We'll also need volunteers 2) Make better testing guidelines when using mock, and maybe add some testing helpers (like we do already have for mox) that will make porting existing tests easier. mreidem already put this on this weeks nova meeting agenda - so that might be a good place to discuss all the issues mentioned here as well. We should really take a stronger stance on this soon IMHO, as this comes up with literally every commit. I think option 3 makes the most sense here (pending anyone saying we should run away screaming from mox3 for some reason). It's actually what I had been assuming since this thread a while back. What precisely is the benefit of moving the existing code to mox3 versus moving the existing code to mock? Is mox3 so similar to mox that the transition would be minimal? This means that we don't need to *require* that tests get converted if you're changing one. It just gets you bonus imaginary internet points. Requiring mock for new tests seems fine. We can grant exceptions in specific cases if necessary. In general, we should be using mock for new tests. My vote would be to use mock for everything new (no brainer), keep old mox stuff around and slowly port it to mock. I see little value in bringing in another mox3 library, especially if we'd end up having to maintain it. FWIW this is exactly what the Cinder team agreed upon a while back and the direction we've been going. There hasn't really been any push-back on this and in most cases the response from people has been Wow, using mock was so much easier/straight forward. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
I added it to the page John linked earlier: https://wiki.openstack.org/wiki/ReviewChecklist -Ben On 2013-12-23 17:08, Shawn Hartsock wrote: Where in the wiki is this written down? Maybe I should read some of these entries. I have looked but I can't find it. On Dec 23, 2013 11:56 AM, Ben Nemec openst...@nemebean.com wrote: On 2013-12-21 07:24, Matt Riedemann wrote: On 12/19/2013 8:51 AM, John Garbutt wrote: On 4 December 2013 17:10, Russell Bryant rbry...@redhat.com wrote: I think option 3 makes the most sense here (pending anyone saying we should run away screaming from mox3 for some reason). It's actually what I had been assuming since this thread a while back. This means that we don't need to *require* that tests get converted if you're changing one. It just gets you bonus imaginary internet points. Requiring mock for new tests seems fine. We can grant exceptions in specific cases if necessary. In general, we should be using mock for new tests. I have lost track a bit here. The above seems like a sane approach. Do we all agree on that now? Can we add the above text into here: https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist [1] John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev [2] Yeah, at some point I wanted to cleanup the various testing guides but until then I like the idea of just putting something simple into the nova review checklist. Basically use mock for new tests, mox can be used in exceptional cases. What I've considered exceptional so far includes changes that will be backported to a stable release where mock isn't being used and cases where you basically have to bend over backwards to work new mock tests into an existing test class that has lots of existing setUp with mox. However, even in the latter case you can usually use mock after resetting the mox setup via self.mox.ResetAll() in the new test case(s). I went ahead and added this to the wiki, so it's now an absolutely inviolate policy. Unless, ya know, someone edits the wiki after me. ;-) Also, I put it in the common section because this doesn't seem like something we should be doing differently per-project. If anyone objects, feel free to add to the discussion here as to why. Thanks. -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev [2] Links: -- [1] https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist [2] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 4 December 2013 17:10, Russell Bryant rbry...@redhat.com wrote: I think option 3 makes the most sense here (pending anyone saying we should run away screaming from mox3 for some reason). It's actually what I had been assuming since this thread a while back. This means that we don't need to *require* that tests get converted if you're changing one. It just gets you bonus imaginary internet points. Requiring mock for new tests seems fine. We can grant exceptions in specific cases if necessary. In general, we should be using mock for new tests. I have lost track a bit here. The above seems like a sane approach. Do we all agree on that now? Can we add the above text into here: https://wiki.openstack.org/wiki/ReviewChecklist#Nova_Review_Checklist John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 2013-12-05 21:38, Jay Pipes wrote: On 12/04/2013 12:10 PM, Russell Bryant wrote: On 12/04/2013 11:16 AM, Nikola Đipanov wrote: Resurrecting this thread because of an interesting review that came up yesterday [1]. It seems that our lack of a firm decision on what to do with the mocking framework has left people confused. In hope to help - I'll give my view of where things are now and what we should do going forward, and hopefully we'll reach some consensus on this. Here's the breakdown: We should abandon mox: * It has not had a release in over 3 years [2] and a patch upstream for 2 * There are bugs that are impacting the project with it (see above) * It will not be ported to python 3 Proposed path forward options: 1) Port nova to mock now: * Literally unmanageable - huge review overhead and regression risk for not so much gain (maybe) [1] 2) Opportunistically port nova (write new tests using mock, when fixing tests, move them to mock): * Will take a really long time to move to mock, and is not really a solution since we are stuck with mock for an undetermined period of time - it's what we are doing now (kind of). 3) Same as 2) but move current codebase to mox3 * Buys us py3k compat, and fresher code * Mox3 and mox have diverged and we would need to backport mox fixes onto the mox3 three and become de-facto active maintainers (as per Peter Feiner's last email - that may not be so easy). I think we should follow path 3) if we can, but we need to: 1) Figure out what is the deal with mox3 and decide if owning it will really be less trouble than porting nova. To be hones - I was unable to even find the code repo for it, only [3]. If anyone has more info - please weigh in. We'll also need volunteers 2) Make better testing guidelines when using mock, and maybe add some testing helpers (like we do already have for mox) that will make porting existing tests easier. mreidem already put this on this weeks nova meeting agenda - so that might be a good place to discuss all the issues mentioned here as well. We should really take a stronger stance on this soon IMHO, as this comes up with literally every commit. I think option 3 makes the most sense here (pending anyone saying we should run away screaming from mox3 for some reason). It's actually what I had been assuming since this thread a while back. What precisely is the benefit of moving the existing code to mox3 versus moving the existing code to mock? Is mox3 so similar to mox that the transition would be minimal? This means that we don't need to *require* that tests get converted if you're changing one. It just gets you bonus imaginary internet points. Requiring mock for new tests seems fine. We can grant exceptions in specific cases if necessary. In general, we should be using mock for new tests. My vote would be to use mock for everything new (no brainer), keep old mox stuff around and slowly port it to mock. I see little value in bringing in another mox3 library, especially if we'd end up having to maintain it. My understanding is that mox3 is a drop-in, Python 3 compatible version of mox. I agree that spending any significant time maintaining mox3 is a bad thing at this point. Mock is part of the stdlib in Python 3 and I don't think we should put a lot of time into reinventing the wheel. That said, as long as mox3 works right now I don't think we should be rewriting mox test cases just to move them to mock either. That's a whole lot of code churn for basically no benefit. So my preference would be to: 1) Use mock for new test cases, with possible exceptions for adding to test classes that already use mox 2) Leave the existing mox test cases alone as long as they work fine with mox3. 3) If any test cases don't work in mox3, rewrite them in mock -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 12/04/2013 12:10 PM, Russell Bryant wrote: On 12/04/2013 11:16 AM, Nikola Đipanov wrote: Resurrecting this thread because of an interesting review that came up yesterday [1]. It seems that our lack of a firm decision on what to do with the mocking framework has left people confused. In hope to help - I'll give my view of where things are now and what we should do going forward, and hopefully we'll reach some consensus on this. Here's the breakdown: We should abandon mox: * It has not had a release in over 3 years [2] and a patch upstream for 2 * There are bugs that are impacting the project with it (see above) * It will not be ported to python 3 Proposed path forward options: 1) Port nova to mock now: * Literally unmanageable - huge review overhead and regression risk for not so much gain (maybe) [1] 2) Opportunistically port nova (write new tests using mock, when fixing tests, move them to mock): * Will take a really long time to move to mock, and is not really a solution since we are stuck with mock for an undetermined period of time - it's what we are doing now (kind of). 3) Same as 2) but move current codebase to mox3 * Buys us py3k compat, and fresher code * Mox3 and mox have diverged and we would need to backport mox fixes onto the mox3 three and become de-facto active maintainers (as per Peter Feiner's last email - that may not be so easy). I think we should follow path 3) if we can, but we need to: 1) Figure out what is the deal with mox3 and decide if owning it will really be less trouble than porting nova. To be hones - I was unable to even find the code repo for it, only [3]. If anyone has more info - please weigh in. We'll also need volunteers 2) Make better testing guidelines when using mock, and maybe add some testing helpers (like we do already have for mox) that will make porting existing tests easier. mreidem already put this on this weeks nova meeting agenda - so that might be a good place to discuss all the issues mentioned here as well. We should really take a stronger stance on this soon IMHO, as this comes up with literally every commit. I think option 3 makes the most sense here (pending anyone saying we should run away screaming from mox3 for some reason). It's actually what I had been assuming since this thread a while back. What precisely is the benefit of moving the existing code to mox3 versus moving the existing code to mock? Is mox3 so similar to mox that the transition would be minimal? This means that we don't need to *require* that tests get converted if you're changing one. It just gets you bonus imaginary internet points. Requiring mock for new tests seems fine. We can grant exceptions in specific cases if necessary. In general, we should be using mock for new tests. My vote would be to use mock for everything new (no brainer), keep old mox stuff around and slowly port it to mock. I see little value in bringing in another mox3 library, especially if we'd end up having to maintain it. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 11/19/2013 05:52 PM, Peter Feiner wrote: On Tue, Nov 19, 2013 at 11:19 AM, Chuck Short chuck.sh...@canonical.com wrote: Hi On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner pe...@gridcentric.ca wrote: A substantive reason for switching from mox to mock is the derelict state of mox releases. There hasn't been a release of mox in three years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover, in the past 3 years, substantial bugs have been fixed in upstream mox. For example, with the year-old fix to https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug in nova would have been caught by an existing test [3]. Alternatively, a copy of the upstream mox code could be added in-tree. Please no, I think we are in an agreement with mox3 and mock. That's cool. As long as the mox* is phased out, the false-positive test results will be fixed. Of course, there's _another_ alternative, which is to retrofit mox3 with the upstream mox fixes (e.g., the bug I cited above exists in mox3). However, the delta between mox3 and upstream mox is pretty huge (I just checked), so effort is probably better spent switching to mock. To that end, I plan on changing the tests I cited above. Resurrecting this thread because of an interesting review that came up yesterday [1]. It seems that our lack of a firm decision on what to do with the mocking framework has left people confused. In hope to help - I'll give my view of where things are now and what we should do going forward, and hopefully we'll reach some consensus on this. Here's the breakdown: We should abandon mox: * It has not had a release in over 3 years [2] and a patch upstream for 2 * There are bugs that are impacting the project with it (see above) * It will not be ported to python 3 Proposed path forward options: 1) Port nova to mock now: * Literally unmanageable - huge review overhead and regression risk for not so much gain (maybe) [1] 2) Opportunistically port nova (write new tests using mock, when fixing tests, move them to mock): * Will take a really long time to move to mock, and is not really a solution since we are stuck with mock for an undetermined period of time - it's what we are doing now (kind of). 3) Same as 2) but move current codebase to mox3 * Buys us py3k compat, and fresher code * Mox3 and mox have diverged and we would need to backport mox fixes onto the mox3 three and become de-facto active maintainers (as per Peter Feiner's last email - that may not be so easy). I think we should follow path 3) if we can, but we need to: 1) Figure out what is the deal with mox3 and decide if owning it will really be less trouble than porting nova. To be hones - I was unable to even find the code repo for it, only [3]. If anyone has more info - please weigh in. We'll also need volunteers 2) Make better testing guidelines when using mock, and maybe add some testing helpers (like we do already have for mox) that will make porting existing tests easier. mreidem already put this on this weeks nova meeting agenda - so that might be a good place to discuss all the issues mentioned here as well. We should really take a stronger stance on this soon IMHO, as this comes up with literally every commit. Cheers, Nikola [1] https://review.openstack.org/#/c/59694/ [2] https://pypi.python.org/pypi/mox [3] https://pypi.python.org/pypi/mox3/0.7.0 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 12/04/2013 11:16 AM, Nikola Đipanov wrote: Resurrecting this thread because of an interesting review that came up yesterday [1]. It seems that our lack of a firm decision on what to do with the mocking framework has left people confused. In hope to help - I'll give my view of where things are now and what we should do going forward, and hopefully we'll reach some consensus on this. Here's the breakdown: We should abandon mox: * It has not had a release in over 3 years [2] and a patch upstream for 2 * There are bugs that are impacting the project with it (see above) * It will not be ported to python 3 Proposed path forward options: 1) Port nova to mock now: * Literally unmanageable - huge review overhead and regression risk for not so much gain (maybe) [1] 2) Opportunistically port nova (write new tests using mock, when fixing tests, move them to mock): * Will take a really long time to move to mock, and is not really a solution since we are stuck with mock for an undetermined period of time - it's what we are doing now (kind of). 3) Same as 2) but move current codebase to mox3 * Buys us py3k compat, and fresher code * Mox3 and mox have diverged and we would need to backport mox fixes onto the mox3 three and become de-facto active maintainers (as per Peter Feiner's last email - that may not be so easy). I think we should follow path 3) if we can, but we need to: 1) Figure out what is the deal with mox3 and decide if owning it will really be less trouble than porting nova. To be hones - I was unable to even find the code repo for it, only [3]. If anyone has more info - please weigh in. We'll also need volunteers 2) Make better testing guidelines when using mock, and maybe add some testing helpers (like we do already have for mox) that will make porting existing tests easier. mreidem already put this on this weeks nova meeting agenda - so that might be a good place to discuss all the issues mentioned here as well. We should really take a stronger stance on this soon IMHO, as this comes up with literally every commit. I think option 3 makes the most sense here (pending anyone saying we should run away screaming from mox3 for some reason). It's actually what I had been assuming since this thread a while back. This means that we don't need to *require* that tests get converted if you're changing one. It just gets you bonus imaginary internet points. Requiring mock for new tests seems fine. We can grant exceptions in specific cases if necessary. In general, we should be using mock for new tests. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On Wed, Dec 4, 2013 at 11:16 AM, Nikola Đipanov ndipa...@redhat.com wrote: 1) Figure out what is the deal with mox3 and decide if owning it will really be less trouble than porting nova. To be hones - I was unable to even find the code repo for it, only [3]. If anyone has more info - please weigh in. We'll also need volunteers [3] https://pypi.python.org/pypi/mox3/0.7.0 That's all I was able to find. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 12/04/2013 06:15 PM, Peter Feiner wrote: On Wed, Dec 4, 2013 at 11:16 AM, Nikola Đipanov ndipa...@redhat.com wrote: 1) Figure out what is the deal with mox3 and decide if owning it will really be less trouble than porting nova. To be hones - I was unable to even find the code repo for it, only [3]. If anyone has more info - please weigh in. We'll also need volunteers [3] https://pypi.python.org/pypi/mox3/0.7.0 That's all I was able to find. The package seems to be owned by people from the community - so maybe someone will respond on this thread. Or if anyone knows who is behind the package - let us know - it would be good to start figuring this out sooner rather than later. N. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On Wed, Dec 4, 2013 at 11:16 AM, Nikola Đipanov ndipa...@redhat.com wrote: On 11/19/2013 05:52 PM, Peter Feiner wrote: On Tue, Nov 19, 2013 at 11:19 AM, Chuck Short chuck.sh...@canonical.com wrote: Hi On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner pe...@gridcentric.ca wrote: A substantive reason for switching from mox to mock is the derelict state of mox releases. There hasn't been a release of mox in three years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover, in the past 3 years, substantial bugs have been fixed in upstream mox. For example, with the year-old fix to https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug in nova would have been caught by an existing test [3]. Alternatively, a copy of the upstream mox code could be added in-tree. Please no, I think we are in an agreement with mox3 and mock. That's cool. As long as the mox* is phased out, the false-positive test results will be fixed. Of course, there's _another_ alternative, which is to retrofit mox3 with the upstream mox fixes (e.g., the bug I cited above exists in mox3). However, the delta between mox3 and upstream mox is pretty huge (I just checked), so effort is probably better spent switching to mock. To that end, I plan on changing the tests I cited above. Resurrecting this thread because of an interesting review that came up yesterday [1]. It seems that our lack of a firm decision on what to do with the mocking framework has left people confused. In hope to help - I'll give my view of where things are now and what we should do going forward, and hopefully we'll reach some consensus on this. Here's the breakdown: We should abandon mox: * It has not had a release in over 3 years [2] and a patch upstream for 2 * There are bugs that are impacting the project with it (see above) * It will not be ported to python 3 Proposed path forward options: 1) Port nova to mock now: * Literally unmanageable - huge review overhead and regression risk for not so much gain (maybe) [1] 2) Opportunistically port nova (write new tests using mock, when fixing tests, move them to mock): * Will take a really long time to move to mock, and is not really a solution since we are stuck with mock for an undetermined period of time - it's what we are doing now (kind of). 3) Same as 2) but move current codebase to mox3 * Buys us py3k compat, and fresher code * Mox3 and mox have diverged and we would need to backport mox fixes onto the mox3 three and become de-facto active maintainers (as per Peter Feiner's last email - that may not be so easy). So I thought we cleared this up already. We convert the current codebase over to mox3, new tests should be done in mock. Eventually we start converting over code to use mock. I think we should follow path 3) if we can, but we need to: 1) Figure out what is the deal with mox3 and decide if owning it will really be less trouble than porting nova. To be hones - I was unable to even find the code repo for it, only [3]. If anyone has more info - please weigh in. We'll also need volunteers Monty and I did this last cycle, its apart of the openstack project, although its not available in gerrit. Which should be fixed so we can start getting bug fixes in for it. 2) Make better testing guidelines when using mock, and maybe add some testing helpers (like we do already have for mox) that will make porting existing tests easier. mreidem already put this on this weeks nova meeting agenda - so that might be a good place to discuss all the issues mentioned here as well. We should really take a stronger stance on this soon IMHO, as this comes up with literally every commit. totally Cheers, Nikola [1] https://review.openstack.org/#/c/59694/ [2] https://pypi.python.org/pypi/mox [3] https://pypi.python.org/pypi/mox3/0.7.0 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 12/4/2013 10:16 AM, Nikola Đipanov wrote: On 11/19/2013 05:52 PM, Peter Feiner wrote: On Tue, Nov 19, 2013 at 11:19 AM, Chuck Short chuck.sh...@canonical.com wrote: Hi On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner pe...@gridcentric.ca wrote: A substantive reason for switching from mox to mock is the derelict state of mox releases. There hasn't been a release of mox in three years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover, in the past 3 years, substantial bugs have been fixed in upstream mox. For example, with the year-old fix to https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug in nova would have been caught by an existing test [3]. Alternatively, a copy of the upstream mox code could be added in-tree. Please no, I think we are in an agreement with mox3 and mock. That's cool. As long as the mox* is phased out, the false-positive test results will be fixed. Of course, there's _another_ alternative, which is to retrofit mox3 with the upstream mox fixes (e.g., the bug I cited above exists in mox3). However, the delta between mox3 and upstream mox is pretty huge (I just checked), so effort is probably better spent switching to mock. To that end, I plan on changing the tests I cited above. Resurrecting this thread because of an interesting review that came up yesterday [1]. It seems that our lack of a firm decision on what to do with the mocking framework has left people confused. In hope to help - I'll give my view of where things are now and what we should do going forward, and hopefully we'll reach some consensus on this. Here's the breakdown: We should abandon mox: * It has not had a release in over 3 years [2] and a patch upstream for 2 * There are bugs that are impacting the project with it (see above) * It will not be ported to python 3 Proposed path forward options: 1) Port nova to mock now: * Literally unmanageable - huge review overhead and regression risk for not so much gain (maybe) [1] 2) Opportunistically port nova (write new tests using mock, when fixing tests, move them to mock): * Will take a really long time to move to mock, and is not really a solution since we are stuck with mock for an undetermined period of time - it's what we are doing now (kind of). 3) Same as 2) but move current codebase to mox3 * Buys us py3k compat, and fresher code * Mox3 and mox have diverged and we would need to backport mox fixes onto the mox3 three and become de-facto active maintainers (as per Peter Feiner's last email - that may not be so easy). I think we should follow path 3) if we can, but we need to: 1) Figure out what is the deal with mox3 and decide if owning it will really be less trouble than porting nova. To be hones - I was unable to even find the code repo for it, only [3]. If anyone has more info - please weigh in. We'll also need volunteers 2) Make better testing guidelines when using mock, and maybe add some testing helpers (like we do already have for mox) that will make porting existing tests easier. mreidem already put this on this weeks nova meeting agenda - so that might be a good place to discuss all the issues mentioned here as well. For anyone interested in seeing the nova meeting agenda notes before the actual meeting (since it covers more than just mock/mox), here you go: https://wiki.openstack.org/wiki/Meetings/Nova Copying here for reference later if necessary: - Testing Guides - we need some guide on using mox vs mock; previously we were forcing all new tests to use mock but then mox3 was pointed out and that seemed to go away, but is anyone using mox3 yet (ceilomerclient, heatclient and ironicclient are all moving to mox3). - ML on stubs vs mox vs mock: http://lists.openstack.org/pipermail/openstack-dev/2013-November/018501.html - nova/tests/README.rst is outdated; lots of TBD and last update was on 2013/05/16: https://github.com/openstack/nova/blob/master/nova/tests/README.rst - Note keystone bug 1252454 where the keystone docs were referencing the nova testing README - Ultimately it seems the best guide is Horizon's: http://docs.openstack.org/developer/horizon/topics/testing.html - We should have something like the Horizon guide in the overall OpenStack hacking guide so the projects can point to a single location, http://docs.openstack.org/developer/hacking/, but even that is sparse on details for writing unit tests (should almost merge or point to the horizon guide), and then the single guide should have notes on stubs vs mox vs mock with pros/cons and best practices / pitfalls. === The net problem is we have a lot of various testing guides throughout the projects, including hacking guides, readmes, wikis, and devref pages, we should really consolidate that into the universal hacking guide (IMO) and then integrate Horizon's excellent guide into that along with whatever rules we have for mox/mox3/mock. We should really take a stronger stance on
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
A substantive reason for switching from mox to mock is the derelict state of mox releases. There hasn't been a release of mox in three years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover, in the past 3 years, substantial bugs have been fixed in upstream mox. For example, with the year-old fix to https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug in nova would have been caught by an existing test [3]. Alternatively, a copy of the upstream mox code could be added in-tree. [1] mox releases: https://code.google.com/p/pymox/downloads/list [2] mox on pypi: https://pypi.python.org/pypi/mox [3] see comments 5 and 6 in https://bugs.launchpad.net/nova/+bug/1251792 On Wed, Nov 13, 2013 at 2:24 PM, Matt Riedemann mrie...@linux.vnet.ibm.com wrote: On 11/12/2013 5:04 PM, Chuck Short wrote: On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin mar...@redhat.com mailto:mar...@redhat.com wrote: On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote: Hi On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin mar...@redhat.com mailto:mar...@redhat.com wrote: On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote: Maybe we should have some 60% rule... that is: If you change more than half of a test... you should *probably* rewrite the test in Mock. A rule needs a reasoning attached to it :) Why do we want people to use mock? Is it really for Python3? If so, I assume that means we've ruled out the python3 port of mox? (Ok by me, but would be good to hear why) And, if that's the case, then we should encourage whoever wants to port mox based tests to mock. The upstream maintainer is not going to port mox to python3 so we have a fork of mox called mox3. Ideally, we would drop the usage of mox in favour of mock so we don't have to carry a forked mox. Isn't that the opposite conclusion you came to here: http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html i.e. using mox3 results in less code churn? Mark. Yes that was my original position but I though we agreed in thread (further on) that we would use mox3 and then migrate to mock further on. Regards chuck ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev So it sounds like we're good with using mox for new tests again? Given Chuck got it into global-requirements here: https://github.com/openstack/requirements/commit/998dda263d7c7881070e3f16e4523ddcd23fc36d We can stave off the need to transition everything from mox to mock? I can't seem to find the nova blueprint to convert everything from mox to mock, maybe it was obsoleted already. Anyway, if mox(3) is OK and we don't need to use mock, it seems like we could add something to the developer guide here because I think this question comes up frequently: http://docs.openstack.org/developer/nova/devref/unit_tests.html Does anyone disagree? BTW, I care about this because I've been keeping in mind the mox/mock transition when doing code reviews and giving a -1 when new tests are using mox (since I thought that was a no-no now). -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
Hi On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner pe...@gridcentric.ca wrote: A substantive reason for switching from mox to mock is the derelict state of mox releases. There hasn't been a release of mox in three years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover, in the past 3 years, substantial bugs have been fixed in upstream mox. For example, with the year-old fix to https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug in nova would have been caught by an existing test [3]. Alternatively, a copy of the upstream mox code could be added in-tree. Please no, I think we are in an agreement with mox3 and mock. [1] mox releases: https://code.google.com/p/pymox/downloads/list [2] mox on pypi: https://pypi.python.org/pypi/mox [3] see comments 5 and 6 in https://bugs.launchpad.net/nova/+bug/1251792 On Wed, Nov 13, 2013 at 2:24 PM, Matt Riedemann mrie...@linux.vnet.ibm.com wrote: On 11/12/2013 5:04 PM, Chuck Short wrote: On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin mar...@redhat.com mailto:mar...@redhat.com wrote: On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote: Hi On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin mar...@redhat.com mailto:mar...@redhat.com wrote: On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote: Maybe we should have some 60% rule... that is: If you change more than half of a test... you should *probably* rewrite the test in Mock. A rule needs a reasoning attached to it :) Why do we want people to use mock? Is it really for Python3? If so, I assume that means we've ruled out the python3 port of mox? (Ok by me, but would be good to hear why) And, if that's the case, then we should encourage whoever wants to port mox based tests to mock. The upstream maintainer is not going to port mox to python3 so we have a fork of mox called mox3. Ideally, we would drop the usage of mox in favour of mock so we don't have to carry a forked mox. Isn't that the opposite conclusion you came to here: http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html i.e. using mox3 results in less code churn? Mark. Yes that was my original position but I though we agreed in thread (further on) that we would use mox3 and then migrate to mock further on. Regards chuck ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev So it sounds like we're good with using mox for new tests again? Given Chuck got it into global-requirements here: https://github.com/openstack/requirements/commit/998dda263d7c7881070e3f16e4523ddcd23fc36d We can stave off the need to transition everything from mox to mock? I can't seem to find the nova blueprint to convert everything from mox to mock, maybe it was obsoleted already. Anyway, if mox(3) is OK and we don't need to use mock, it seems like we could add something to the developer guide here because I think this question comes up frequently: http://docs.openstack.org/developer/nova/devref/unit_tests.html Does anyone disagree? BTW, I care about this because I've been keeping in mind the mox/mock transition when doing code reviews and giving a -1 when new tests are using mox (since I thought that was a no-no now). -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On Tue, Nov 19, 2013 at 11:19 AM, Chuck Short chuck.sh...@canonical.com wrote: Hi On Tue, Nov 19, 2013 at 10:43 AM, Peter Feiner pe...@gridcentric.ca wrote: A substantive reason for switching from mox to mock is the derelict state of mox releases. There hasn't been a release of mox in three years: the latest, mox-0.5.3, was released in 2010 [1, 2]. Moreover, in the past 3 years, substantial bugs have been fixed in upstream mox. For example, with the year-old fix to https://code.google.com/p/pymox/issues/detail?id=16, a very nasty bug in nova would have been caught by an existing test [3]. Alternatively, a copy of the upstream mox code could be added in-tree. Please no, I think we are in an agreement with mox3 and mock. That's cool. As long as the mox* is phased out, the false-positive test results will be fixed. Of course, there's _another_ alternative, which is to retrofit mox3 with the upstream mox fixes (e.g., the bug I cited above exists in mox3). However, the delta between mox3 and upstream mox is pretty huge (I just checked), so effort is probably better spent switching to mock. To that end, I plan on changing the tests I cited above. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 11/12/2013 5:04 PM, Chuck Short wrote: On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin mar...@redhat.com mailto:mar...@redhat.com wrote: On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote: Hi On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin mar...@redhat.com mailto:mar...@redhat.com wrote: On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote: Maybe we should have some 60% rule... that is: If you change more than half of a test... you should *probably* rewrite the test in Mock. A rule needs a reasoning attached to it :) Why do we want people to use mock? Is it really for Python3? If so, I assume that means we've ruled out the python3 port of mox? (Ok by me, but would be good to hear why) And, if that's the case, then we should encourage whoever wants to port mox based tests to mock. The upstream maintainer is not going to port mox to python3 so we have a fork of mox called mox3. Ideally, we would drop the usage of mox in favour of mock so we don't have to carry a forked mox. Isn't that the opposite conclusion you came to here: http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html i.e. using mox3 results in less code churn? Mark. Yes that was my original position but I though we agreed in thread (further on) that we would use mox3 and then migrate to mock further on. Regards chuck ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev So it sounds like we're good with using mox for new tests again? Given Chuck got it into global-requirements here: https://github.com/openstack/requirements/commit/998dda263d7c7881070e3f16e4523ddcd23fc36d We can stave off the need to transition everything from mox to mock? I can't seem to find the nova blueprint to convert everything from mox to mock, maybe it was obsoleted already. Anyway, if mox(3) is OK and we don't need to use mock, it seems like we could add something to the developer guide here because I think this question comes up frequently: http://docs.openstack.org/developer/nova/devref/unit_tests.html Does anyone disagree? BTW, I care about this because I've been keeping in mind the mox/mock transition when doing code reviews and giving a -1 when new tests are using mox (since I thought that was a no-no now). -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
Hi On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin mar...@redhat.com wrote: On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote: Maybe we should have some 60% rule... that is: If you change more than half of a test... you should *probably* rewrite the test in Mock. A rule needs a reasoning attached to it :) Why do we want people to use mock? Is it really for Python3? If so, I assume that means we've ruled out the python3 port of mox? (Ok by me, but would be good to hear why) And, if that's the case, then we should encourage whoever wants to port mox based tests to mock. The upstream maintainer is not going to port mox to python3 so we have a fork of mox called mox3. Ideally, we would drop the usage of mox in favour of mock so we don't have to carry a forked mox. Or maybe it has nothing to do with Python3 at all? Maybe we just like mock more? But do we like it enough to have a mixture of mock and mox across the codebase? Mark. - Original Message - From: John Garbutt j...@johngarbutt.com To: Mark McLoughlin mar...@redhat.com, OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Tuesday, November 12, 2013 9:31:25 AM Subject: Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test? On 11 November 2013 23:18, Mark McLoughlin mar...@redhat.com wrote: On Mon, 2013-11-11 at 12:07 +, John Garbutt wrote: On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services) andrea.r...@hp.com wrote: Hi Generally mock is supposed to be used over mox now for python 3 support. That is my understanding too +1 But I don't think we should waste all our time re-writing all our mox and stub tests. Lets just leave this to happen organically for now as we add and refactor tests. We probably need to take the hit at some point, but that doesn't feel like we should do that right now. Hmm, I don't follow this stance. Adding Python3 support is a goal we all share. If we're saying that the use of mox stands in the way of that goal, but that we'd really prefer if people didn't work on porting tests from mox to mock yet ... then are we saying we don't value people working on porting to Python3? And if we plan to use a Python3 compatible version of mox[1], then isn't the Python3 argument irrelevant and saying use mock for new tests just means we'll end up with a mixture of mox and mock? Good point, I forgot about the port of mox to python3. I liked the idea of prefer mock, with a view that at some point in the future there is only a small amount of mox related code left, that can easily get moved to mock. I guess its a trade off between review capacity, risk of breaking existing tests, and risk of never reaching that end goal. We already have stubs and mox, which do tend to fight each other, adding a third does seem like a bad plan, unless there is a very good reason, which I always had in my head as python3 support. Hmm... I do prefer mock to mox, but not that strongly. John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 2013-11-12 15:27, Shawn Hartsock wrote: Good point. I assume someone made a comparison similar to this: * http://garybernhardt.github.io/python-mock-comparison/ ... and evangelized a choice. I had assumed that Mock vs mox was not merely based on Python3 support but had something to do with Mock versus Mox. Does anyone have that context in their head? Would they mind sharing it? Not in my head, but the internet remembers everything :-) http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html In short, the majority of respondents liked Mock better, it seems to be more widely used, and Mock is part of the stdlib in Python 3.3. -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On Tue, Nov 12, 2013 at 4:49 PM, Mark McLoughlin mar...@redhat.com wrote: On Tue, 2013-11-12 at 16:42 -0500, Chuck Short wrote: Hi On Tue, Nov 12, 2013 at 4:24 PM, Mark McLoughlin mar...@redhat.com wrote: On Tue, 2013-11-12 at 13:11 -0800, Shawn Hartsock wrote: Maybe we should have some 60% rule... that is: If you change more than half of a test... you should *probably* rewrite the test in Mock. A rule needs a reasoning attached to it :) Why do we want people to use mock? Is it really for Python3? If so, I assume that means we've ruled out the python3 port of mox? (Ok by me, but would be good to hear why) And, if that's the case, then we should encourage whoever wants to port mox based tests to mock. The upstream maintainer is not going to port mox to python3 so we have a fork of mox called mox3. Ideally, we would drop the usage of mox in favour of mock so we don't have to carry a forked mox. Isn't that the opposite conclusion you came to here: http://lists.openstack.org/pipermail/openstack-dev/2013-July/012474.html i.e. using mox3 results in less code churn? Mark. Yes that was my original position but I though we agreed in thread (further on) that we would use mox3 and then migrate to mock further on. Regards chuck ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
Hi Generally mock is supposed to be used over mox now for python 3 support. That is my understanding too As for when to use mock vs stubs, I think you'll get different opinions from different people. Stubs are quick and easy and that's what I used early when I started contributing to the project, but since then have preferred mox/mock since they validate that methods are actually called with specific parameters, which can get lost when simply stubbing a method call out. In other words, if I'm stubbing a method and doing assertions within it (which you'll usually see), if that method is never called (maybe the code changed since the test was written), the assertions are lost and the test is essentially broken. So I think in general it's best to use mock now unless you have a good reason not to. AFAIK We all agree that for mocking we move to mock but there is no indication about using mock or stub, and that's good to me. Mocking and stubbing are two different philosophies, I don't think one is better than the other. I found this article [1] really useful. HTH Regards [1] http://martinfowler.com/articles/mocksArentStubs.html -- Andrea Rosa ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services) andrea.r...@hp.com wrote: Hi Generally mock is supposed to be used over mox now for python 3 support. That is my understanding too +1 But I don't think we should waste all our time re-writing all our mox and stub tests. Lets just leave this to happen organically for now as we add and refactor tests. We probably need to take the hit at some point, but that doesn't feel like we should do that right now. As for when to use mock vs stubs, I think you'll get different opinions from different people. Stubs are quick and easy and that's what I used early when I started contributing to the project, but since then have preferred mox/mock since they validate that methods are actually called with specific parameters, which can get lost when simply stubbing a method call out. In other words, if I'm stubbing a method and doing assertions within it (which you'll usually see), if that method is never called (maybe the code changed since the test was written), the assertions are lost and the test is essentially broken. So I think in general it's best to use mock now unless you have a good reason not to. +1 All new tests should use mock AFAIK We all agree that for mocking we move to mock but there is no indication about using mock or stub, and that's good to me. Mocking and stubbing are two different philosophies, I don't think one is better than the other. I found this article [1] really useful. HTH Regards [1] http://martinfowler.com/articles/mocksArentStubs.html In mock things are stubs by default, which really handy. I like how the behaviour verification in mock sits in the same place as the state verification. In terms of books, I am quite a fan of the book by Roy Osherove. John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
Got it. Thanks. Jay 2013/11/11 Davanum Srinivas dava...@gmail.com Feedback from some of the Nova sessions were, If you are writing new tests, try to use mock. Writing new tests to cover more code (esp drivers) is more preferable to any effort that just converts from mox to mock -- dims On Sun, Nov 10, 2013 at 11:25 PM, Noorul Islam K M noo...@noorul.com wrote: Jay Lau jay.lau@gmail.com writes: Hi, I noticed that we are now using mock, mox and stub for unit test, just curious do we have any guidelines for this, in which condition shall we use mock, mox or stub? There is already a blueprint [1] in Nova project to replace Mox with mock. Also it has a link to ML thread [2]. Regards, Noorul [1] https://blueprints.launchpad.net/nova/+spec/mox-to-mock-conversion [2] http://lists.openstack.org/pipermail/openstack-dev/2013-July/012484.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: http://davanum.wordpress.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
On Mon, 2013-11-11 at 12:07 +, John Garbutt wrote: On 11 November 2013 10:27, Rosa, Andrea (HP Cloud Services) andrea.r...@hp.com wrote: Hi Generally mock is supposed to be used over mox now for python 3 support. That is my understanding too +1 But I don't think we should waste all our time re-writing all our mox and stub tests. Lets just leave this to happen organically for now as we add and refactor tests. We probably need to take the hit at some point, but that doesn't feel like we should do that right now. Hmm, I don't follow this stance. Adding Python3 support is a goal we all share. If we're saying that the use of mox stands in the way of that goal, but that we'd really prefer if people didn't work on porting tests from mox to mock yet ... then are we saying we don't value people working on porting to Python3? And if we plan to use a Python3 compatible version of mox[1], then isn't the Python3 argument irrelevant and saying use mock for new tests just means we'll end up with a mixture of mox and mock? Thanks, Mark. [1] - https://github.com/emonty/pymox ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
I don't see anything explicit in the wiki and hacking guides, they mainly just say to have unit tests for everything and tell you how to run/debug them. Generally mock is supposed to be used over mox now for python 3 support. There is also a blueprint to remove the usage of mox in neutron: https://blueprints.launchpad.net/neutron/+spec/remove-mox For all new patches, we should be using mock over mox because of the python 3 support of mock (and lack thereof for mox). As for when to use mock vs stubs, I think you'll get different opinions from different people. Stubs are quick and easy and that's what I used early when I started contributing to the project, but since then have preferred mox/mock since they validate that methods are actually called with specific parameters, which can get lost when simply stubbing a method call out. In other words, if I'm stubbing a method and doing assertions within it (which you'll usually see), if that method is never called (maybe the code changed since the test was written), the assertions are lost and the test is essentially broken. So I think in general it's best to use mock now unless you have a good reason not to. On 11/10/2013 7:40 AM, Jay Lau wrote: Hi, I noticed that we are now using mock, mox and stub for unit test, just curious do we have any guidelines for this, in which condition shall we use mock, mox or stub? Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
Jay Lau jay.lau@gmail.com writes: Hi, I noticed that we are now using mock, mox and stub for unit test, just curious do we have any guidelines for this, in which condition shall we use mock, mox or stub? There is already a blueprint [1] in Nova project to replace Mox with mock. Also it has a link to ML thread [2]. Regards, Noorul [1] https://blueprints.launchpad.net/nova/+spec/mox-to-mock-conversion [2] http://lists.openstack.org/pipermail/openstack-dev/2013-July/012484.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Do we have some guidelines for mock, stub, mox when writing unit test?
Feedback from some of the Nova sessions were, If you are writing new tests, try to use mock. Writing new tests to cover more code (esp drivers) is more preferable to any effort that just converts from mox to mock -- dims On Sun, Nov 10, 2013 at 11:25 PM, Noorul Islam K M noo...@noorul.com wrote: Jay Lau jay.lau@gmail.com writes: Hi, I noticed that we are now using mock, mox and stub for unit test, just curious do we have any guidelines for this, in which condition shall we use mock, mox or stub? There is already a blueprint [1] in Nova project to replace Mox with mock. Also it has a link to ML thread [2]. Regards, Noorul [1] https://blueprints.launchpad.net/nova/+spec/mox-to-mock-conversion [2] http://lists.openstack.org/pipermail/openstack-dev/2013-July/012484.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: http://davanum.wordpress.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
