Re: [rspec-users] Why not MockEverthing or why use fixtures for all tests?
[Big Snip] There are a few bad assumptions in your colleague's response, so to set the record straight: * test coverage and tests which use the interaction-based test approach are not mutually exclusive * you can have crappy tests which take the state-based approach and crappy tests which use a interaction-based approach * interaction-based testing is not merely limited to contrived examples on people's blogs, it is a real practice which adds value on lots of real-world projects * using factories to generate required objects in tests has several pros over the use of fixtures, and very very very few cons State-based testing and interaction-based testing both have their place. There are number of reasons why they are both useful, but I'm going to pick two: object decomposition (and coordinators) and integration testing. Others have mentioned the value of writing tests with the interface you want so I'm going to leave that out. As an application grows in features and complexity (business logic of course) good developers will decompose the problem into a number of simple objects. Some of these objects are responsible for doing the work and others are responsible for coordinating other objects to do the work. Objects which are responsible for coordinating are great candidates for using interaction-based testing, because you are concerned in the interaction, not the state. If you don't have integration tests then using an interaction-based testing approach is not worth it because you need something that is going to test the real objects working with real objects. In Rails you can write integration tests as Rail's ActionController::IntegrationTests, Rail's functional tests, RSpec stories, or RSpec controller tests w/view isolation turned off. IMO, one false benefit of only using a state-based approach when writing a full-fledged application is that every object is essentially an integration test at some level. You are always testing everything with everything that it touches. This can lead to having one failure in one model make several other model tests fail, and it can make several controller tests failing (as well as any other object which touches the model that is failing). I see this has a big negative because it makes it more difficult to pinpoint the issue. People will end up tracking it down, but it can be time consuming and frustrating. Now on the flip side people will complain that they renamed a model method and re-ran all of their tests and everything passed, but when running the application a bug exists. Doh, we forgot to update the controller that relied on calling that model method. It is normal to say/think, well that should have failed because the method doesn't exist on the model. (It sounds like David Chelimsky may have something in trunk to help with this.) The main problem here though is that an integration test didn't fail exposing that you weren't done with your change. Thinking back to coordinating objects, my controllers don't contain business logic in them because they are application layer classes, they aren't apart of the domain of my software. They are only used by the application to allow the software to fulfill the requirements of my customer. Controllers are coordinators, not DOERS. They ask other objects to fulfill a business requirement for them like moving stocks from one portfolio to the another. So I used interaction-based testing here to ensure that my controller is finding a stock, finding a portfolio and asking a portfolio manager to move the stock to the designed portfolio. I don't need to have those things written or even fully implemented to ensure my controller works as I expect. I should be able to see that my controller does what it should be doing, even if the pieces it will use to do the work in the application aren't finished. Now if those aren't implemented I should have an integration test which fails showing me that the feature for moving stocks from one portfolio to another is not completed, but that isn't what I'm testing in my controller. Also after my controller works as expected I can go make sure the PortfolioManager works as expected, and then I can go down and make sure the Stock model does what I expect. When these objects are working correctly individual I run my integration tests to ensure they work well together. Another drawback of only using state-based testing is that you always have to develop bottom up. You have to start with the low level components and work your way out. I used to write code this way. I think I have progressed beyond that, and now I write things in a Acceptance Test Driven Development style. I start by writing an integration test from the user's perspective proving that the feature doesn't work, and then I move to the view, and then to the controller, then to any manager/factory/presenter/service objects that are required, and then down
Re: [rspec-users] Why not MockEverthing or why use fixtures for all tests?
On Mar 19, 2008, at 1:03 PM, David Chelimsky wrote: On Wed, Mar 19, 2008 at 10:42 AM, Glenn Ford [EMAIL PROTECTED] wrote: [Big Snip] There are a few bad assumptions in your colleague's response, so to set the record straight: * test coverage and tests which use the interaction-based test approach are not mutually exclusive * you can have crappy tests which take the state-based approach and crappy tests which use a interaction-based approach * interaction-based testing is not merely limited to contrived examples on people's blogs, it is a real practice which adds value on lots of real-world projects * using factories to generate required objects in tests has several pros over the use of fixtures, and very very very few cons State-based testing and interaction-based testing both have their place. There are number of reasons why they are both useful, but I'm going to pick two: object decomposition (and coordinators) and integration testing. Others have mentioned the value of writing tests with the interface you want so I'm going to leave that out. As an application grows in features and complexity (business logic of course) good developers will decompose the problem into a number of simple objects. Some of these objects are responsible for doing the work and others are responsible for coordinating other objects to do the work. Objects which are responsible for coordinating are great candidates for using interaction-based testing, because you are concerned in the interaction, not the state. If you don't have integration tests then using an interaction-based testing approach is not worth it because you need something that is going to test the real objects working with real objects. In Rails you can write integration tests as Rail's ActionController::IntegrationTests, Rail's functional tests, RSpec stories, or RSpec controller tests w/view isolation turned off. IMO, one false benefit of only using a state-based approach when writing a full-fledged application is that every object is essentially an integration test at some level. You are always testing everything with everything that it touches. This can lead to having one failure in one model make several other model tests fail, and it can make several controller tests failing (as well as any other object which touches the model that is failing). I see this has a big negative because it makes it more difficult to pinpoint the issue. People will end up tracking it down, but it can be time consuming and frustrating. Now on the flip side people will complain that they renamed a model method and re-ran all of their tests and everything passed, but when running the application a bug exists. Doh, we forgot to update the controller that relied on calling that model method. It is normal to say/think, well that should have failed because the method doesn't exist on the model. (It sounds like David Chelimsky may have something in trunk to help with this.) The main problem here though is that an integration test didn't fail exposing that you weren't done with your change. Thinking back to coordinating objects, my controllers don't contain business logic in them because they are application layer classes, they aren't apart of the domain of my software. They are only used by the application to allow the software to fulfill the requirements of my customer. Controllers are coordinators, not DOERS. They ask other objects to fulfill a business requirement for them like moving stocks from one portfolio to the another. So I used interaction-based testing here to ensure that my controller is finding a stock, finding a portfolio and asking a portfolio manager to move the stock to the designed portfolio. I don't need to have those things written or even fully implemented to ensure my controller works as I expect. I should be able to see that my controller does what it should be doing, even if the pieces it will use to do the work in the application aren't finished. Now if those aren't implemented I should have an integration test which fails showing me that the feature for moving stocks from one portfolio to another is not completed, but that isn't what I'm testing in my controller. Also after my controller works as expected I can go make sure the PortfolioManager works as expected, and then I can go down and make sure the Stock model does what I expect. When these objects are working correctly individual I run my integration tests to ensure they work well together. Another drawback of only using state-based testing is that you always have to develop bottom up. You have to start with the low level components and work your way out. I used to write code this way. I think I have progressed beyond that, and now I write things in a Acceptance Test Driven Development style. I start by writing an integration test from the user's perspective proving
Re: [rspec-users] Why not MockEverthing or why use fixtures for all tests?
Hi David On 18 Mar 2008, at 08:15, David Schmidt wrote: Here is his email to me, less his signature as I don't want to make this personal. I'd like to see what the RSpec user community has to say in response to his comments, below: [snip email] For me, it basically boils down to: use whichever is appropriate for what you're testing, as long as it provides good coverage and makes the tests easy to write. If the tests are hard to write, and you're mocking/stubbing all over the place, it's a smell that your implementation is probably not loosely coupled enough - perhaps you need to refactor. Mocks are a very useful testing construct, but using them religiously everywhere is as bad as not using them religiously, IMHO. Have you read Mocks aren't Stubs[1]? It's a very good treatise on the advantages and disadvantages of using mocks in testing. Pat Maddox also wrote an article on this subject recently that I found very helpful[2]. Thanks Chris [1] http://martinfowler.com/articles/mocksArentStubs.html [2] http://evang.eli.st/blog/2008/2/3/it-s-not-about-state-or-interactions-it-s-about-behavior ___ rspec-users mailing list rspec-users@rubyforge.org http://rubyforge.org/mailman/listinfo/rspec-users
Re: [rspec-users] Why not MockEverthing or why use fixtures for all tests?
On Tue, Mar 18, 2008 at 5:04 AM, Chris Parsons [EMAIL PROTECTED] wrote: Hi David On 18 Mar 2008, at 08:15, David Schmidt wrote: Here is his email to me, less his signature as I don't want to make this personal. I'd like to see what the RSpec user community has to say in response to his comments, below: [snip email] For me, it basically boils down to: use whichever is appropriate for what you're testing, as long as it provides good coverage and makes the tests easy to write. If the tests are hard to write, and you're mocking/stubbing all over the place, it's a smell that your implementation is probably not loosely coupled enough - perhaps you need to refactor. Mocks are a very useful testing construct, but using them religiously everywhere is as bad as not using them religiously, IMHO. Have you read Mocks aren't Stubs[1]? It's a very good treatise on the advantages and disadvantages of using mocks in testing. Pat Maddox also wrote an article on this subject recently that I found very helpful[2]. I agree wholeheartedly! I'd also recommend Mock Roles, Not Objects[3] When I'm spec'ing views, for example, I tend to follow this path: 1. start with an expectation, watch it fail 2. get it to pass with hard-coded values in the view 3. replace the hard-coded view code with the code I wish I had[4] 4. mock/stub whatever I need at the time in the example to get it to pass This last step varies depending on the state of the system. If the model class I want doesn't exist, I use a mock object. If it does exist, I'll use stub_model[5] to provide isolation from the db, but allow the real model code to be invoked. If I used a mock to begin with, I may switch it later. The point is that while I'm in the view I want to stay focused on the view. FWIW. Cheers, David [3] jmock.org/oopsla2004.pdf [4] Writing the code I wish I had is something I learned from James Grenning at Object Mentor. [5] stub_model is a new method in git - will be released with 1.1.4. It creates a real model instance and prevents database access a la unit_record (but on an instance by instance basis). This allows you to reap the benefit of isolation from the db while reducing the false positives that concern so many with using mocks in Ruby. Thanks Chris [1] http://martinfowler.com/articles/mocksArentStubs.html [2] http://evang.eli.st/blog/2008/2/3/it-s-not-about-state-or-interactions-it-s-about-behavior ___ rspec-users mailing list rspec-users@rubyforge.org http://rubyforge.org/mailman/listinfo/rspec-users ___ rspec-users mailing list rspec-users@rubyforge.org http://rubyforge.org/mailman/listinfo/rspec-users
Re: [rspec-users] Why not MockEverthing or why use fixtures for all tests?
David Schmidt wrote: Hello fellow RSpec users. Before you all start warming up your flame throwers please let me explain my Subject line. I've been working over 4 months on a large Rails project with a few other developers. Test coverage was spotty at best, though they *were* RSpec tests. One of the other developers and I had started adding more tests, mostly controller tests using the methodology given at rspec.info for writing controller tests isolated from the model and view layers using stubs and mocks. Recently a new project manager was put in place and he brought in another developer. This developer promptly started to re-write all the *existing* controller (and later view) tests, removing all mocks and stubs and replacing them with code to use fixtures. (He also deletes many comments he finds in the code if *he* thinks they're obvious, but that's another story...). His commit messages include comments like Stop mocking around and More fixes due to our test mockery. When challenged on why he's re-writing these tests instead of writing new, missing tests (even tests using fixtures) he replied with this e-mail with the subject Why not MockEverything. (Note that I *do* use fixtures for model tests but follow the RSpec documentation and use mocks/stubs for controller and view tests for isolation.) In the email this developer mentions tests broken by the addition of conditional to the view. This conditional used a model method not previously used in the view, and the addition of one stub was sufficient to fix the view test in question. Here is his email to me, less his signature as I don't want to make this personal. I'd like to see what the RSpec user community has to say in response to his comments, below: --- Why not MockEverything --- David I've removed the mocks on purpuse. Not that I have sufficient ills with them to meddle without a *need*. We committed simple template fixes adding a conditional and there, yet the tests broke. Now this was to be expected, the tests were constructed by exhaustively mocking out all methods called on the object. Add a simple conditional be it harmless as it is now means another method needs to be mocked out. The MockEverything approach is not healthy, judicious use is preferable. One thing is to write a short sample in a blog and another is to have a working app with lots of tests. From all my apps that I have worked on this has by far the lowest coverage both in profile and in test value. There is no discussion we are all committed to tests. To better see what constitutes good practice I recommend you to inspect the source of RadiantCMS a beautiful and well engineered app recently rewrote to use rspec instead of Test::Unit: http://dev.radiantcms.org/browser/trunk/radiant/spec Observe how the code is restrained in mocking, real objects are preferred wherever possible. Incidentally they don't use fixtures rather factories to create *real* objects. Now the factory part is a separate issue I'll don't discuss here, as it has its own disadvantages especially a project with many models ... With real objects your test will not be brittle, and their value will be kept even after adjusting the templates or doing other small refactorings. Contrary to common misconception test speed will not be affected either. Especially for view tests where you don't even have to save to the db upon preparing the test rig. Beside Radiant there where efforts to rspec Typo and Mephisto (both noted rails blog engines). Still these were half harted conversions so my arguments based on them would not have the same weight. RadiantCMS is enough - they are used on ruby-lang.org and have converted 100% to rspec ... plus they also have good coverage showing that they actually believe in tests. So please look into Radiant, you'll find it most helpful I think. --- END OF EMAIL--- Thank you, David Schmidt ___ rspec-users mailing list rspec-users@rubyforge.org http://rubyforge.org/mailman/listinfo/rspec-users I was going to start a thread about mocks and fixtures this morning too, so Ill use this one. Let me first say that I am a very very recent comer to RSpec so my opinions dont carry much weight, but ... I have come to the tentative conclusion that mocking is fine in view specs where you are really only interested that certain assigns have been made and that they respond to certain messages. In fact mocks are ideal. Possibly in models too. However in controller specs I think you find examples where fixtures are just the best way to go. In these cases, from what I have seen, mocking leads to brittle and frankly worthless tests (a half-arsed test is worse than no test at all, right?). The case that crystalised that
Re: [rspec-users] Why not MockEverthing or why use fixtures for all tests?
On Mar 18, 2008, at 6:36 AM, Alan Larkin wrote: David Schmidt wrote: Hello fellow RSpec users. Before you all start warming up your flame throwers please let me explain my Subject line. I've been working over 4 months on a large Rails project with a few other developers. Test coverage was spotty at best, though they *were* RSpec tests. One of the other developers and I had started adding more tests, mostly controller tests using the methodology given at rspec.info for writing controller tests isolated from the model and view layers using stubs and mocks. Recently a new project manager was put in place and he brought in another developer. This developer promptly started to re-write all the *existing* controller (and later view) tests, removing all mocks and stubs and replacing them with code to use fixtures. (He also deletes many comments he finds in the code if *he* thinks they're obvious, but that's another story...). His commit messages include comments like Stop mocking around and More fixes due to our test mockery. When challenged on why he's re-writing these tests instead of writing new, missing tests (even tests using fixtures) he replied with this e-mail with the subject Why not MockEverything. (Note that I *do* use fixtures for model tests but follow the RSpec documentation and use mocks/stubs for controller and view tests for isolation.) In the email this developer mentions tests broken by the addition of conditional to the view. This conditional used a model method not previously used in the view, and the addition of one stub was sufficient to fix the view test in question. Here is his email to me, less his signature as I don't want to make this personal. I'd like to see what the RSpec user community has to say in response to his comments, below: --- Why not MockEverything --- David I've removed the mocks on purpuse. Not that I have sufficient ills with them to meddle without a *need*. We committed simple template fixes adding a conditional and there, yet the tests broke. Now this was to be expected, the tests were constructed by exhaustively mocking out all methods called on the object. Add a simple conditional be it harmless as it is now means another method needs to be mocked out. The MockEverything approach is not healthy, judicious use is preferable. One thing is to write a short sample in a blog and another is to have a working app with lots of tests. From all my apps that I have worked on this has by far the lowest coverage both in profile and in test value. There is no discussion we are all committed to tests. To better see what constitutes good practice I recommend you to inspect the source of RadiantCMS a beautiful and well engineered app recently rewrote to use rspec instead of Test::Unit: http://dev.radiantcms.org/browser/trunk/radiant/spec Observe how the code is restrained in mocking, real objects are preferred wherever possible. Incidentally they don't use fixtures rather factories to create *real* objects. Now the factory part is a separate issue I'll don't discuss here, as it has its own disadvantages especially a project with many models ... With real objects your test will not be brittle, and their value will be kept even after adjusting the templates or doing other small refactorings. Contrary to common misconception test speed will not be affected either. Especially for view tests where you don't even have to save to the db upon preparing the test rig. Beside Radiant there where efforts to rspec Typo and Mephisto (both noted rails blog engines). Still these were half harted conversions so my arguments based on them would not have the same weight. RadiantCMS is enough - they are used on ruby-lang.org and have converted 100% to rspec ... plus they also have good coverage showing that they actually believe in tests. So please look into Radiant, you'll find it most helpful I think. --- END OF EMAIL--- Thank you, David Schmidt ___ rspec-users mailing list rspec-users@rubyforge.org http://rubyforge.org/mailman/listinfo/rspec-users I was going to start a thread about mocks and fixtures this morning too, so Ill use this one. Let me first say that I am a very very recent comer to RSpec so my opinions dont carry much weight, but ... I have come to the tentative conclusion that mocking is fine in view specs where you are really only interested that certain assigns have been made and that they respond to certain messages. In fact mocks are ideal. Possibly in models too. However in controller specs I think you find examples where fixtures are just the best way to go. In these cases, from what I have seen, mocking leads to brittle and frankly worthless tests (a
Re: [rspec-users] Why not MockEverthing or why use fixtures for all tests?
Hello, On Tue, Mar 18, 2008 at 4:15 AM, David Schmidt [EMAIL PROTECTED] wrote: Hello fellow RSpec users. Before you all start warming up your flame throwers please let me explain my Subject line. I've been working over 4 months on a large Rails project with a few other developers. Test coverage was spotty at best, though they *were* RSpec tests. One of the other developers and I had started adding more tests, mostly controller tests using the methodology given at rspec.info for writing controller tests isolated from the model and view layers using stubs and mocks. Recently a new project manager was put in place and he brought in another developer. This developer promptly started to re-write all the *existing* controller (and later view) tests, removing all mocks and stubs and replacing them with code to use fixtures. (He also deletes many comments he finds in the code if *he* thinks they're obvious, but that's another story...). His commit messages include comments like Stop mocking around and More fixes due to our test mockery. When challenged on why he's re-writing these tests instead of writing new, missing tests (even tests using fixtures) he replied with this e-mail with the subject Why not MockEverything. (Note that I *do* use fixtures for model tests but follow the RSpec documentation and use mocks/stubs for controller and view tests for isolation.) In the email this developer mentions tests broken by the addition of conditional to the view. This conditional used a model method not previously used in the view, and the addition of one stub was sufficient to fix the view test in question. Here is his email to me, less his signature as I don't want to make this personal. I'd like to see what the RSpec user community has to say in response to his comments, below: --- Why not MockEverything --- David I've removed the mocks on purpuse. Not that I have sufficient ills with them to meddle without a *need*. We committed simple template fixes adding a conditional and there, yet the tests broke. Now this was to be expected, the tests were constructed by exhaustively mocking out all methods called on the object. Add a simple conditional be it harmless as it is now means another method needs to be mocked out. The MockEverything approach is not healthy, judicious use is preferable. One thing is to write a short sample in a blog and another is to have a working app with lots of tests. From all my apps that I have worked on this has by far the lowest coverage both in profile and in test value. There is no discussion we are all committed to tests. To better see what constitutes good practice I recommend you to inspect the source of RadiantCMS a beautiful and well engineered app recently rewrote to use rspec instead of Test::Unit: http://dev.radiantcms.org/browser/trunk/radiant/spec Observe how the code is restrained in mocking, real objects are preferred wherever possible. Incidentally they don't use fixtures rather factories to create *real* objects. Now the factory part is a separate issue I'll don't discuss here, as it has its own disadvantages especially a project with many models ... With real objects your test will not be brittle, and their value will be kept even after adjusting the templates or doing other small refactorings. Contrary to common misconception test speed will not be affected either. Especially for view tests where you don't even have to save to the db upon preparing the test rig. Beside Radiant there where efforts to rspec Typo and Mephisto (both noted rails blog engines). Still these were half harted conversions so my arguments based on them would not have the same weight. RadiantCMS is enough - they are used on ruby-lang.org and have converted 100% to rspec ... plus they also have good coverage showing that they actually believe in tests. So please look into Radiant, you'll find it most helpful I think. --- END OF EMAIL--- Thank you, David Schmidt There are a few bad assumptions in your colleague's response, so to set the record straight: * test coverage and tests which use the interaction-based test approach are not mutually exclusive * you can have crappy tests which take the state-based approach and crappy tests which use a interaction-based approach * interaction-based testing is not merely limited to contrived examples on people's blogs, it is a real practice which adds value on lots of real-world projects * using factories to generate required objects in tests has several pros over the use of fixtures, and very very very few cons State-based testing and interaction-based testing both have their place. There are number of reasons why they are both useful, but I'm going to pick two: object decomposition (and coordinators) and integration testing. Others have mentioned the value of writing tests with the interface you want so I'm going to leave that out. As an application grows in features and
