On Dec 16, 2007 7:43 PM, Scott Taylor <[EMAIL PROTECTED]> wrote: > > Francis and Pat probably know my thoughts on this, already, but as > far as I can see it, mocks (at least the message based ones) are > popular for one reason in the rails / active-record world: > > Speed. Mocks are extremely fast. I don't think it's uncommon for > those who write specs for rails projects to have a full test suite > running in under 20 seconds if they are mocking all dependencies. > Primarily, this means using mocks for all associations on a Rails > model, and using only mocks for controller specs. My experience with AR is that AR itself (mainly object instantiation) is slow, not the queries. Mocking the queries did not result in a worthwhile test run time savings. Rails creates lots of objects, which causes lots of slowness. Its death by a thousand cuts.
I guess one could mock out the entire AR object, but I'm not convinced that it would result in large performance benefits in many cases. I've tried doing this a couple of times and did not save much time at all. Of course, this was done in view examples on a project that uses Markaby (which is slow). Whatever you do, I recommend taking performance metrics of your suite as you try to diagnose the slowness. The results will probably be surprising. > > The issue of speed seems secondary, but I can already tell how costly > a long build cycle is. At work our test suite has about 1800 specs, > and takes around 3 minutes to run (and hits the database in all model > specs). My coworker actually released something into production on > Friday before he left which failed the build. Obviously - this has a > serious effect on the end user, who is currently receiving errors. > If the test suite took 20 seconds to run, he would be running it all > the time, and this error would have never occurred. The fact that > they don't run quickly means that he isn't going to run them all the > time, and will need to rely on a CI server to beep or do something > obnoxious like that (which isn't an option in the shared office space > in which we are currently working). > > Plus - let's be honest: We use tests as our feedback loop. The > tighter we can get this, the closer we can stay to the code. When the > specs take over a few minutes to run, we no longer have the luxury of > running the whole suite every time we make a little one line change. > We are forced to run specs from one file, or a subset of the specs > from one file, and we loose a certain amount of feedback on how our > code is integrating. > > Yes - there are other reasons for using mocks - like defining > interfaces that don't exist (or may not exist for a long time); So > far the main reason I've seen them used is for speed. I think this > is a major problem with activerecord - and one which only be solved > only by moving to an ORM with a different design pattern (like a true > DataMapper). Lafcadio would probably be in the running for this sort > of thing - a library which can isolate the database and the API with > a middle layer, which can easily be mocked either by a mock library > which could be a drop in replacement for the database, or a middle > layer which could easily be stubbed out with message based stubs. One thing about AR that hurts is all of the copies of the same record. It would be really nice if there was only one instance of each record in the thread. This would help performance and significantly reduce the need to reload the object. > > As always, it's good to hear this sort of discussion going on. > (Francis: It was exactly this sort of discussion that got me > involved with RSpec in the first place) > > Regards, > > Scott > > > > > > On Dec 16, 2007, at 6:22 PM, Francis Hwang wrote: > > > Coming to this thread a bit late: > > > > I think I'm pretty close to Dan, in practice: I'm not a big fan of > > fine-grained isolation in writing your tests. The practice seems to > > me like it would just bug you down. When I'm writing a behavior for a > > particular thing, such as a controller, I don't want to have to worry > > about the precise messages that are passed to its collaborators. I > > try to think in a fairly "black box" manner about it: Presupposing > > that there's a given document in a database table, when I make an > > HTTP request that's looking for that document, I should get that > > document in such-and-such a format. Ideally I wouldn't specify too > > much whether the controller hits Document.find or > > Document.find_by_sql or gets it out of some disk cache or gets the > > data by doing a magical little dance in a faerie circle off in the > > forest. It's really not my test's problem. > > On the other hand, I do think mocking is extremely useful when you're > > dealing with very externalized services with narrow, rigid interfaces > > that you can't implicitly test all the time. At work I have to write > > lots of complex tests around a specific web service, but I don't have > > a lot of control over it, so I wrote a fairly complex mock for that > > service. But even then it's a different sort of mock: It's more state- > > aware than surface-aware, which is part of the point as I see it. Of > > course, writing those sorts of mocks is much more time-consuming. > > > > If you haven't seen it before, Martin Fowler has a pretty good > > article about the differences in styles: http://martinfowler.com/ > > articles/mocksArentStubs.html > > > > Francis Hwang > > http://fhwang.net/ > > > > > > > > On Dec 16, 2007, at 5:59 PM, Dan North wrote: > > > >> Pat. > >> > >> I'm going to reply by promising to reply. You've asked a ton of > >> really useful and insightful questions. I can't do them justice > >> without sitting down and spending a bunch of time thinking about > >> them. > >> > >> I'm going to be off the radar for a bit over Christmas - I've had > >> an insane year and I've promised myself (and my wife) some quiet > >> time. Your questions have a little star next to them in my gmail > >> inbox, which means at the very least they'll be ignored less than > >> the other mail I have to respond to :) > >> > >> The one sentence response, though, is that I honestly don't know > >> (which is why I need to think about it). I can tell you I think I > >> isolate services from their dependencies using mocks, I think I > >> never stub domain objects (I definitely never mock them, but > >> stubbing them is different), I can't say how I test layers because > >> I think we have a different definition of layers. > >> > >> The reason I'm being being so vague is that I usually specify > >> behaviour from the outside in, starting with the "outermost" > >> objects (the ones that appear in the scenario steps) and working > >> inwards as I implement each bit of behaviour. That way I discover > >> service dependencies that I introduce as mocks, and other domain > >> objects that become, well, domain objects. Then there are other > >> little classes that fall out of the mix that seem to make sense as > >> I go along. I don't usually start out with much more of a strategy > >> than that. I can't speak as a tester because I'm not one, so I > >> can't really give you a sensible answer for how isolated my tests > >> are. I simply don't have tests at that level. At an acceptance > >> level my scenarios only ever use real objects wired together doing > >> full end-to-end testing. Sometimes I'll swap in a lighter-weight > >> implementation (say using an in-memory database rather than a > >> remote one, or an in-thread Java web container like Jetty rather > >> than firing up Tomcat), but all the wiring is still the same (say > >> JDBC or HTTP-over-the-wire). I'm still not entirely sure how this > >> maps to Rails, but in Java MVC web apps I would want the controller > >> examples failing if the model's behaviour changed in a particular > >> way, so I can't think of a reason why I would want fake domain > >> objects. > >> > >> Like I said, I'll have a proper think and get back to you. > >> > >> Cheers, > >> Dan > >> > >> On Dec 15, 2007 7:17 AM, Pat Maddox < [EMAIL PROTECTED]> wrote: > >> On Dec 8, 2007 4:06 AM, Dan North < [EMAIL PROTECTED]> wrote: > >>> I prefer the mantra "mock roles, not objects", in other words, > >> mock things > >>> that have behaviour (services, components, resources, whatever your > >>> preferred term is) rather than stubbing out domain objects > >> themselves. If > >>> you have to mock domain objects it's usually a smell that your > >> domain > >>> implementation is too tightly coupled to some infrastructure. > >> > >> Assuming you could easily write Rails specs using the real domain > >> objects, but not hit the database, would you "never" mock domain > >> objects (where "never" means you deviate only in extraordinary > >> circumstances)? I'm mostly curious in the interaction between > >> controller and model...if you use real models, then changes to the > >> model code could very well lead to failing controller specs, even > >> though the controller's logic is still correct. > >> > >> What is your opinion on isolating tests? Do you try to test each > >> class in complete isolation, mocking its collaborators? When you use > >> interaction-based tests, do you always leave those in place, or do > >> you > >> substitute real objects as you implement them (and if so, do your > >> changes move to a more state-based style)? How do you approach > >> specifying different layers within an app? Do you use real > >> implementations if there are lightweight ones available, or do you > >> mock everything out? > >> > >> I realize that's a ton of questions...I'd be very grateful (and > >> impressed!) if you took the time to answer any of them. Also I'd > >> love > >> to hear input from other people as well. > >> > >> Pat > >> > >> _______________________________________________ > >> rspec-users mailing list > >> rspec-users@rubyforge.org > >> http://rubyforge.org/mailman/listinfo/rspec-users > > > > _______________________________________________ > > rspec-users mailing list > > rspec-users@rubyforge.org > > http://rubyforge.org/mailman/listinfo/rspec-users > > _______________________________________________ > rspec-users mailing list > rspec-users@rubyforge.org > http://rubyforge.org/mailman/listinfo/rspec-users > _______________________________________________ rspec-users mailing list rspec-users@rubyforge.org http://rubyforge.org/mailman/listinfo/rspec-users
