+1 - I think we are saying the same thing (not sure if you meant to
reply to me?)
On 6 January 2015 at 14:35, Timothy Baldridge <tbaldri...@gmail.com> wrote:
> I think the answer to questions like this is that you are testing the wrong
> thing, or more correctly you are writing incomplete tests.
>
> In your example, you stubbed out check-pw. But calling check-pw has a
> contract, a contract that (at the moment) only exists in your head, but a
> contract none-the-less. That contract needs to be tested, from both sides.
> Tests should invoke all instances of check-pw. In addition you should have a
> test that pairs a login form with a check-pw and runs tests against this
> system.
>
> Some people call these tests "integration tests" or "system tests". But I
> think of them as contract tests.
>
> Here's a diagram of the problem:
>
> login -----> check-pw
>
> I've found that most code that uses mocking will test the login and the
> check-pw bits, but completely neglect testing the "arrow" between them, and
> when that happens, you get exactly the experience you described.
>
> The other thing I'd like to mention is that I have found it very valuable to
> sit down and think about what code is actually being hit by a test. In your
> example, if check-pw and the db are both mocked, what is actually being
> tested? In your example all you are testing with those functions mocked is
> that Clojure is capable of compiling a function that calls two other
> functions. I can't tell you how many times I've looked at mocked tests and
> realized that the only thing being tested is something like read-string,
> get, or destructuring.
>
> So my personal approach is this: write very coarse tests that exercise the
> entire system. These will catch the protocol mis-matches. Then if you want
> more detail for when tests do fail, write more specific tests. In short:
>
> System (integration) tests: so I feel good about my codebase
> Unit (smaller) tests: so I can figure out what went wrong when the larger
> tests fail.
>
> Timothy
>
> On Tue, Jan 6, 2015 at 6:26 AM, Colin Yates <colin.ya...@gmail.com> wrote:
>>
>> I don't think there is an easy answer here, and note that this is a
>> problem generic to mocking (i.e. not clojure or midje specific).
>>
>> The usual advice applies though:
>> - do you really need to mock? Unit testing is about the coarseness of
>> granularity which is defined more by cohesion and abstractions than "one
>> function and only this function" (e.g. remove the problem by not overly
>> mocking)
>> - make everything fail then fix rather than fix and then upgrade (i.e.
>> update every instance of the call/mock to check-pw before the implementation
>> of check-pw).
>> - Clojure's lack of types means the compiler can't help. Schema or
>> core.typed can. This isn't *the* answer, but I have found it very helpful.
>>
>> As mentioned elsewhere, mocking in general is a very powerful tool, but it
>> is does need wielding carefully. These problems are easier to swallow in
>> strongly typed languages because of IDE support (changing parameters around
>> in Java with IntelliJ is a matter of a few key presses for example).
>>
>> Hope this helps.
>>
>>
>> On Tuesday, 6 January 2015 08:22:36 UTC, Akos Gyimesi wrote:
>>>
>>>
>>> On Sat, Jan 3, 2015, at 02:46 AM, Brian Marick wrote:
>>> >
>>> > > I use TDD and mocking/stubbing (conjure) to test each layer of my
>>> > > code.
>>> > > The problem is when I change the function signature and the tests do
>>> > > not
>>> > > break, because the mocks/stubs do not know when their argument lists
>>> > > no
>>> > > longer agree with the underlying function they are mocking. Is there
>>> > > a
>>> > > way to catch this? Short of a test suite that eschews stubbing in
>>> > > favor
>>> > > of full setup/teardown of DB data for each test?
>>> >
>>> > Could you give an example? I use mocks fairly heavily, and I don't seem
>>> > to have this problem. Perhaps it's because I change the tests before
>>> > the
>>> > code?
>>>
>>> Although the subject changed a little bit, I would be still interested
>>> in your approach to refactoring if there is heavy use of mocking. Let me
>>> give you an example:
>>>
>>> Let's say I am writing a login form, trying to use the top-down approach
>>> you described. My approach could be the following:
>>>
>>> (unfinished check-pw)
>>>
>>> (fact "login-form succeeds if user enters the correct password"
>>> (login-form-success? {:username "admin" :password "secret"}) => true
>>> (provided
>>> (db/get-user "admin") => (contains (:password "my-secret-hash"))
>>> (check-pw "my-secret-hash" "secret") => true))
>>>
>>> (defn login-form-success? [user-input]
>>> (let [user (db/get-user (:username user-input))]
>>> (check-pw (:password user) (:password user-input))))
>>>
>>> Then I finish the check-pw function and everything works.
>>>
>>> Now, later that day I decide that I pass the whole user object to the
>>> check-pw function. Maybe I want to use the user ID as a salt, or maybe I
>>> just want to leave the possibility for checking password expiration,
>>> etc. So I modify the test and the implementation of check-pw so that the
>>> first parameter is the user object, not the password hash.
>>>
>>> Suddenly my co-worker comes to me saying "hey, I need you on a meeting
>>> right now!" I close my laptop, and an hour later I think "where were
>>> we?..." I run all the tests, and they all pass, so I commit.
>>>
>>> Except... I forgot to modify all the invocations of check-pw in both the
>>> test and the implementation. Every test pass, so I have no way of
>>> finding out the problem without careful code review or by examining the
>>> stack traces from the live code.
>>>
>>> While this bug is easy to catch, what if my function is mocked in
>>> several places, and I fail to rewrite all of them properly?
>>>
>>> Do you have any advice on what you would have done differently here to
>>> avoid this bug?
>>>
>>> Regards,
>>> Akos
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> “One of the main causes of the fall of the Roman Empire was that–lacking
> zero–they had no way to indicate successful termination of their C
> programs.”
> (Robert Firth)
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/T8fIW27kDYE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
