Thanks for the great feedback, Gustavo! I have some follow-up inline. It reflects my understanding of what you wrote, so don't hate me if any of it reflects a *mis*understanding. :) If anything I say *seems* to imply some deeper criticism or veiled aspersion, just assume I didn't actually mean to imply anything and take my statement/question at face value. I'm a pretty straight-forward person and I likely just phrased things poorly. :)
-eric On Fri, Feb 13, 2015 at 6:00 AM, Gustavo Niemeyer <[email protected]> wrote: > On Wed, Feb 11, 2015 at 4:53 PM, Eric Snow <[email protected]> wrote: >> tl;dr Using fakes for testing works well so I wrote a base fake type. [1] >> >> While working on the GCE provider, Wayne and I started taking a >> different approach to unit testing than the usual 1. expose an >> external dependency as an unexported var; 2.export it in >> export_test.go; 3. patch it out in tests. [2] Instead we wrote an >> interface for the provider's low-level API and implemented the >> provider relative to that. [3] Then in tests we used a fake >> implementation of that interface instead of the concrete one. Instead >> of making the actual API requests, the fake simply tracked method >> calls and controlled return values. > > This is a "mock object" under some well known people's terminology [1]. With all due respect to Fowler, the terminology in this space is fairly muddled still. :) > > There are certainly benefits, but there are also very well known > problems in that approach which should be kept in mind. We've > experienced them very closely in the first Python-based implementation > of juju, and I did many other times elsewhere. I would probably not > have written [2] if I had a time machine. > > The most problematic aspect of this approach is that tests are pretty > much always very closely tied to the implementation, in a way that you > suddenly cannot touch the implementation anymore without also fixing a > vast number tests to comply. Let's look at this from the context of "unit" (i.e. function signature) testing. By "implementation" do you mean you mean the function you are testing, or the low-level API the function is using, or both? If the low-level API then it seems like the "real fake object" you describe further on would help by moving at least part of the test setup down out of the test and down into the fake. However aren't you then just as susceptible to changes in the fake with the same maintenance consequences? Ultimately I just don't see how you can avoid depending on low-level details ("closely tied to the implementation") in your tests and still have confidence that you are testing things rigorously. I think the best you can do is define an interface encapsulating the low-level behavior your functions depend on and then implement them relative to that interface and write your tests with that interface faked out. Also, the testing world puts a lot are emphasis on branch coverage in tests. It almost sounds like you are suggesting that is not such an important goal. Could you clarify? Perhaps I'm inferring too much from what you've said. :) > Tests organized in this fashion also tend > to obfuscate the important semantics being tested, and instead of that > you see a sequence of apparently meaningless calls and results out of > context. Understanding and working on these tests just a month after > you cooked them is a nightmare. I think this is alleviated if you implement and test against an interface that strictly defines the low-level API on which you depend. > > The perfect test breaks if and only if the real assumption being > challenged changes. Anything walking away from this is decreasing the > test quality. Agreed. The challenge is in managing the preconditions of each test in a way that is robust to low-level changes but also doesn't "obfuscate the important semantics being tested". In my experience, this is not solved by any single testing approach so a judiciously chosen mix is required. > > As a recommendation to avoid digging a hole -- one that is pretty > difficult to climb out of once you're in -- instead of testing method > calls and cooking fake return values in your own test, build a real > fake object: one that pretends to be a real implementation of that > interface, and understands the business logic of it. Then, have > methods on it that allow tailoring its behavior, but in a high-level > way, closer to the problem than to the code. Ah, I like that! So to rephrase, instead of a type where you just track calls and explicitly control return values, it is better to use a type that implements your expectations about the low-level system, exposed via the same API as the actual one? This would likely still involve both to implement the same interface, right? The thing I like about that approach is that is forces you to "document" your expectations (i.e. dependencies) as code. The problem is that you pay (in development time and in complexity) for an extra layer to engineer and maintain, and your tests are less isolated (meaning the scope of each test increases). The benefit of codified expectations is roughly achievable through comments rather than code, but comments have their own deficiencies, so that also isn't the perfect solution to communicating expectations about the low-level behavior. Regardless, as I noted in an earlier message, I think testing needs to involve: 1. a mix of high branch coverage through isolated unit tests, 2. "enough" testing to ensure your expectations for the low-level API are met, 3. "enough" coverage of the full stack (at least common-path) via integration tests. Your recommendation on a low-level implementation to use for testing is a good one (and one I'll make use of), but it's only one piece of the testing puzzle. That said, I don't think you point is that it's the only testing approach one should use. :) I appreciate you bringing it up. -- Juju-dev mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
