On 09/22/2014 08:58 AM, Matthew Booth wrote: > If you missed the inaugural OpenStack Bootstrapping Hour, it's here: > http://youtu.be/jCWtLoSEfmw . I think this is a fantastic idea and big > thanks to Sean, Jay and Dan for doing this. I liked the format, the > informal style and the content. Unfortunately I missed the live event, > but I can confirm that watching it after the event worked just fine > (thanks for reading out live questions for the stream!). > > I'd like to make a brief defence of faking, which perhaps predictably in > a talk about mock took a bit of a bashing. > > Firstly, when not to fake. As Jay pointed out, faking adds an element of > complexity to a test, so if you can achieve what you need to with a > simple mock then you should. But, as the quote goes, you should "make > things as simple as possible, but not simpler." > > Here are some simple situations where I believe fake is the better solution: > > * Mock assertions aren't sufficiently expressive on their own > > For example, imagine your code is calling: > > def complex_set(key, value) > > You want to assert that on completion of your unit, the final value > assigned to <key> was <value>. This is difficult to capture with mock > without risking false assertion failures if complex_set sets other keys > which you aren't interested in, or if <key>'s value is set multiple > times, but you're only interested in the last one. A little fake > function which stores the final value assigned to <key> does this simply > and accurately without adding a great deal of complexity. e.g. > > mykey = [None] > def fake_complex_set(key, value): > if key == 'FOO': > mykey[0] = value > > with mock.patch.object(unit, 'complex_set', side_effect=fake_complex_set): > run_test() > self.assertEquals('expected', mykey[0]) > > Summary: fake method is a custom mock assertion. > > * A simple fake function is simpler than a complex mock dance > > For example, you're mocking 2 functions: start_frobincating(key) and > wait_for_frobnication(key). They can potentially be called overlapping > with different keys. The desired mock return value of one is dependent > on arguments passed to the other. This is better mocked with a couple of > little fake functions and some external state, or you risk introducing > artificial constraints on the code under test. > > Jay pointed out that faking methods creates more opportunities for > errors. For this reason, in the above cases, you want to keep your fake > function as simple as possible (but no simpler). However, there's a big > one: the fake driver! > > This may make less sense outside of driver code, although faking the > image service came up in the talk. Without looking at the detail, that > doesn't necessarily sound awful to me, depending on context. In the > driver, though, the ultimate measure of correctness isn't a Nova call: > it's the effect produced on the state of an external system. > > For the VMware driver we have nova.tests.virt.vmwareapi.fake. This is a > lot of code: 1599 lines as of writing. It contains bugs, and it contains > inaccuracies, and both of these can mess up tests. However: > > * It's vastly simpler than the system it models (vSphere server) > * It's common code, so gets fixed over time > * It allows tests to run almost all driver code unmodified > > So, for example, it knows that you can't move a file before you create > it. It knows that creating a VM creates a bunch of different files, and > where they're created. It knows what objects are created by the server, > and what attributes they have. And what attributes they don't have. If > you do an object lookup, it knows which objects to return, and what > their properties are. > > All of this knowledge is vital to testing, and if it wasn't contained in > the fake driver, or something like it[1], would have to be replicated > across all tests which require it. i.e. It may be 1599 lines of > complexity, but it's all complexity which has to live somewhere anyway. > > Incidentally, this is fresh in my mind because of > https://review.openstack.org/#/c/122760/ . Note the diff stat: +70, > -161, and the rewrite has better coverage, too :) It executes the > function under test, it checks that it produces the correct outcome, and > other than that it doesn't care how the function is implemented. > > TL;DR > > * Bootstrap hour is awesome > * Don't fake if you don't have to > * However, there are situations where it's a good choice
Yes. I agree with all of these things. For reference, we use several complex-ish fakes in nodepool testing because it's the sanest thing to do. Also, the nova fake-virt driver is actually completely amazing and allows us to test system state things that otherwise would be hard to model. However - 100 times yes - don't go faking things when a mock will do the job - and that's almost always. > Thanks for reading :) > > Matt > > [1] There are other ways to skin this cat, but ultimately if you aren't > actually spinning up a vSphere server, you're modelling it somehow. > _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
