On Fri, Oct 28, 2011 at 8:19 AM, John W Vines <[email protected]> wrote:
> I will start off by saying that myself, as well as most of the committers, > are no familiar with EasyMock, PowerMock, or really any Mocking tools. But > I've done some reading to determine what the scope of mocking can get us. So > pardon me if I bore you because I'm just going to lay out what I know in > hopes of A. making sure my understanding is correct and B. to see if what I > think we should use it for is within the scope of Mocking as well as > feasible. > > 1. With exception for PowerMock, mocking tools are nothing more than tools > which give the ability to define a pre-set set of responses to an interface. > That is, if we wanted to implement a low level iterator to replace the > Reader we use to read items off disk/out of the in memory map, we could, but > we could only give it a set of responses to give when queried. We could not > (through just the mocking tools) set it up to give programmatic responses, > such as returning the result of another method, unless we 'pre-load' the > method result. That is, we couldn't have one mock method which put an item > in a map and another mock method pull it out again, unless we define the > pull method to return that specific value for the Xth call to the pull > method to which that correlated. > That isn't exactly correct. You can actually pass in the data structure so when someone does a call (e.g. put(Thing)) you can catch that call and and actually pass that into a data structure which is accessed by reference. For example, check out EasyMock's IAnswer where you can programatically determine a 'return' result. > > 2. PowerMock lets us essentially do the same as things like EasyMock, > except we're not bound to interfaces. This way we can use a defined class > and only overwrite the methods we want with predefined results. > > 3. What I want to see us doing, at a very high level, is to have the > ability to mock an entire TServer to the extent where we will use something > to replace Zookeeper (We should probably turn our ZK work with an interface) > with a MockZookeeper (not generated through a Mock util) which is nothing > more than a Map. Same thing with the FileReader, except a SortedMap, the > loggers, and the master. This way we could fully implement a whole TServer > without worry about HDFS and Zookeeper. To a similar extent I would like to > see this done for all core components, but mocking the various connectors we > use to get done what we need to. I see a few sets of Mock class we will have > to create. But with less chance of divergence in behavior then we currently > experience with our MockAccumulo setup. > I feel like this kind of thing (particularly with a fully mocked TServer) would be part of Accumulo-14<https://issues.apache.org/jira/browse/ACCUMULO-14>and the testing suite. Otherwise, you end up with divergent implementations and have the same mess where the mocks don't have any real correlation to how the system works. However, a lot of times you really don't want to (and shouldn't be) mocking the full interface for a component (eg. a full mock TSever) since then you are not really testing much of anything other than you are making the calls to the mock that you can see you are making from the code (I'm thinking here particularly of code that basically does pass through of calls to other objects). In that case you really need to have an integration test to make sure that each piece works with others, since they would be clearly tightly coupled. The mock is really optimal for cases where you have loose coupling between objects and only need to cover a few calls to an external class. You are really looking to test the logic/computation of the object and only cover a few elements of interaction with other objects via mocking. > 4. My principal concern about 3 above is divergence with the code (less so > than our current setup but with emulating thrift interfaces we could get > problems). But also the feasibility of mocking thrift connections. But if > all this works out, I think we'll be able to optimize our code coverage > better than our current setup without the requirement to do a full Accumulo > setup for junit tests. > > This is already the problem with the MockAccumulo (and everything under client.impl.mock) so we want to avoid re-redoing that - divergent behavior is only bad :) In short, mocking is really powerful and great for doing isolated testing, but in the end is only as good as the people who use it. It is very easy to go overboard with the mocking and basically mock out an entire, tightly coupled interface that has no reflection on reality. This leads to people thinking their code works because it matches the mock responses, which actually look nothing like what reality would serve up to their code (one case where I have been bitten by this in the past if mocking out what I _thought_ ZooKeeper would return). > Cheers > John > > ----- Original Message ----- > | From: "Jesse Yates" <[email protected]> > | To: [email protected] > | Sent: Thursday, October 27, 2011 2:21:33 PM > | Subject: Re: Mocking framework > | On Thu, Oct 27, 2011 at 10:55 AM, Drew Farris <[email protected]> wrote: > | > | > I agree in principle without having reviewed the tests accompanying > | > ACCUMULO-53. > | > > | > I've used EasyMock in the past and have been very happy with the > | > ability to programatically mock implementations of interfaces, > | > define > | > test input and assert a proper sequence of calls, and output. In > | > general I think mock object testing is a useful tool, especially in > | > cases where the existing interface implementations have heavyweight > | > external dependencies, such as for mocking JDBC result sets or > | > portions of the Hadoop framework. I've used mock objects to write > | > unit > | > tests for mappers and reducers by mocking out the hadoop context, > | > counters, etc. Arguably this would be have been better achieved with > | > MRUnit, which wasn't on my radar at the time. > | > > | > I found that I didn't have to jump through hoops to make code > | > testable. Private methods could really be private instead of > | > protected > | > or public in order to facilitate testing. > | > | > | The comment about testing private methods was only a comment on the > | power of > | the framework, not on what should be done. IMHO you shouldn't be > | testing > | private methods w/o a very good reason. > | > | > | > I also appreciate the > | > ability to not only assert return values, but assert a series of > | > method calls and side effects. > | > > | > The danger with this is only testing in isolation using mocks -- If > | > the mocks don't follow the contract or conventions the > | > implementations > | > the tests are useless or deceptive. Implementation class behavior > | > can > | > change in ways that are invisible from an api perspective, so it's > | > important that these sorts of changes be reflected adequately by the > | > mocks themselves. > | > | > | Of course there is also the case where you end up just checking to > | make sure > | the object under test calls a series of methods on your mock - not a > | really > | informative test and bring of the correct interface issue you > | mentioned. > | > | > | > Coupled with adequate integration tests this danger > | > is minimized. Generally I think the tradeoff in increased > | > testability > | > makes up for these drawbacks. > | > > | > | +1 for good integration tests; they tend to mitigate that issue. > | > | > | > > | > Drew > | > > | > On Thu, Oct 27, 2011 at 12:49 PM, Jesse Yates > | > <[email protected]> > | > wrote: > | > > Hi all, > | > > > | > > Currently Accumulo doesn't have any mocking framework. However, in > | > > some > | > > recent work (ACCUMULO-53) doing good testing was greatly > | > > facilitated by > | > > adding in a mocking framework. They make it easy to only test the > | > > piece > | > of > | > > the architecture that we are interested in testing, helping to > | > > isolate > | > and > | > > ensure that we are really fixing problems in the code. > | > > > | > > So, I would like to propose adding the following test-time > | > > dependencies > | > for > | > > testing: > | > > > | > > EasyMock (http://easymock.org/) > | > > This is a really popular mocking framework for java and is really > | > > easy to > | > > use. I've rarely run into a situation that wasn't covered with > | > > what > | > EasyMock > | > > does. > | > > > | > > PowerMock (http://code.google.com/p/powermock/) > | > > This is the utility to cover what other mocking frameworks > | > > (including > | > > Mockito) can't do - capturing creation of objects (new > | > > Whatever(...)) and > | > > returning mocks instead, mocking static calls, mocking standard > | > > java lib > | > > calls, and accessing private methods/fields. > | > > > | > > This combination has proven to be incredibly effective and easy to > | > > use in > | > > the past, allowing for a really comprehensive testing suite. > | > > > | > > If you want to see an example of this stuff in action, check out > | > > the > | > patch > | > > 'java_ACCUMULO-53_v2.patch' on ACCUMULO-53. > | > > > | > > What does everyone think? > | > > > | > > Thanks! > | > > --Jesse > | > > > | > > ------------------- > | > > Jesse Yates > | > > 240-888-2200 > | > > @jesse_yates > | > > > | > > | > | > | > | -- > | ------------------- > | Jesse Yates > | 240-888-2200 > | @jesse_yates > -- ------------------- Jesse Yates 240-888-2200 @jesse_yates
