brett asked me to look into the idea of a plugin testing framework and having mulled it over a bit and talked to some folks about it I wanted to spill out my thoughts here and a couple of stabs at breaking the nut cleanly. Also, in the interests of having people read this and not have it drag on I'll jump right to the chase.
We need someway to be able to gather code coverage metrics from plugin testing so we have at least a modicum of belief that changes didn't break anything, and since plugins can be so amazingly complex in many cases and just downright trivial in others there can't be a one size fits all solution....so I propose that we break the problem down into two neat little parts. First would be actual plugin unit testing were we have a simple way to instantiating the plugin and any of the pretty normal variables that get injected, collections, strings, bool, etc. then we have basic junit type testing on these objects, either through careful use of the execute method on the mojo's or just being able to test the supporting methods like special filename manipulation, downloading a wsdl from a website, perhaps even generating some code. Basically, anything and everything that you can massage on the plugin that doesn't stray into complex objects like MavenProject, or ArtifactManagers or anything. Now, some of plugins can be completely tested by this mechanism while others might not actually fit too tell into this lower level testing. That is where the integration testing comes into more of a play. I talked to john about this and we were kind of a mind that stubbing and mocking up these project objects and whatnot don't represent reality very well and perhaps the best solution would be to follow the route of the integration plugin where we can craft little projects in the plugin directory that represent real life usages of the plugin, then we verify that the right thing was done. This verification could take the form of validating the resulting directory structure matched an outcome, certain files matched an outcome, or the state of objects in the project matched the template. This all seems pretty obvious, but I wanted to at least start a conversation about how we ought to test these plugins. I like this clear delineation in that it draws a line in the sand as to what can get tested where between src/test/java and src/it/*. It also encourages a clean plugin design as the unit testing should make it simple for you to easily test the checking of variable existence and states and helper methods. Although, I know vincent has had some success with mocking up some of the more complex objects so I would be interested in his thoughts on the matter...and bretts as well on if I am barking up the wrong tree... so, thoughts? jesse -- jesse mcconnell jesseDOTmcconnellATgmailDOTcom
