On Jul 2, 2012, at 6:58 PM, Vincent Massol wrote: > > On Jul 2, 2012, at 6:53 PM, Vincent Massol wrote: > >> Hi devs, >> >> I've finally spent the time to measure our TPC (Test Percentage Coverage) >> using Clover. >> >> The reports are very interesting and show where we need to spend our efforts >> in testing (be them automated or manual). >> >> Here are the raw top level results: >> =========================== >> >> All results: http://maven.xwiki.org/site/clover/20120701/ >> >> * Only Commons (this means unit tests and integration tests in the commons >> repository only): 62.4% >> * Only Rendering: 75.5% >> * Only Platform: 36.8% >> * Commons+Rendering (ie Rendering tests exercise also Commons code): 72.1% >> * Commons+Rendering+Platform: 47.4% >> * Commons+Rendering+Platform+Enterprise: > > The value is 63.1% > >> >> So our code is roughly tested at X% (note that 100% TPC doesn't mean the >> code is working, it means each line of code has been tested in some way but >> it doesn't mean all combinations have been tested!). > > s/X/63.1%/ > > I also forgot to mention that several functional tests failed, almost in each > functional test module… > > Thanks > -Vincent > >> A good value is between 70-80%. >> >> We're actually better than I thought (I had 50% or lower in mind) :) However >> there are huge disparity with code having 90+% and code having 0% coverage >> (402 classes have 0% coverage!). >> >> Quick high level analysis: >> ==================== >> >> * Commons is missing quite a lot of unit tests >> * Wikimodel in Rendering is dragging the TPC down, I think we were above 80% >> before we included WikiModel) >> * Platform has a very low level of unit tests which means it's easy to break >> stuff in platform and it's hard to verify that what we add in platform works >> well in general (means going through longer stabilization phases than needed) >> * Our functional tests are quite effective (even though they take a very >> very long time to execute) since they make the TPC go from 47% to 63% for >> Commons+Rendering+Platform. The bad TPC for Platform is probably improved by >> the functional tests. >> >> Next steps >> ========= >> >> * IMO all developers need to look in details at the results and familiarize >> themselves with the Clover reports >> * Each dev should look at stuff he has coded and ensure that they have a >> good level of unit tests >> * All new code should get at the very minimum between 70-80% unit test >> coverage. For example the new diff module in commons only has only 43.4% >> coverage which is far from enough >> * Spend time to automate the generation of these reports. We need a >> dedicated agent for this since it "contaminates" the local repository with >> clovered jars. The overall generation time is around 4-5 hours. >> * Dig more around the reports to identify functional areas of XWiki missing >> automated tests. These areas must be covered by manual testing at each >> release. >> * Decide if we want a more rigorous strategy for new code, and how to get >> alerts for modules missing coverage.
List of XWiki modules that are not tested or badly tested: * Portlet * Some Actions: Download revision, Delete version, Reset version, Tex Action (mathematic formula rendering), View Attach Rev, Redirect, Prop Enable, Prop Disable, etc * GWT API * Feed plugin * Chart plugin (we need to remove it IMO) * Rights Manager plugin * Database Migrations are not tested at all by automated tests * User directory plugin (do we still need it?) * Stats * Repository and Extension missing some tests but Thomas/Marius are adding them * Monitor plugin/api * Workspaces * Autotag plugin * WikiManager plugin * Global Search plugin (didn't even know about it!) * ApplicationManager plugin * Graphviz plugin * SVG plugin * Usertools plugin * Openoffice importer * The oldcore model is not that well tested (around 50-60% which is low for something so important) * Activity stream is not well tested * Rendering 1.0 is not that well tested (but we don't care ;)) * Mail plugin (probably need to be retired and possibly replaced by the work done by Jeremie) Note that it's hard to find "features" from the TPC since feature are cross-modules. IMO a lot of these modules need to be moved out of the platform and moved to contrib and not be bundled by default. Thanks -Vincent >> >> WDYT? >> >> Let's share your analysis in this thread so that we can brainstorm about >> what to do with these reports >> >> Thanks >> -Vincent >> > _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
