|
The duration of the test results in the JUnit Plugin is just the time that is being reported on the <testcase .../> line. This time contains only the time it took to run the @Before, @Test and @After annotated methods. The time that it took for the class creation, the object creation and for the @BeforeClass and @AfterClass annotated methods will show up in the <testsuite .../> line. The way things are now, there are cases with crappy tests that show up as decently fast, just because a large bunk of the time is not taken into account. Would it be possible to add this time as well?
Sample file with the times from a test class (class code attached):
<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="0" failures="0" hostname="localhost" name="com.mypackage.SomeTest" skipped="0" tests="3" time="13.75" timestamp="2014-11-03T15:08:13">
<testcase classname="com.mypackage.SomeTest" name="testSomething1" time="0.215" />
<testcase classname="com.mypackage.SomeTest" name="testSomething2" time="0.211" />
<testcase classname="com.mypackage.SomeTest" name="testSomething3" time="0.211" />
</testsuite>
Screenshot1.png
This is the snapshot of the plugin output. As you can see, it only takes into account the ~210ms from before, after and test. The bulk of the time however (13sec vs 630ms) is spent in other parts of the test class and that time is not accounted for.
Screenshot2.png
This is a snapshot from a modified version of the plugin. This modified version will take the extra time and distribute it evenly between all the tests in the suite.
So, if this is something that would be considered good to include in the plugin, my questions are:
1. Would there be a need to make this configurable (as in provide an option when publishing junit results about what to do with the extra time)?
2. Which would be the "preferred" way to include the time. I've played with more versions:
- create a new <init> test case that would be added to the suite and contain the extra time (clear extra time, but adds a non existent test case)
- distribute the extra time evenly to the test cases in the test suite
- distribute the extra time according to the time weight to the test cases in the test suite (a test case taking 2ms would get twice the amount of extra time compared to a test case taking 1ms).
|
