I've run the tests against with just the cherry picked commit on top of
the 1.4-rc-2.
I think the difference between the testng and junit test results are
caused by the different template setup. the junit template creates just
10 test classes and the testng one creates 100.
I changed the template for my tests to use 1000 testclasses in the
withJunit and withTestNg template and 100 Test classes in the
verboseJunit and verboseTestNg.
See the results below:
Test | 1.4-rc1 | JATL | Difference
---------------------------------------------------
withTestNG | 41.524s | 41.914s | +0.94 %
withJUnit | 26.554s | 26.33s | -0.85 %
withVerboseTestNG | 2.248m | 2.36m | +5.01 %
withVerboseJUnit | 2.276m | 2.29m | +0.61 %
Test | 1.4-rc1 | JATL | Difference
----------------------------------------------------
withTestNG | 23.647MB | 24.447MB| +3.38 %
withJUnit | 20.461MB | 19.689MB| -3.77 %
withVerboseTestNG | 20.009MB | 23.405MB| +16.97%
withVerboseJUnit | 20.969MB | 24.584MB| +17.24%
I can't see a real benefit of porting our current solution to jatl. I'm
suprised to see, that the verbose test output with the jatl based
solution needs more memory as the former dom based one. Maybe I can
tweak the stderr and stdout a bit more, but I don't really see a
potential for optimisation here atm.
My next option is to spike the usage of our SimpleXmlWriter to generate
the test reports.
cheers!
René
--
Principal Engineer,
Gradleware Inc. - Gradle Training, Support, Consulting
[email protected]
http://gradleware.com
Adam Murdoch wrote:
On 18/01/2013, at 9:38 AM, Rene Groeschke wrote:
Hi devs,
As part of story to make the hmtl report generation more
efficient(https://github.com/gradle/gradle/blob/master/design-docs/testing-improvements.md#story-html-test-report-generation-is-efficient),
I've spiked porting the test html report generation in gradle to use
jatl (http://code.google.com/p/jatl/).
Especially when dealing with verbose tests (lot of stdout and stderr)
gradle should benefit from that.
I've pushed this spike on a seperate branch for the moment
(https://github.com/gradle/gradle/tree/jatl-testreports).
Here are the performance test results:
Speed results:
Test | 1.4-rc-1| JATL | Difference
---------------------------------------------------
withTestNG | 9.842s | 10.087s | +2.49 %
withJUnit | 8.042s | 8.364s | +4.01 %
withVerboseTestNG | 20.222s | 21.334s | +5.50 %
withVerboseJUnit | 18.487s | 17.726s | -4.12 %
withVerboseTestNG*| 1.803m | 1.793m | -0.58 %
withVerboseJUnit* | 1.829m | 1.601m | -12.51 %
Memory results:
Test | 1.4-rc1 | JATL | Difference
----------------------------------------------------
withTestNG | 22.657MB | 26.108MB| +15.23 %
withJUnit | 25.854MB | 25.386MB| -1.81 %
withVerboseTestNG | 22.738MB | 25.048MB| +10.16 %
withVerboseJUnit | 25.175MB | 23.927MB| -4.96 %
withVerboseTestNG*| 24.263MB | 24.8MB | +2.21% %
withVerboseJUnit* | 24.646MB | 23.869MB| -3.15 %
* modified to output 10000 lines of stdout and stderr per Test
As you can see especially verbose junit tests do benefit from the
change, but I wonder where the weak results for testng comes from.
From my point of view, I havn't seen any other change in the current
master that explains the difference, since we use the same report
infrastructure for generating the html reports for junit and testng.
Perhaps run the tests again, against a distribution built from the git
revision just before you made the change, to isolate your changes from
other changes in master since 1.4-rc-1.
--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
