Another option would be to improve jatl and push the changes upstream. On 19/01/2013, at 0:34, Rene Groeschke <[email protected]> wrote:
> I've run the tests against with just the cherry picked commit on top of the > 1.4-rc-2. > > I think the difference between the testng and junit test results are caused > by the different template setup. the junit template creates just 10 test > classes and the testng one creates 100. > > I changed the template for my tests to use 1000 testclasses in the withJunit > and withTestNg template and 100 Test classes in the verboseJunit and > verboseTestNg. > > See the results below: > > Test | 1.4-rc1 | JATL | Difference > --------------------------------------------------- > withTestNG | 41.524s | 41.914s | +0.94 % > withJUnit | 26.554s | 26.33s | -0.85 % > withVerboseTestNG | 2.248m | 2.36m | +5.01 % > withVerboseJUnit | 2.276m | 2.29m | +0.61 % > > Test | 1.4-rc1 | JATL | Difference > ---------------------------------------------------- > withTestNG | 23.647MB | 24.447MB| +3.38 % > withJUnit | 20.461MB | 19.689MB| -3.77 % > withVerboseTestNG | 20.009MB | 23.405MB| +16.97% > withVerboseJUnit | 20.969MB | 24.584MB| +17.24% > > I can't see a real benefit of porting our current solution to jatl. I'm > suprised to see, that the verbose test output with the jatl based solution > needs more memory as the former dom based one. Maybe I can tweak the stderr > and stdout a bit more, but I don't really see a potential for optimisation > here atm. > > My next option is to spike the usage of our SimpleXmlWriter to generate the > test reports. > > > cheers! > René > -- > Principal Engineer, > Gradleware Inc. - Gradle Training, Support, Consulting > [email protected] > http://gradleware.com > > > > > Adam Murdoch wrote: >> >> On 18/01/2013, at 9:38 AM, Rene Groeschke wrote: >> >>> Hi devs, >>> >>> As part of story to make the hmtl report generation more >>> efficient(https://github.com/gradle/gradle/blob/master/design-docs/testing-improvements.md#story-html-test-report-generation-is-efficient), >>> I've spiked porting the test html report generation in gradle to use >>> jatl (http://code.google.com/p/jatl/). >>> >>> Especially when dealing with verbose tests (lot of stdout and stderr) >>> gradle should benefit from that. >>> >>> I've pushed this spike on a seperate branch for the moment >>> (https://github.com/gradle/gradle/tree/jatl-testreports). >>> >>> Here are the performance test results: >>> >>> Speed results: >>> Test | 1.4-rc-1| JATL | Difference >>> --------------------------------------------------- >>> withTestNG | 9.842s | 10.087s | +2.49 % >>> withJUnit | 8.042s | 8.364s | +4.01 % >>> withVerboseTestNG | 20.222s | 21.334s | +5.50 % >>> withVerboseJUnit | 18.487s | 17.726s | -4.12 % >>> withVerboseTestNG*| 1.803m | 1.793m | -0.58 % >>> withVerboseJUnit* | 1.829m | 1.601m | -12.51 % >>> >>> Memory results: >>> Test | 1.4-rc1 | JATL | Difference >>> ---------------------------------------------------- >>> withTestNG | 22.657MB | 26.108MB| +15.23 % >>> withJUnit | 25.854MB | 25.386MB| -1.81 % >>> withVerboseTestNG | 22.738MB | 25.048MB| +10.16 % >>> withVerboseJUnit | 25.175MB | 23.927MB| -4.96 % >>> withVerboseTestNG*| 24.263MB | 24.8MB | +2.21% % >>> withVerboseJUnit* | 24.646MB | 23.869MB| -3.15 % >>> >>> * modified to output 10000 lines of stdout and stderr per Test >>> >>> As you can see especially verbose junit tests do benefit from the >>> change, but I wonder where the weak results for testng comes from. >>> From my point of view, I havn't seen any other change in the current >>> master that explains the difference, since we use the same report >>> infrastructure for generating the html reports for junit and testng. >> >> Perhaps run the tests again, against a distribution built from the git >> revision just before you made the change, to isolate your changes from >> other changes in master since 1.4-rc-1. >> >> >> -- >> Adam Murdoch >> Gradle Co-founder >> http://www.gradle.org >> VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting >> http://www.gradleware.com > > > --------------------------------------------------------------------- > To unsubscribe from this list, please visit: > > http://xircles.codehaus.org/manage_email > > --------------------------------------------------------------------- To unsubscribe from this list, please visit: http://xircles.codehaus.org/manage_email
