We use the perl test::class framework to create functional/system level tests which we don't run in Jenkins. Our tests have specific HW targets we need to run against. These HW targets are shared so cannot be dedicated to CI builds. Also some of the tests take a long time to run. My typical workflow is to run them on my computer then generate an aggregate html report to review or publish to a web site.
I would be surprised that removing 'testsuites' would break compatibility with Jenkins. I have Jenkins running maven (java) projects with lots of junit tests. When maven executes the tests it generates lots of individual junit xml reports, none of which contain the 'testsuites' element. All the individual xml reports get fed to Jenkins and Jenkins seems to generate its test results page just fine. -Khai -----Original Message----- From: Graham TerMarsch [mailto:c...@howlingfrog.com] Sent: Monday, June 27, 2011 1:13 PM To: Khai Do; perl-qa@perl.org Subject: Re: testsuites element in TAP::Formatter::JUnit reports On June 27, 2011, Khai Do wrote: > I am attempting to aggregate all of my (junit) tap reports and generate a > html report using Ant’s junitreport task. However the junitreport task > fails with "not a valid testsuite XML document" because the tap reports > are not in a standard junit format. I've also tried using maven to > generate html reports and it also fails in the same way. So yes, the tap > xml file is compatible with Jenkins but not ant or maven. If the > 'testsuites' element is removed it will be compatible with all the tools. > Also I think it would look nicer in the Jenkins test reports. IIRC, the use of the <testsuites> element is standard when using nested JUnit tests. Unfortunately, removing the <testsuites> element from the JUnit output that T:F:JUnit (and TAP::Harness::JUnit) create would break compatibility with Hudson/Jenkins (which is what these packages were designed to interoperate with). In order for the output to contain multiple <testsuite> elements, Hudson/Jenkins requires that they be wrapped in a <testsuites> element to indicate that multiple tests are present. > I entered this bug on your project: > https://rt.cpan.org/Public/Bug/Display.html?id=69058 Included is a link to > the junit schema and a sample ant project showing the error. > > Right now my workaround is exactly as you suggested. I have an ant task > that strips the 'testsuites' element before running generating the html > report. I appreciate the report, but to be honest am inclined to suggest that you simply continue to use your workaround; as noted in the documentation this module is designed for integration w/a Hudson/Jenkins instance, and I'm hesitant to update the module in such a way that it would break said compatibility. In all honesty, you're the first person I've heard from who has tried using the module to integrate TAP output via an Ant junitreport task; everyone else I'm heard from or have worked with has specifically been using Hudson/Jenkins as the reporter. -- Graham TerMarsch
