Good to hear from you again. Thank you for your kind offer of help. Given
the time I have to give to the task and the rustiness of my Unix skills, I
suspect it would take as much elapsed time for me to ferret out the files as
it would for you to gather them. When you get a chance, I'd appreciate it if
you could put them together for me in whatever format is easy for you to
Just as an introduction, one of the graphs I created shows the number of
tests per test suite and the number of failures and errors per test suite.
(The x axis is roughly 2^n.) The astonishing thing for me in this case is
the fairly clear power law distribution in the number of failures. Why in
the world would that be? The number of tests/suite is odd because the trend
is clear and then it drops off a cliff. In the corpus I have been studying
(~90M testcases run, ~50 person/years of effort), nobody has ever run a
suite with more than 11880 tests. Again, why?
This is the analysis I hope to expand with more data. I appreciate your
offer of assistance. I'll keep you posted as I continue my analysis.
From: Stefan Bodewig [mailto:[EMAIL PROTECTED]
Sent: Wednesday, September 10, 2008 6:17 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: Test data wanted
[I don't think there will be anything of sensitive nature in our
conversation so we could as well move over to the public [EMAIL PROTECTED]
On Tue, 9 Sep 2008, <[EMAIL PROTECTED]> wrote:
> In preparation for the next round of JUnit features, I am analyzing
> how people are currently running tests. I'm particularly interested in
> test suite size, test run times, and patterns of test success and
> failure. I found a few XML files associated with Gump that seem to
> have been generated by the junit Ant task or something with a
> compatible schema.
Yes, almost all JUnit tests run during a Gump run are run either inside of
Ant's <junit> task or Maven's surefire plugin. Many but probably not all of
them will be set up to create XML output.
My guess is that most of them still use JUnit 3 style tests.
> They would be excellent input for my analysis. However, it would be a
> lot of work for me to track down all such files associated with Gump.
> Can you think of an efficient way for me to gather all the test log
We used to have a <junitreport> tag in our Gump descriptors but it probably
isn't used too much.
If you want to enjoy installing a working Gump yourself, you'd "only"
have to grep through all XML files generated by a full run and search for
Alternatively I can do so for you, but this will only work if I pick the
right time (one Gump run finished and the next one hasn't started
yet) on vmgump - and have time to actually put together an archive.
It may take a few days before this happens.