Stefan, Good to hear from you again. Thank you for your kind offer of help. Given the time I have to give to the task and the rustiness of my Unix skills, I suspect it would take as much elapsed time for me to ferret out the files as it would for you to gather them. When you get a chance, I'd appreciate it if you could put them together for me in whatever format is easy for you to create.
Just as an introduction, one of the graphs I created shows the number of tests per test suite and the number of failures and errors per test suite. (The x axis is roughly 2^n.) The astonishing thing for me in this case is the fairly clear power law distribution in the number of failures. Why in the world would that be? The number of tests/suite is odd because the trend is clear and then it drops off a cliff. In the corpus I have been studying (~90M testcases run, ~50 person/years of effort), nobody has ever run a suite with more than 11880 tests. Again, why? This is the analysis I hope to expand with more data. I appreciate your offer of assistance. I'll keep you posted as I continue my analysis. Cheers, Kent -----Original Message----- From: Stefan Bodewig [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 10, 2008 6:17 AM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: Test data wanted Hi Kent, [I don't think there will be anything of sensitive nature in our conversation so we could as well move over to the public [EMAIL PROTECTED] list.] On Tue, 9 Sep 2008, <[EMAIL PROTECTED]> wrote: > In preparation for the next round of JUnit features, I am analyzing > how people are currently running tests. I'm particularly interested in > test suite size, test run times, and patterns of test success and > failure. I found a few XML files associated with Gump that seem to > have been generated by the junit Ant task or something with a > compatible schema. Yes, almost all JUnit tests run during a Gump run are run either inside of Ant's <junit> task or Maven's surefire plugin. Many but probably not all of them will be set up to create XML output. My guess is that most of them still use JUnit 3 style tests. > They would be excellent input for my analysis. However, it would be a > lot of work for me to track down all such files associated with Gump. > Can you think of an efficient way for me to gather all the test log > files? We used to have a <junitreport> tag in our Gump descriptors but it probably isn't used too much. If you want to enjoy installing a working Gump yourself, you'd "only" have to grep through all XML files generated by a full run and search for <testsuite> 8-) Alternatively I can do so for you, but this will only work if I pick the right time (one Gump run finished and the next one hasn't started yet) on vmgump - and have time to actually put together an archive. It may take a few days before this happens. Stefan
