I continue to really appreciate the information in these reports!
I don't know if the following is easy or hard, but thought I would
ask. I just spent some time trying to track down from which change
in the tests did a reoccuring regression come from. This would seem
the last piece of automated information that you could provide in
your current automatic "attempted failure analysis". I just spent
time doing this by hand which involved going to the full history page
and then doing binary search clicking down to into the details a
couple of levels and seeing when the bug started and if it was
consistent on that particular platform.
It would help a lot if the current report said something like:
test failed on xxx platform N out M times, first encountered in
test report XXX with svn changes YYY.
For instance I was trying to do some analysis on the stress memory
heap issue, and found:
test failed on linux platform 100% time (actually I didn't check
every run, just did a few probes), first encountered in
http://www.multinet.no/~solberg/public/Apache/Derby/Limited/testSummary-370210.html
3 changes happened in that test run:
http://www.multinet.no/~solberg/public/Apache/Derby/UpdateInfo/370210.txt
Of which I would look at 370059 as it makes an ij change with respect to
muti-threaded ij.
If the reports automatically recorded this kind of info it probably
would be less likely for regression issues to sit around.
[EMAIL PROTECTED] wrote:
[Auto-generated mail]
*Derby* 383955/2006-03-07 19:45:58 CET
*derbyall*
Failed Tests OK Skip Duration Platform
-------------------------------------------------------
*Jvm: 1.5*
3 643 640 0 103.82% CYGWIN_NT-5.1_i686-unknown
1 643 642 0 112.71% CYGWIN_NT-5.2_i686-unknown
1 643 642 0 109.33% Linux-2.4.21-27.ELsmp_i686-athlon
0 643 643 0 113.03% Linux-2.6.14-1.1644_FC4_i686-i686
1 643 642 0 99.60% SunOS-5.10_i86pc-i386
1 643 642 0 128.33% SunOS-5.10_sun4u-sparc
0 643 643 0 117.57% SunOS-5.9_sun4u-sparc
Details in http://www.multinet.no/~solberg/public/Apache/Derby/Limited/testSummary-383955.html
Attempted failure analysis in
http://www.multinet.no/~solberg/public/Apache/Failures/Derby/383955.html
*Jvm: 1.4*
2 640 638 2 102.31% CYGWIN_NT-5.1_i686-unknown
1 640 639 2 103.59% Linux-2.4.21-27.ELsmp_i686-athlon
1 640 639 2 113.38% Linux-2.6.14-1.1644_FC4_i686-i686
3 640 637 2 101.74% SunOS-5.10_i86pc-i386
Details in http://www.multinet.no/~solberg/public/Apache/DerbyJvm1.4/Limited/testSummary-383955.html
Attempted failure analysis in
http://www.multinet.no/~solberg/public/Apache/Failures/DerbyJvm1.4/383955.html
Changes in http://www.multinet.no/~solberg/public/Apache/Derby/UpdateInfo/383955.txt
(All results in http://www.multinet.no/~solberg/public/Apache/index.html)
