Stepan Mishura wrote: > Currently, the next milestone candidate (r580985) is under testing.
Thanks. Please start a new thread with the summary of its testing so we (or 'I' <g>) don't get confused about which tests we are talking about. > There were a number of investigation reports for r579330 and I'd like > to update its summary to track the progress. All Just to be clear, these are the summaries for the older build then... Regards, Tim > * Class library: > - Windows: 1 crash (hard to reproduce) > - Linux: 1 crash (HARMONY-4873), 1 failure (incorrect test - HARMONY-4872) > * Eclipse 3.2 unit tests: > - Windows: > 4 failures - 1 is expected, 3 hard to reproduce (exclude list > was updated) > - Linux: > issue with org.eclipse.jdt.core.tests.compiler.regression.TestAll > suite > timeout still remains. > * Eclipse 3.3 unit tests: > Success rate on Windows - 99.77% and Linux 99.46%. > This results will be taken as baseline for tracking regressions. > * Eclipse Geronimo Application (EGA) scenario x 48h > - Windows: > failed after 8 running hours, most probably the problem > with scenario automation - no crashes/exceptions in the output > * Eclipse TPTP Tests: > - Linux: 1 intermittent failure > * Functional Test Suite: > 4 regressions were found - HARMONY-4866, HARMONY-4874, > HARMONY-4857, HARMONY-4861 > 6 tests are invalid - HARMONY-4864(2 cases), HARMONY-4859 (1 > case), JIRA for others?? > HARMONY-4867 - should be defined whether this is regression or not > Also there are many failures due to timeout or intermittent failures. > The suite definitely must be improved. > * Geronimo Unit Tests > 2 regressions on Windows/Linux - no evaluation yet > * JDKTools tests: > Windows: 1 failure - reproducible only if the whole suite is run. > (HARMONY-4868) > * Reliability Test Suite: > - Windows: 13 failures - according to the evaluation report most > of them are intermittent and additional investigation is required > - Linux: 14 failures - the same as for Windows > * Stress Test Suite: > - Windows: 3 failures > - Linux: 11 failures > Most of test failures due to timeout or OutOfMemoryError - the > suite must be improved. There is HARMONY-4114 with 1Mb patch for the > suite and harness but it is too risky to apply it during code freeze - > will be applied in M4. > * VTS VM Test Suite > - Linux: 1 intermittent failure - problems in DRLVM shutdown code > (HARMONY-????). > > Thanks, > Stepan. > > On 9/28/07, Stepan Mishura <[EMAIL PROTECTED]> wrote: >> Hi all, >> >> OK. Here is testing status for r579330 M3 milestone candidate. >> It is based on testing status page[1] and investigation reports for >> suites on dev-list. >> It is provided to decide whether r579330 is OK to be released or we >> should fix found issues and build next release candidate. >> >> Status reporter please correct me if I miss something. >> >> I'd like to say that this in this milestone we used BTI for the first >> time to get 100% of testing results. Some issues in BTI were fixed but >> there are some still to be resolved, for example instability of >> several suites, mostly Eclipse unit tests on Linux. >> >> >> The following suites passed on Linux/Windows ia32 platform: Ant >> Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests, >> DRLVM regression tests, Jetty scenario, Scimark, Struts scenario, >> Tomcat scenario >> >> The the next list of issues we have (per suite): >> - Classlibrary unit tests: >> 1 crash on Windows that is hard to reproduce: >> 1 crash on Linux that is reproducible and it is bug in DRLVM >> (HARMONY-????): >> 1 test failure on Linux (incorrect test HARMONY-????) >> >> - Eclipse unit tests 3.2 >> Windows: 4 failures. 1 failure is expected (exclude list should be >> updated). 3 failure are not reproducible >> Linux: crash of >> org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due >> timeout >> >> - Eclipse unit tests 3.3 >> Windows: 49 failures, pass rate 99.77%. We haven't tried the suite >> on M2 so it is a good baseline for M3. >> Linux: the same problem as for 3.2. I've setup another CC Linux >> host for the suite and see no crashes, pass rate is 99.45% >> >> So I'm going to publish these reports for M3 and run 3.2 suite on >> these lucky host to obtain results without the crash. >> >> - Eclipse Geronimo Application (EGA) scenario x 48h >> Windows: there may be and issue with the scenario automation - the >> scenario run more then 8 hours. As I understand several iterations >> were done successfully. And it failed because of missing window, there >> are no crash/exceptions in the log. >> >> Also the scenario passed on Linux. >> >> - Eclipse TPTP Tests >> 1 failure on Linux - most probably intermittent >> >> - Functional Test Suite >> 21 failures on Windows most of then failure due to timeout. 1 >> regression (HARMONY-4857) >> Uploading Linux failed - I'm going to check why. From the first >> glance the results are same as for Windows. >> >> - Geronimo Unit Tests >> 2 possible(I haven't seen analysis report) regressions on >> Windows/Linux. These test passed on M2: >> org.apache.geronimo.tomcat.ContainerTest >> org.apache.geronimo.tomcat.JAASSecurityTest >> >> - JDKTools Tests >> 1 failure on Windows. The test stably fails when it runs in a bunch >> with others and may fail if run standalone : >> >> - Reliability Test Suite >> The status is a bit unclear to me. There is 1 test is incorrect >> test. And some tests failed because of timeout but its increase didn't >> help. >> Also some tests were removed from exclude list for r579330 but some >> of them failed. >> >> - Stress Test Suite >> 3 failures on Windows and 11 on Linux - I haven't seen analysis report >> >> - VTS VM Test Suite >> 1 failure on Linux - the test passed on previous snapshot should be >> analysed. >> >> Any comments are welcome. >> >> [1] >> http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html >> >> Thanks, >> Stepan Mishura >> Intel Enterprise Solutions Software Division >
