> > - Reliability Test Suite > > The status is a bit unclear to me. There is 1 test is incorrect > > test. And some tests failed because of timeout but its increase didn't > > help.
Really it helped for 5 tests of 9, but 9 different tests failed. Almost all reliability tests failures are intermittent. And my local run for r579492 showed much lower failure ratio: 6 valid fails on Windows and 3 on Linux x86. So cruise control hosts could catch additional race conditions or similar problems somehow. > > Also some tests were removed from exclude list for r579330 but some > > of them failed. Also strange, they never failed on my test runs. Need additional investigation. > > Is anybody familiar enough with the history of the reliability and > stress test results to offer an opinion here? I watched reliability results, and they improved from 25-30 usual fails in August to 10-15 on the latest builds. More precisely: Date Windows-x86 Linux-x86 failures Sept/25 13 14 Sept/22 9 13 Sept/20 ?? 10 Sept/18 15 15 Sept/16 18 15 Sept/14 15 18 Sept/11 10 9 Sept/7 ?? 12 Thanks, Andrey On 9/28/07, Tim Ellison <[EMAIL PROTECTED]> wrote: > This is a wonderful summary, thanks Stepan. > > Comments in-lined below. > > Stepan Mishura wrote: > > Hi all, > > > > OK. Here is testing status for r579330 M3 milestone candidate. > > It is based on testing status page[1] and investigation reports for > > suites on dev-list. > > It is provided to decide whether r579330 is OK to be released or we > > should fix found issues and build next release candidate. > > > > Status reporter please correct me if I miss something. > > > > I'd like to say that this in this milestone we used BTI for the first > > time to get 100% of testing results. Some issues in BTI were fixed but > > there are some still to be resolved, for example instability of > > several suites, mostly Eclipse unit tests on Linux. > > Yes, this is getting a lot more testing, which is great to see. > > > The following suites passed on Linux/Windows ia32 platform: Ant > > Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests, > > DRLVM regression tests, Jetty scenario, Scimark, Struts scenario, > > Tomcat scenario > > Cool. > > > The the next list of issues we have (per suite): > > - Classlibrary unit tests: > > 1 crash on Windows that is hard to reproduce: > > 1 crash on Linux that is reproducible and it is bug in DRLVM > > (HARMONY-????): > > 1 test failure on Linux (incorrect test HARMONY-????) > > I looked into that one failing test in java.net and I maintain that the > test is prone to failure because it is using the deprecated > Support_PortManager. I created HARMONY-4872 to track it, but believe it > is not a blocking issue for M3. > > > - Eclipse unit tests 3.2 > > Windows: 4 failures. 1 failure is expected (exclude list should be > > updated). 3 failure are not reproducible > > Linux: crash of > > org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due > > timeout > > It crashes rather than fails when timing out? Is there a JIRA for this > one too that you know about? > > > - Eclipse unit tests 3.3 > > Windows: 49 failures, pass rate 99.77%. We haven't tried the suite > > on M2 so it is a good baseline for M3. > > Linux: the same problem as for 3.2. I've setup another CC Linux > > host for the suite and see no crashes, pass rate is 99.45% > > Looks good enough for where we are today. > > > So I'm going to publish these reports for M3 and run 3.2 suite on > > these lucky host to obtain results without the crash. > > > > - Eclipse Geronimo Application (EGA) scenario x 48h > > Windows: there may be and issue with the scenario automation - the > > scenario run more then 8 hours. As I understand several iterations > > were done successfully. And it failed because of missing window, there > > are no crash/exceptions in the log. > > > > Also the scenario passed on Linux. > > That is looking good too then. > > > - Eclipse TPTP Tests > > 1 failure on Linux - most probably intermittent > > > > - Functional Test Suite > > 21 failures on Windows most of then failure due to timeout. 1 > > regression (HARMONY-4857) > > Uploading Linux failed - I'm going to check why. From the first > > glance the results are same as for Windows. > > Would be good to figure out what is happening with HARMONY-4857. How > about we mark this as an M3 blocker until we understand it better? > > > - Geronimo Unit Tests > > 2 possible(I haven't seen analysis report) regressions on > > Windows/Linux. These test passed on M2: > > org.apache.geronimo.tomcat.ContainerTest > > org.apache.geronimo.tomcat.JAASSecurityTest > > JIRA issues for these too? > > > - JDKTools Tests > > 1 failure on Windows. The test stably fails when it runs in a bunch > > with others and may fail if run standalone : > > > > - Reliability Test Suite > > The status is a bit unclear to me. There is 1 test is incorrect > > test. And some tests failed because of timeout but its increase didn't > > help. > > Also some tests were removed from exclude list for r579330 but some > > of them failed. > > > > - Stress Test Suite > > 3 failures on Windows and 11 on Linux - I haven't seen analysis report > > Is anybody familiar enough with the history of the reliability and > stress test results to offer an opinion here? > > > - VTS VM Test Suite > > 1 failure on Linux - the test passed on previous snapshot should be > > analysed. > > > > Any comments are welcome. > > Good job! I think that although there are many failures reported, there > are many more tests and testing scenarios being exercised. We should > chase down the critical bugs for M3, but be prepared to triage the > others into the M4 task list. > > Regards, > Tim > > > [1] > > http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html > > > > Thanks, > > Stepan Mishura > > Intel Enterprise Solutions Software Division > > > -- Thanks, Andrey
