2007/10/3, Stepan Mishura <[EMAIL PROTECTED]>: > On 10/2/07, Tim Ellison <[EMAIL PROTECTED]> wrote: > > Stepan Mishura wrote: > > > On 10/2/07, Mikhail Loenko <[EMAIL PROTECTED]> wrote: > > >> Do we see any M3 blockers so far? > > > > > > As I understand there are no critical/blocker issues for r579330. > > > > ? there were a number of regression on that revision that were blockers > > and have been fixed. > > > > All fixes for issues raised on dev-list were included to the next > milestone candidate. Did I miss any? I'm not aware of any missed must-have patches, too.
> > > > Roughly, > > > - several regressions were found (most of them were fixed and included > > > into the new milestone candidate r580985), BWT, I encourage everyone > > > to use M3 umbrella for tracking [1] > > > > Agreed. I see two still open against that umbrella task. I think we > > can unlink HARMONY-4865 (tptp failures), but HARMONY-4184 looks like a > > candidate to fix, right? > > > > Alexey (Varlamov) did a quick fix for HARMONY-4184. > > Alexey, could you update its status? Did your quick fix(r580932) > resolve the issue? Surely it did, I think it is enough for M3. Complete fix better be done in M4, there is a bunch of issues around security policy which are risky to fix in last minute. > > > > - there are couple of suites to be improved (too many timeouts and > > > intermittent failures): Reliability/Stress/Functional > > > - automation for some suites should to be fixed: EGA/EUT > > > - few reports are missed: Geronimo Unit Tests (looks like regressions) > > > > This is all good stuff to capture in our list of things to do for M4. > > > > Yes, I hope to fix all these issues for M4 to make auto-testing more reliable. > > Thanks, > Stepan. > > > Hopefully we can declare M3 soonest and unfreeze the code. > > > > Regards, > > Tim > > > > > [1] https://issues.apache.org/jira/browse/HARMONY-4843 > > > > > > Thanks, > > > Stepan. > > > > > >> Thanks, > > >> Mikhail > > >> > > >> 2007/10/2, Stepan Mishura <[EMAIL PROTECTED]>: > > >>> Hi, > > >>> > > >>> Currently, the next milestone candidate (r580985) is under testing. > > >>> > > >>> There were a number of investigation reports for r579330 and I'd like > > >>> to update its summary to track the progress. All > > >>> > > >>> * Class library: > > >>> - Windows: 1 crash (hard to reproduce) > > >>> - Linux: 1 crash (HARMONY-4873), 1 failure (incorrect test - > > >>> HARMONY-4872) > > >>> * Eclipse 3.2 unit tests: > > >>> - Windows: > > >>> 4 failures - 1 is expected, 3 hard to reproduce (exclude list > > >>> was updated) > > >>> - Linux: > > >>> issue with > > >>> org.eclipse.jdt.core.tests.compiler.regression.TestAll suite > > >>> timeout still remains. > > >>> * Eclipse 3.3 unit tests: > > >>> Success rate on Windows - 99.77% and Linux 99.46%. > > >>> This results will be taken as baseline for tracking regressions. > > >>> * Eclipse Geronimo Application (EGA) scenario x 48h > > >>> - Windows: > > >>> failed after 8 running hours, most probably the problem > > >>> with scenario automation - no crashes/exceptions in the output > > >>> * Eclipse TPTP Tests: > > >>> - Linux: 1 intermittent failure > > >>> * Functional Test Suite: > > >>> 4 regressions were found - HARMONY-4866, HARMONY-4874, > > >>> HARMONY-4857, HARMONY-4861 > > >>> 6 tests are invalid - HARMONY-4864(2 cases), HARMONY-4859 (1 > > >>> case), JIRA for others?? > > >>> HARMONY-4867 - should be defined whether this is regression or not > > >>> Also there are many failures due to timeout or intermittent failures. > > >>> The suite definitely must be improved. > > >>> * Geronimo Unit Tests > > >>> 2 regressions on Windows/Linux - no evaluation yet > > >>> * JDKTools tests: > > >>> Windows: 1 failure - reproducible only if the whole suite is run. > > >>> (HARMONY-4868) > > >>> * Reliability Test Suite: > > >>> - Windows: 13 failures - according to the evaluation report most > > >>> of them are intermittent and additional investigation is required > > >>> - Linux: 14 failures - the same as for Windows > > >>> * Stress Test Suite: > > >>> - Windows: 3 failures > > >>> - Linux: 11 failures > > >>> Most of test failures due to timeout or OutOfMemoryError - the > > >>> suite must be improved. There is HARMONY-4114 with 1Mb patch for the > > >>> suite and harness but it is too risky to apply it during code freeze - > > >>> will be applied in M4. > > >>> * VTS VM Test Suite > > >>> - Linux: 1 intermittent failure - problems in DRLVM shutdown code > > >>> (HARMONY-????). > > >>> > > >>> Thanks, > > >>> Stepan. > > >>> > > >>> On 9/28/07, Stepan Mishura <[EMAIL PROTECTED]> wrote: > > >>>> Hi all, > > >>>> > > >>>> OK. Here is testing status for r579330 M3 milestone candidate. > > >>>> It is based on testing status page[1] and investigation reports for > > >>>> suites on dev-list. > > >>>> It is provided to decide whether r579330 is OK to be released or we > > >>>> should fix found issues and build next release candidate. > > >>>> > > >>>> Status reporter please correct me if I miss something. > > >>>> > > >>>> I'd like to say that this in this milestone we used BTI for the first > > >>>> time to get 100% of testing results. Some issues in BTI were fixed but > > >>>> there are some still to be resolved, for example instability of > > >>>> several suites, mostly Eclipse unit tests on Linux. > > >>>> > > >>>> > > >>>> The following suites passed on Linux/Windows ia32 platform: Ant > > >>>> Scenario (or self-hosting), Axis application, Dacapo, DRLVM tests, > > >>>> DRLVM regression tests, Jetty scenario, Scimark, Struts scenario, > > >>>> Tomcat scenario > > >>>> > > >>>> The the next list of issues we have (per suite): > > >>>> - Classlibrary unit tests: > > >>>> 1 crash on Windows that is hard to reproduce: > > >>>> 1 crash on Linux that is reproducible and it is bug in DRLVM > > >>>> (HARMONY-????): > > >>>> 1 test failure on Linux (incorrect test HARMONY-????) > > >>>> > > >>>> - Eclipse unit tests 3.2 > > >>>> Windows: 4 failures. 1 failure is expected (exclude list should be > > >>>> updated). 3 failure are not reproducible > > >>>> Linux: crash of > > >>>> org.eclipse.jdt.core.tests.compiler.regression.TestAll. It crashes due > > >>>> timeout > > >>>> > > >>>> - Eclipse unit tests 3.3 > > >>>> Windows: 49 failures, pass rate 99.77%. We haven't tried the suite > > >>>> on M2 so it is a good baseline for M3. > > >>>> Linux: the same problem as for 3.2. I've setup another CC Linux > > >>>> host for the suite and see no crashes, pass rate is 99.45% > > >>>> > > >>>> So I'm going to publish these reports for M3 and run 3.2 suite on > > >>>> these lucky host to obtain results without the crash. > > >>>> > > >>>> - Eclipse Geronimo Application (EGA) scenario x 48h > > >>>> Windows: there may be and issue with the scenario automation - the > > >>>> scenario run more then 8 hours. As I understand several iterations > > >>>> were done successfully. And it failed because of missing window, there > > >>>> are no crash/exceptions in the log. > > >>>> > > >>>> Also the scenario passed on Linux. > > >>>> > > >>>> - Eclipse TPTP Tests > > >>>> 1 failure on Linux - most probably intermittent > > >>>> > > >>>> - Functional Test Suite > > >>>> 21 failures on Windows most of then failure due to timeout. 1 > > >>>> regression (HARMONY-4857) > > >>>> Uploading Linux failed - I'm going to check why. From the first > > >>>> glance the results are same as for Windows. > > >>>> > > >>>> - Geronimo Unit Tests > > >>>> 2 possible(I haven't seen analysis report) regressions on > > >>>> Windows/Linux. These test passed on M2: > > >>>> org.apache.geronimo.tomcat.ContainerTest > > >>>> org.apache.geronimo.tomcat.JAASSecurityTest > > >>>> > > >>>> - JDKTools Tests > > >>>> 1 failure on Windows. The test stably fails when it runs in a bunch > > >>>> with others and may fail if run standalone : > > >>>> > > >>>> - Reliability Test Suite > > >>>> The status is a bit unclear to me. There is 1 test is incorrect > > >>>> test. And some tests failed because of timeout but its increase didn't > > >>>> help. > > >>>> Also some tests were removed from exclude list for r579330 but some > > >>>> of them failed. > > >>>> > > >>>> - Stress Test Suite > > >>>> 3 failures on Windows and 11 on Linux - I haven't seen analysis report > > >>>> > > >>>> - VTS VM Test Suite > > >>>> 1 failure on Linux - the test passed on previous snapshot should be > > >>>> analysed. > > >>>> > > >>>> Any comments are welcome. > > >>>> > > >>>> [1] > > >>>> http://people.apache.org/~mloenko/snapshot_testing/script/r579330/index.html > > >>>> > > >>>> Thanks, > > >>>> Stepan Mishura > > >>>> Intel Enterprise Solutions Software Division >
