On 12/18/07, Stepan Mishura <[EMAIL PROTECTED]> wrote: > Hi Andrey, > > Thanks for your evaluation. See my comments below. > > On 12/17/07, Andrey Yakushev <[EMAIL PROTECTED]> wrote: > > My comments on the reliability suite results: > > > > > * Reliability > > > > There are several tests with known intermittent failures in this suite > > which were not put to exclude list in order to control new regression > > appearance. Thus several failures are expected. > > > > > > > > Failed on x86 platforms but they also failed on M3 > > > api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it) > > > > That's because BTI workspace wasn't updated on CC host. > > > > OK. I'll fix it. > > > > api.net.SingleConnectTest > > > > Known intermittent test issue, the reason was described in HARMONY-5266. > > > > HARMONY-5266 is about HttpConnectionTest.
Yes, But I noted that the same race condition exists in SingleConnectTest and MultiConnectTest. I will open the corresponding JIRA ASAP. > > > > > > > - Linux_x86: > > > > > > Regresions: > > > api.kernel.object.ObjectFinalizeTest > > > > exception after 1705 successful iterations - probably regression - I > > couldn't reproduce it yet in my local runs. > > > > > api.kernel.throwable.StackTraceExcptsTest > > > > hang - could be similar to HARMONY-5288 issue > > > > > api.text.DecimalFormat_Locales > > > > hang - could be similar to HARMONY-5288 issue > > > > > api.zip.ZipEntryIterateThreadingTest > > > > Enable to open file - I saw that on M3 - not a regression. > > > > Why not to exclude tests that fails on M3 too? It's infrequent intermittent failure, presumably covered by HARMONY-4215. It was not excluded for new regressions monitoring. By the way, all such rare failures were excluded for Linux x86_64 platform and it's pretty long list - 18 tests currently. > > > > api.zip.ZlibTest > > > > Known intermittent failure HARMONY-4215, HARMONY-5166 - not a regression > > > > > vm.finalization.FinalizeThrowRestoreTest > > > > VM crash after 142 successful iterations - new issue, could be similar > > to HARMONY-5304 - regression > > > > > vm.stack.StackUnwindTest > > > > Hang after 1025 successful iterations - I saw it previously on M3, > > probably HARMONY-4215 - not a regression > > > > > > > > They also failed on M3: > > > api.nio.channels.filechannel.FileChannelMapTest > > > api.nio.channels.filechannel.MappedByteBufferForceTest > > > api.serialization.SerializableClassesTest > > > > > > > My local run but on the debug release gives only 3 failures: > > > > I think the snapshot build should be tried first to reproduce the failures. Yes, you are right, but I used default "drlvm" as usual, and it turned out to be the debug build. It was not M4 release build checking, it was my usual status checking over weekend. > > > api.kernel.thread.VolatileVariableTest.PetersonTest - hang 32223 > > successful iterations - I saw that intermittently on M3 - not > > regression - probably this is HARMONY-4215 > > api.nio.channels.filechannel.MappedByteBufferForceTest - not regression > > -was on > > vm.finalization.FinalizeThrowRestoreTest - assertion after 66 > > successful iterations - could be similar to HARMONY-5304 > > > > There are also several known failures in valid excluded tests (they > > were not run on CC) > > > > api.kernel.thread.LifeSimulationTest.Life > > vm.classloading.ClassCastTest > > vm.classloading.ClassReflectionTest > > > > > - Windows_x86 > > > > > > Regresions: > > > api.kernel.string.InternThreadingTest - regression > > > api.kernel.thread.Calculation.CalcTest > > > api.kernel.thread.RecursiveThreadTest > > > api.kernel.thread.StackTraceTest > > > api.kernel.thread.Synchronization > > > api.kernel.thread.ThreadArrayTest > > > api.kernel.threadgroup.EnumerateTest > > > api.net.DatagramTest > > > api.serialization.SerializableClassesTest > > > vm.stack.StackUnwindTest > > > > > > > My local run results on the release build showed mostly different failures: > > > > Did you mean snapshot build? Yes, here I used it via "jvm.under.test" option. > > api.net.NetClient - not regression - known intermittent failure - > > HARMONY-4215 > > api.net.SingleConnectTest - test issue - HARMONY-5266 > > api.net.DatagramTest - hang on iteration #4 - probably HARMONY-4215, > > but could be a regression > > api.io.IOTest - i/o failure - require investigation - probably regression > > api.kernel.thread.ThreadArrayTest.ThreadArrayTest - OutOfMemoryError > > after 958 successful iterations > > api.kernel.thread.Calculation.CalcTest - OutOfMemoryError after 17317 > > successful iterations > > api.kernel.object.ObjectFinalizeTest - exception after 461 successful > > iterations > > > > There are also several failures in excluded tests (they were not run on CC) > > > > vm.classloading.ClassReflectionTest - not regression - HARMONY-3680 > > vm.classloading.ClassAttributesTest - not regression - HARMONY-3680 > > api.kernel.thread.LifeSimulationTest.Life - not regression - > > OutOfMemoryError after 21507 successful iterations - HARMONY-5266 > > api.kernel.hooks.AddRmPropertiesHooksTest - regression - HARMONY-5172 > > > > > - Linux_x86_64: > > > > > > Regressions: > > > vm.classloading.ClassCastTest > > > > This is HARMONY-5304.I suggest fixing it now if possible. > > Is it blocker for releasing M4? I don't think so, it's rather uncommon failure. I saw Pavel is trying to fix this. But it would be nice to fix it, because this is a regression as compared with M3. > > > > > My local run results on the debug build also showed 7 failures only in > > excluded tests (they were not run on CC): > > > > api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest > > api.kernel.thread.LifeSimulationTest.Life > > api.nio.channels.filechannel.MappedByteBufferForceTest > > api.net.DatagramTest > > vm.classloading.ClassAttributesTest > > vm.classloading.ClassReflectionTest > > vm.classloading.DelegationModelTest > > > > Details are available at > > http://wiki.apache.org/harmony/Reliability_Tests_on_DRLVM > > > > Wow, the wiki page really informative! (BTW, s/bag/bug/ should be applied to > it) Corrected. > > > My resume: > > > > My tests run results: > > > > Platform | failures | regressions | pass rate > > Windows x86 | 11 failures | 4 regressions | 87.2% > > Linux x86 | 6 failures | 1 regressions | 93% > > Linux x86_64 | 7 failures | 1 regression | 92% > > > > This is better results then M3 had. Usually on M3 we had had 11-13 > > failures: > > http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html > > > > So I think the reliability in M4 was improved in general although we > > still have an opportunity for improvements :) > > > > Thanks for the summary! > > -Stepan. > > > Thanks, > > Andrey > > > > > > On 12/14/07, Stepan Mishura <[EMAIL PROTECTED]> wrote: > > > Hi, > > > > > > Please see testing status[1] for r603534 below. > > > Note to my comments: > > > - I marked tests as regression if they passed on M3 [2] > > > - I question tests as intermittent if they passed on the previous > > > snapshot (r603433) [3] > > > - There are some tests that failed on M3 too. (move them to the exclude > > > lists?) > > > - Not all failures have JIRA issue assigned so please help me to find > > > related JIRA number (if there is any). > > > > > > All comments and clarifications are greatly appreciated! > > > > > > ===================================================== > > > We have the next suites failed on the snapshot. > > > > > > * Eclipse Unit Tests 3.3 > > > - Linux_x86: 4 failures > > > - Windows_x86: pass rate 96.92% > > > - Linux_x86_64: pass rate 95.29% > > > > > > Would be nice to get failures evaluation. > > > We definitely improved stability of the Eclipse Unit Test suite since > > > M3 (Thanks to Vladimir Beliaev). > > > But the are still some issues such as intermittent suite crashes (that > > > make me very upset). > > > > > > * EGAx48: > > > - Linux_x86 & Linux_x86_64 : failed. (HARMONY-5250 / regression?) > > > - Windows_x86: > > > Still no status – the scenario causes CC host down after running > > > several hours. > > > Currently I can not figure out if this is the host's issue or > > > problem introduced by Harmony. > > > > > > > > > * EHWA(GUI) > > > - Linux_x86 & Linux_x86_64 : failed. > > > The scenario is quite unstable. Most probably I'll drop the > > > scenario in M5 from snapshot test and use only EHWA-API. > > > > > > > > > * Functional: > > > F_KeyFactoryTest_01: regression on all platforms (HARMONY-4857) > > > > > > - Linux_x86: > > > JComponent: intermittent? > > > - Linux_x86_64: > > > btest6353: fails on M3 too > > > - Windows_x86_64: > > > DataOutputStream, writeObjectReadObject0004, IfElse1, btest5717, > > > btest7214 : intermittent? > > > btest6353: fails on M3 too > > > > > > * GUT: > > > ContainerTest, JAASSecurityTest - fail on all platforms > > > HARMONY-4895 says that this bug in Geronimo and not a regression. > > > > > > - Linux_x86: > > > HARMONY-5302 - says that this a bug in tests (I don't understand > > > then why they passed on M3) > > > security\jaas\ConfigurationEntryTest > > > security\jaas\LoginKerberousNonGeronimoTest > > > security\jaas\LoginKerberousTest > > > security\jaas\LoginSQLTest > > > security\network\protocol\SubjectCarryingProtocolTest > > > security\remoting\jmx\RemoteLoginTest > > > > > > - Linux_x86_64: 53 failures (11 on M3) > > > - Windows_x86_64: 22 failures (28 on M3) > > > > > > > > > * JDKTools > > > - Windows_x86_64: 3 failures by TimeoutException. > > > Regression? I previously incorrectly reported that they are > > > intermittent. They stably fail on M4 builds. > > > > > > * Reliability > > > > > > Failed on x86 platforms but they also failed on M3 > > > api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it) > > > api.net.SingleConnectTest > > > > > > - Linux_x86: > > > > > > Regresions: > > > api.kernel.object.ObjectFinalizeTest > > > api.kernel.throwable.StackTraceExcptsTest > > > api.text.DecimalFormat_Locales > > > api.zip.ZipEntryIterateThreadingTest > > > api.zip.ZlibTest > > > vm.finalization.FinalizeThrowRestoreTest > > > vm.stack.StackUnwindTest > > > > > > They also failed on M3: > > > api.nio.channels.filechannel.FileChannelMapTest > > > api.nio.channels.filechannel.MappedByteBufferForceTest > > > api.serialization.SerializableClassesTest > > > > > > - Windows_x86 > > > > > > Regresions: > > > api.kernel.string.InternThreadingTest - regression > > > api.kernel.thread.Calculation.CalcTest > > > api.kernel.thread.RecursiveThreadTest > > > api.kernel.thread.StackTraceTest > > > api.kernel.thread.Synchronization > > > api.kernel.thread.ThreadArrayTest > > > api.kernel.threadgroup.EnumerateTest > > > api.net.DatagramTest > > > api.serialization.SerializableClassesTest > > > vm.stack.StackUnwindTest > > > > > > - Linux_x86_64: > > > > > > Regressions: > > > vm.classloading.ClassCastTest > > > > > > * Stress > > > > > > - Linux_x86: > > > StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 / > > > test issue?) > > > Fragmentation, FragmentationFinalizer, FragmentationReference: > > > HARMONY-5159 (wasn't run during M3 so not regression) > > > > > > - Windows_x86: > > > StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 / > > > test issue?) > > > FragmentationReference: HARMONY-5159 (wasn't run during M3 so not > > > regression) > > > > > > - Linux_x86_64: > > > jni.arguments,jni.arrays - fails on M3 too so not regression / > > > HARMONY-???? > > > MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression) > > > > > > - Windows_x86_64: > > > NotSynchThreads.ManyClasses - crash (HARMONY-5003), regression > > > Fragmentation, FragmentationFinalizer, FragmentationReference: > > > HARMONY-5159 (wasn't run during M3 so not regression) > > > gc.mem.MemoryTest3 - regression / HARMONY-???? > > > jni.stack - fails on M3 too so not regression / HARMONY-???? > > > EVENT014- crash (HARMONY-????)/ regression > > > MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression) > > > StressThreads12Test - crash (HARMONY-????)/ regression > > > > > > * VTS VM > > > - Windows_x86: > > > dstore_20701 - intermittent? > > > > > > [1] > > > http://people.apache.org/~mloenko/snapshot_testing/script/r603534/index.html > > > [2] > > > http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html > > > [3] > > > http://people.apache.org/~mloenko/snapshot_testing/script/r603433/index.html > > > > > > Thanks, > > > Stepan Mishura > > > Intel Enterprise Solutions Software Division > > > > > > > > > -- > > Thanks, > > Andrey > > > > > -- > Stepan Mishura > Intel Enterprise Solutions Software Division > -- Thanks, Andrey
