Hi Andrey, Thanks for your evaluation. See my comments below.
On 12/17/07, Andrey Yakushev <[EMAIL PROTECTED]> wrote: > My comments on the reliability suite results: > > > * Reliability > > There are several tests with known intermittent failures in this suite > which were not put to exclude list in order to control new regression > appearance. Thus several failures are expected. > > > > > Failed on x86 platforms but they also failed on M3 > > api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it) > > That's because BTI workspace wasn't updated on CC host. > OK. I'll fix it. > > api.net.SingleConnectTest > > Known intermittent test issue, the reason was described in HARMONY-5266. > HARMONY-5266 is about HttpConnectionTest. > > > > - Linux_x86: > > > > Regresions: > > api.kernel.object.ObjectFinalizeTest > > exception after 1705 successful iterations - probably regression - I > couldn't reproduce it yet in my local runs. > > > api.kernel.throwable.StackTraceExcptsTest > > hang - could be similar to HARMONY-5288 issue > > > api.text.DecimalFormat_Locales > > hang - could be similar to HARMONY-5288 issue > > > api.zip.ZipEntryIterateThreadingTest > > Enable to open file - I saw that on M3 - not a regression. > Why not to exclude tests that fails on M3 too? > > api.zip.ZlibTest > > Known intermittent failure HARMONY-4215, HARMONY-5166 - not a regression > > > vm.finalization.FinalizeThrowRestoreTest > > VM crash after 142 successful iterations - new issue, could be similar > to HARMONY-5304 - regression > > > vm.stack.StackUnwindTest > > Hang after 1025 successful iterations - I saw it previously on M3, > probably HARMONY-4215 - not a regression > > > > > They also failed on M3: > > api.nio.channels.filechannel.FileChannelMapTest > > api.nio.channels.filechannel.MappedByteBufferForceTest > > api.serialization.SerializableClassesTest > > > > My local run but on the debug release gives only 3 failures: > I think the snapshot build should be tried first to reproduce the failures. > api.kernel.thread.VolatileVariableTest.PetersonTest - hang 32223 > successful iterations - I saw that intermittently on M3 - not > regression - probably this is HARMONY-4215 > api.nio.channels.filechannel.MappedByteBufferForceTest - not regression -was > on > vm.finalization.FinalizeThrowRestoreTest - assertion after 66 > successful iterations - could be similar to HARMONY-5304 > > There are also several known failures in valid excluded tests (they > were not run on CC) > > api.kernel.thread.LifeSimulationTest.Life > vm.classloading.ClassCastTest > vm.classloading.ClassReflectionTest > > > - Windows_x86 > > > > Regresions: > > api.kernel.string.InternThreadingTest - regression > > api.kernel.thread.Calculation.CalcTest > > api.kernel.thread.RecursiveThreadTest > > api.kernel.thread.StackTraceTest > > api.kernel.thread.Synchronization > > api.kernel.thread.ThreadArrayTest > > api.kernel.threadgroup.EnumerateTest > > api.net.DatagramTest > > api.serialization.SerializableClassesTest > > vm.stack.StackUnwindTest > > > > My local run results on the release build showed mostly different failures: > Did you mean snapshot build? > api.net.NetClient - not regression - known intermittent failure - HARMONY-4215 > api.net.SingleConnectTest - test issue - HARMONY-5266 > api.net.DatagramTest - hang on iteration #4 - probably HARMONY-4215, > but could be a regression > api.io.IOTest - i/o failure - require investigation - probably regression > api.kernel.thread.ThreadArrayTest.ThreadArrayTest - OutOfMemoryError > after 958 successful iterations > api.kernel.thread.Calculation.CalcTest - OutOfMemoryError after 17317 > successful iterations > api.kernel.object.ObjectFinalizeTest - exception after 461 successful > iterations > > There are also several failures in excluded tests (they were not run on CC) > > vm.classloading.ClassReflectionTest - not regression - HARMONY-3680 > vm.classloading.ClassAttributesTest - not regression - HARMONY-3680 > api.kernel.thread.LifeSimulationTest.Life - not regression - > OutOfMemoryError after 21507 successful iterations - HARMONY-5266 > api.kernel.hooks.AddRmPropertiesHooksTest - regression - HARMONY-5172 > > > - Linux_x86_64: > > > > Regressions: > > vm.classloading.ClassCastTest > > This is HARMONY-5304.I suggest fixing it now if possible. Is it blocker for releasing M4? > > My local run results on the debug build also showed 7 failures only in > excluded tests (they were not run on CC): > > api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest > api.kernel.thread.LifeSimulationTest.Life > api.nio.channels.filechannel.MappedByteBufferForceTest > api.net.DatagramTest > vm.classloading.ClassAttributesTest > vm.classloading.ClassReflectionTest > vm.classloading.DelegationModelTest > > Details are available at > http://wiki.apache.org/harmony/Reliability_Tests_on_DRLVM > Wow, the wiki page really informative! (BTW, s/bag/bug/ should be applied to it) > My resume: > > My tests run results: > > Platform | failures | regressions | pass rate > Windows x86 | 11 failures | 4 regressions | 87.2% > Linux x86 | 6 failures | 1 regressions | 93% > Linux x86_64 | 7 failures | 1 regression | 92% > > This is better results then M3 had. Usually on M3 we had had 11-13 > failures: > http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html > > So I think the reliability in M4 was improved in general although we > still have an opportunity for improvements :) > Thanks for the summary! -Stepan. > Thanks, > Andrey > > > On 12/14/07, Stepan Mishura <[EMAIL PROTECTED]> wrote: > > Hi, > > > > Please see testing status[1] for r603534 below. > > Note to my comments: > > - I marked tests as regression if they passed on M3 [2] > > - I question tests as intermittent if they passed on the previous > > snapshot (r603433) [3] > > - There are some tests that failed on M3 too. (move them to the exclude > > lists?) > > - Not all failures have JIRA issue assigned so please help me to find > > related JIRA number (if there is any). > > > > All comments and clarifications are greatly appreciated! > > > > ===================================================== > > We have the next suites failed on the snapshot. > > > > * Eclipse Unit Tests 3.3 > > - Linux_x86: 4 failures > > - Windows_x86: pass rate 96.92% > > - Linux_x86_64: pass rate 95.29% > > > > Would be nice to get failures evaluation. > > We definitely improved stability of the Eclipse Unit Test suite since > > M3 (Thanks to Vladimir Beliaev). > > But the are still some issues such as intermittent suite crashes (that > > make me very upset). > > > > * EGAx48: > > - Linux_x86 & Linux_x86_64 : failed. (HARMONY-5250 / regression?) > > - Windows_x86: > > Still no status – the scenario causes CC host down after running > > several hours. > > Currently I can not figure out if this is the host's issue or > > problem introduced by Harmony. > > > > > > * EHWA(GUI) > > - Linux_x86 & Linux_x86_64 : failed. > > The scenario is quite unstable. Most probably I'll drop the > > scenario in M5 from snapshot test and use only EHWA-API. > > > > > > * Functional: > > F_KeyFactoryTest_01: regression on all platforms (HARMONY-4857) > > > > - Linux_x86: > > JComponent: intermittent? > > - Linux_x86_64: > > btest6353: fails on M3 too > > - Windows_x86_64: > > DataOutputStream, writeObjectReadObject0004, IfElse1, btest5717, > > btest7214 : intermittent? > > btest6353: fails on M3 too > > > > * GUT: > > ContainerTest, JAASSecurityTest - fail on all platforms > > HARMONY-4895 says that this bug in Geronimo and not a regression. > > > > - Linux_x86: > > HARMONY-5302 - says that this a bug in tests (I don't understand > > then why they passed on M3) > > security\jaas\ConfigurationEntryTest > > security\jaas\LoginKerberousNonGeronimoTest > > security\jaas\LoginKerberousTest > > security\jaas\LoginSQLTest > > security\network\protocol\SubjectCarryingProtocolTest > > security\remoting\jmx\RemoteLoginTest > > > > - Linux_x86_64: 53 failures (11 on M3) > > - Windows_x86_64: 22 failures (28 on M3) > > > > > > * JDKTools > > - Windows_x86_64: 3 failures by TimeoutException. > > Regression? I previously incorrectly reported that they are > > intermittent. They stably fail on M4 builds. > > > > * Reliability > > > > Failed on x86 platforms but they also failed on M3 > > api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it) > > api.net.SingleConnectTest > > > > - Linux_x86: > > > > Regresions: > > api.kernel.object.ObjectFinalizeTest > > api.kernel.throwable.StackTraceExcptsTest > > api.text.DecimalFormat_Locales > > api.zip.ZipEntryIterateThreadingTest > > api.zip.ZlibTest > > vm.finalization.FinalizeThrowRestoreTest > > vm.stack.StackUnwindTest > > > > They also failed on M3: > > api.nio.channels.filechannel.FileChannelMapTest > > api.nio.channels.filechannel.MappedByteBufferForceTest > > api.serialization.SerializableClassesTest > > > > - Windows_x86 > > > > Regresions: > > api.kernel.string.InternThreadingTest - regression > > api.kernel.thread.Calculation.CalcTest > > api.kernel.thread.RecursiveThreadTest > > api.kernel.thread.StackTraceTest > > api.kernel.thread.Synchronization > > api.kernel.thread.ThreadArrayTest > > api.kernel.threadgroup.EnumerateTest > > api.net.DatagramTest > > api.serialization.SerializableClassesTest > > vm.stack.StackUnwindTest > > > > - Linux_x86_64: > > > > Regressions: > > vm.classloading.ClassCastTest > > > > * Stress > > > > - Linux_x86: > > StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 / > > test issue?) > > Fragmentation, FragmentationFinalizer, FragmentationReference: > > HARMONY-5159 (wasn't run during M3 so not regression) > > > > - Windows_x86: > > StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 / > > test issue?) > > FragmentationReference: HARMONY-5159 (wasn't run during M3 so not > > regression) > > > > - Linux_x86_64: > > jni.arguments,jni.arrays - fails on M3 too so not regression / > > HARMONY-???? > > MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression) > > > > - Windows_x86_64: > > NotSynchThreads.ManyClasses - crash (HARMONY-5003), regression > > Fragmentation, FragmentationFinalizer, FragmentationReference: > > HARMONY-5159 (wasn't run during M3 so not regression) > > gc.mem.MemoryTest3 - regression / HARMONY-???? > > jni.stack - fails on M3 too so not regression / HARMONY-???? > > EVENT014- crash (HARMONY-????)/ regression > > MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression) > > StressThreads12Test - crash (HARMONY-????)/ regression > > > > * VTS VM > > - Windows_x86: > > dstore_20701 - intermittent? > > > > [1] > > http://people.apache.org/~mloenko/snapshot_testing/script/r603534/index.html > > [2] > > http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html > > [3] > > http://people.apache.org/~mloenko/snapshot_testing/script/r603433/index.html > > > > Thanks, > > Stepan Mishura > > Intel Enterprise Solutions Software Division > > > > > -- > Thanks, > Andrey > -- Stepan Mishura Intel Enterprise Solutions Software Division
