My comments on the reliability suite results: > * Reliability
There are several tests with known intermittent failures in this suite which were not put to exclude list in order to control new regression appearance. Thus several failures are expected. > > Failed on x86 platforms but they also failed on M3 > api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it) That's because BTI workspace wasn't updated on CC host. > api.net.SingleConnectTest Known intermittent test issue, the reason was described in HARMONY-5266. > > - Linux_x86: > > Regresions: > api.kernel.object.ObjectFinalizeTest exception after 1705 successful iterations - probably regression - I couldn't reproduce it yet in my local runs. > api.kernel.throwable.StackTraceExcptsTest hang - could be similar to HARMONY-5288 issue > api.text.DecimalFormat_Locales hang - could be similar to HARMONY-5288 issue > api.zip.ZipEntryIterateThreadingTest Enable to open file - I saw that on M3 - not a regression. > api.zip.ZlibTest Known intermittent failure HARMONY-4215, HARMONY-5166 - not a regression > vm.finalization.FinalizeThrowRestoreTest VM crash after 142 successful iterations - new issue, could be similar to HARMONY-5304 - regression > vm.stack.StackUnwindTest Hang after 1025 successful iterations - I saw it previously on M3, probably HARMONY-4215 - not a regression > > They also failed on M3: > api.nio.channels.filechannel.FileChannelMapTest > api.nio.channels.filechannel.MappedByteBufferForceTest > api.serialization.SerializableClassesTest > My local run but on the debug release gives only 3 failures: api.kernel.thread.VolatileVariableTest.PetersonTest - hang 32223 successful iterations - I saw that intermittently on M3 - not regression - probably this is HARMONY-4215 api.nio.channels.filechannel.MappedByteBufferForceTest - not regression -was on vm.finalization.FinalizeThrowRestoreTest - assertion after 66 successful iterations - could be similar to HARMONY-5304 There are also several known failures in valid excluded tests (they were not run on CC) api.kernel.thread.LifeSimulationTest.Life vm.classloading.ClassCastTest vm.classloading.ClassReflectionTest > - Windows_x86 > > Regresions: > api.kernel.string.InternThreadingTest - regression > api.kernel.thread.Calculation.CalcTest > api.kernel.thread.RecursiveThreadTest > api.kernel.thread.StackTraceTest > api.kernel.thread.Synchronization > api.kernel.thread.ThreadArrayTest > api.kernel.threadgroup.EnumerateTest > api.net.DatagramTest > api.serialization.SerializableClassesTest > vm.stack.StackUnwindTest > My local run results on the release build showed mostly different failures: api.net.NetClient - not regression - known intermittent failure - HARMONY-4215 api.net.SingleConnectTest - test issue - HARMONY-5266 api.net.DatagramTest - hang on iteration #4 - probably HARMONY-4215, but could be a regression api.io.IOTest - i/o failure - require investigation - probably regression api.kernel.thread.ThreadArrayTest.ThreadArrayTest - OutOfMemoryError after 958 successful iterations api.kernel.thread.Calculation.CalcTest - OutOfMemoryError after 17317 successful iterations api.kernel.object.ObjectFinalizeTest - exception after 461 successful iterations There are also several failures in excluded tests (they were not run on CC) vm.classloading.ClassReflectionTest - not regression - HARMONY-3680 vm.classloading.ClassAttributesTest - not regression - HARMONY-3680 api.kernel.thread.LifeSimulationTest.Life - not regression - OutOfMemoryError after 21507 successful iterations - HARMONY-5266 api.kernel.hooks.AddRmPropertiesHooksTest - regression - HARMONY-5172 > - Linux_x86_64: > > Regressions: > vm.classloading.ClassCastTest This is HARMONY-5304.I suggest fixing it now if possible. My local run results on the debug build also showed 7 failures only in excluded tests (they were not run on CC): api.kernel.thread.ExcptHandlerTest.ExcptHandlerTest api.kernel.thread.LifeSimulationTest.Life api.nio.channels.filechannel.MappedByteBufferForceTest api.net.DatagramTest vm.classloading.ClassAttributesTest vm.classloading.ClassReflectionTest vm.classloading.DelegationModelTest Details are available at http://wiki.apache.org/harmony/Reliability_Tests_on_DRLVM My resume: My tests run results: Platform | failures | regressions | pass rate Windows x86 | 11 failures | 4 regressions | 87.2% Linux x86 | 6 failures | 1 regressions | 93% Linux x86_64 | 7 failures | 1 regression | 92% This is better results then M3 had. Usually on M3 we had had 11-13 failures: http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html So I think the reliability in M4 was improved in general although we still have an opportunity for improvements :) Thanks, Andrey On 12/14/07, Stepan Mishura <[EMAIL PROTECTED]> wrote: > Hi, > > Please see testing status[1] for r603534 below. > Note to my comments: > - I marked tests as regression if they passed on M3 [2] > - I question tests as intermittent if they passed on the previous > snapshot (r603433) [3] > - There are some tests that failed on M3 too. (move them to the exclude > lists?) > - Not all failures have JIRA issue assigned so please help me to find > related JIRA number (if there is any). > > All comments and clarifications are greatly appreciated! > > ===================================================== > We have the next suites failed on the snapshot. > > * Eclipse Unit Tests 3.3 > - Linux_x86: 4 failures > - Windows_x86: pass rate 96.92% > - Linux_x86_64: pass rate 95.29% > > Would be nice to get failures evaluation. > We definitely improved stability of the Eclipse Unit Test suite since > M3 (Thanks to Vladimir Beliaev). > But the are still some issues such as intermittent suite crashes (that > make me very upset). > > * EGAx48: > - Linux_x86 & Linux_x86_64 : failed. (HARMONY-5250 / regression?) > - Windows_x86: > Still no status – the scenario causes CC host down after running > several hours. > Currently I can not figure out if this is the host's issue or > problem introduced by Harmony. > > > * EHWA(GUI) > - Linux_x86 & Linux_x86_64 : failed. > The scenario is quite unstable. Most probably I'll drop the > scenario in M5 from snapshot test and use only EHWA-API. > > > * Functional: > F_KeyFactoryTest_01: regression on all platforms (HARMONY-4857) > > - Linux_x86: > JComponent: intermittent? > - Linux_x86_64: > btest6353: fails on M3 too > - Windows_x86_64: > DataOutputStream, writeObjectReadObject0004, IfElse1, btest5717, > btest7214 : intermittent? > btest6353: fails on M3 too > > * GUT: > ContainerTest, JAASSecurityTest - fail on all platforms > HARMONY-4895 says that this bug in Geronimo and not a regression. > > - Linux_x86: > HARMONY-5302 - says that this a bug in tests (I don't understand > then why they passed on M3) > security\jaas\ConfigurationEntryTest > security\jaas\LoginKerberousNonGeronimoTest > security\jaas\LoginKerberousTest > security\jaas\LoginSQLTest > security\network\protocol\SubjectCarryingProtocolTest > security\remoting\jmx\RemoteLoginTest > > - Linux_x86_64: 53 failures (11 on M3) > - Windows_x86_64: 22 failures (28 on M3) > > > * JDKTools > - Windows_x86_64: 3 failures by TimeoutException. > Regression? I previously incorrectly reported that they are > intermittent. They stably fail on M4 builds. > > * Reliability > > Failed on x86 platforms but they also failed on M3 > api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it) > api.net.SingleConnectTest > > - Linux_x86: > > Regresions: > api.kernel.object.ObjectFinalizeTest > api.kernel.throwable.StackTraceExcptsTest > api.text.DecimalFormat_Locales > api.zip.ZipEntryIterateThreadingTest > api.zip.ZlibTest > vm.finalization.FinalizeThrowRestoreTest > vm.stack.StackUnwindTest > > They also failed on M3: > api.nio.channels.filechannel.FileChannelMapTest > api.nio.channels.filechannel.MappedByteBufferForceTest > api.serialization.SerializableClassesTest > > - Windows_x86 > > Regresions: > api.kernel.string.InternThreadingTest - regression > api.kernel.thread.Calculation.CalcTest > api.kernel.thread.RecursiveThreadTest > api.kernel.thread.StackTraceTest > api.kernel.thread.Synchronization > api.kernel.thread.ThreadArrayTest > api.kernel.threadgroup.EnumerateTest > api.net.DatagramTest > api.serialization.SerializableClassesTest > vm.stack.StackUnwindTest > > - Linux_x86_64: > > Regressions: > vm.classloading.ClassCastTest > > * Stress > > - Linux_x86: > StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 / > test issue?) > Fragmentation, FragmentationFinalizer, FragmentationReference: > HARMONY-5159 (wasn't run during M3 so not regression) > > - Windows_x86: > StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 / > test issue?) > FragmentationReference: HARMONY-5159 (wasn't run during M3 so not > regression) > > - Linux_x86_64: > jni.arguments,jni.arrays - fails on M3 too so not regression / HARMONY-???? > MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression) > > - Windows_x86_64: > NotSynchThreads.ManyClasses - crash (HARMONY-5003), regression > Fragmentation, FragmentationFinalizer, FragmentationReference: > HARMONY-5159 (wasn't run during M3 so not regression) > gc.mem.MemoryTest3 - regression / HARMONY-???? > jni.stack - fails on M3 too so not regression / HARMONY-???? > EVENT014- crash (HARMONY-????)/ regression > MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression) > StressThreads12Test - crash (HARMONY-????)/ regression > > * VTS VM > - Windows_x86: > dstore_20701 - intermittent? > > [1] > http://people.apache.org/~mloenko/snapshot_testing/script/r603534/index.html > [2] > http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html > [3] > http://people.apache.org/~mloenko/snapshot_testing/script/r603433/index.html > > Thanks, > Stepan Mishura > Intel Enterprise Solutions Software Division > -- Thanks, Andrey
