Stepan Mishura wrote: > Hi, > > Please see testing status[1] for r603534 below. > Note to my comments: > - I marked tests as regression if they passed on M3 [2] > - I question tests as intermittent if they passed on the previous > snapshot (r603433) [3] > - There are some tests that failed on M3 too. (move them to the exclude > lists?) > - Not all failures have JIRA issue assigned so please help me to find > related JIRA number (if there is any). > > All comments and clarifications are greatly appreciated! > > ===================================================== > We have the next suites failed on the snapshot. > > * Eclipse Unit Tests 3.3 > - Linux_x86: 4 failures > - Windows_x86: pass rate 96.92% > - Linux_x86_64: pass rate 95.29% > > Would be nice to get failures evaluation. > We definitely improved stability of the Eclipse Unit Test suite since > M3 (Thanks to Vladimir Beliaev). > But the are still some issues such as intermittent suite crashes (that > make me very upset).
And yet these numbers are high pass rates, so it implies that the milestone is quite stable. Clearly if we were testing a release candidate we would want to improve these even further. > * EGAx48: > - Linux_x86 & Linux_x86_64 : failed. (HARMONY-5250 / regression?) > - Windows_x86: > Still no status – the scenario causes CC host down after running > several hours. > Currently I can not figure out if this is the host's issue or > problem introduced by Harmony. We should look into these to see if they are regressions. I'll help. > * EHWA(GUI) > - Linux_x86 & Linux_x86_64 : failed. > The scenario is quite unstable. Most probably I'll drop the > scenario in M5 from snapshot test and use only EHWA-API. Ok. > * Functional: > F_KeyFactoryTest_01: regression on all platforms (HARMONY-4857) > > - Linux_x86: > JComponent: intermittent? As Andrey says, these should not stop us declaring M4. > - Linux_x86_64: > btest6353: fails on M3 too > - Windows_x86_64: > DataOutputStream, writeObjectReadObject0004, IfElse1, btest5717, > btest7214 : intermittent? > btest6353: fails on M3 too Intermittent and non-regression failures on the 64-bit platforms I would suggest we need to log in JIRA and target for the next milestone. > * GUT: > ContainerTest, JAASSecurityTest - fail on all platforms > HARMONY-4895 says that this bug in Geronimo and not a regression. Ok. > - Linux_x86: > HARMONY-5302 - says that this a bug in tests (I don't understand > then why they passed on M3) > security\jaas\ConfigurationEntryTest > security\jaas\LoginKerberousNonGeronimoTest > security\jaas\LoginKerberousTest > security\jaas\LoginSQLTest > security\network\protocol\SubjectCarryingProtocolTest > security\remoting\jmx\RemoteLoginTest > > - Linux_x86_64: 53 failures (11 on M3) > - Windows_x86_64: 22 failures (28 on M3) These need further investigation IMO. > * JDKTools > - Windows_x86_64: 3 failures by TimeoutException. > Regression? I previously incorrectly reported that they are > intermittent. They stably fail on M4 builds. Assuming these pass on 32-bit platforms? Is there a JIRA describing this one already? > * Reliability > > Failed on x86 platforms but they also failed on M3 > api.net.HttpConnectionTest (I wonder why HARMONY-5266 didn't fixed it) > api.net.SingleConnectTest > > - Linux_x86: > > Regresions: > api.kernel.object.ObjectFinalizeTest > api.kernel.throwable.StackTraceExcptsTest > api.text.DecimalFormat_Locales > api.zip.ZipEntryIterateThreadingTest > api.zip.ZlibTest > vm.finalization.FinalizeThrowRestoreTest > vm.stack.StackUnwindTest Regressions :-( > They also failed on M3: > api.nio.channels.filechannel.FileChannelMapTest > api.nio.channels.filechannel.MappedByteBufferForceTest > api.serialization.SerializableClassesTest > > - Windows_x86 > > Regresions: > api.kernel.string.InternThreadingTest - regression > api.kernel.thread.Calculation.CalcTest > api.kernel.thread.RecursiveThreadTest > api.kernel.thread.StackTraceTest > api.kernel.thread.Synchronization > api.kernel.thread.ThreadArrayTest > api.kernel.threadgroup.EnumerateTest > api.net.DatagramTest > api.serialization.SerializableClassesTest > vm.stack.StackUnwindTest Regressions :-( > - Linux_x86_64: > > Regressions: > vm.classloading.ClassCastTest > > * Stress > > - Linux_x86: > StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 / > test issue?) > Fragmentation, FragmentationFinalizer, FragmentationReference: > HARMONY-5159 (wasn't run during M3 so not regression) > > - Windows_x86: > StackUnwindingManyObjectsTests: 5 fails on M3 too (HARMONY-5158 / > test issue?) > FragmentationReference: HARMONY-5159 (wasn't run during M3 so not > regression) Ok. > - Linux_x86_64: > jni.arguments,jni.arrays - fails on M3 too so not regression / > HARMONY-???? > MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression) > > - Windows_x86_64: > NotSynchThreads.ManyClasses - crash (HARMONY-5003), regression > Fragmentation, FragmentationFinalizer, FragmentationReference: > HARMONY-5159 (wasn't run during M3 so not regression) > gc.mem.MemoryTest3 - regression / HARMONY-???? > jni.stack - fails on M3 too so not regression / HARMONY-???? > EVENT014- crash (HARMONY-????)/ regression > MEMORY001, MEMORY003 - HARMONY-5047 (fails on M3 too so not regression) > StressThreads12Test - crash (HARMONY-????)/ regression A variety of things to investigate in there it seems. > * VTS VM > - Windows_x86: > dstore_20701 - intermittent? > > [1] > http://people.apache.org/~mloenko/snapshot_testing/script/r603534/index.html > [2] > http://people.apache.org/~mloenko/snapshot_testing/script/r580985/index.html > [3] > http://people.apache.org/~mloenko/snapshot_testing/script/r603433/index.html Another great summary Stepan. Thanks for doing this! Regards, Tim
