[
https://issues.apache.org/jira/browse/GEODE-502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524476#comment-15524476
]
Dan Smith commented on GEODE-502:
---------------------------------
We're still seeing this. More I've attached a failure (with stdout) from
a1938b386f1ed9064529df7f4206af913d4461c2. One interesting thing is that we do
see a 7 second GC pause from one of the members, I'm not sure if that would
affect the startup.
One possible workaround might be to relaunch all the VMs in the process manager
if they fail to start up.
> CI Failure: TestSuite$1.warning
> -------------------------------
>
> Key: GEODE-502
> URL: https://issues.apache.org/jira/browse/GEODE-502
> Project: Geode
> Issue Type: Bug
> Reporter: Dan Smith
> Assignee: Dan Smith
> Labels: CI
>
> I saw this in a internal CI run. The first test fails with the below error
> about VMs not launching in 30 seconds. Then later tests fail with a cause of
> "Log writer has already been created"
> This is actually a failure to launch the dunit framework, which is why it's
> misreporting the test name as "TestSuite$1.warning"; the failure happens
> during the constructor of the test case.
> {noformat}
> junit.framework.AssertionFailedError: Exception in constructor:
> testPutGetTimeout (java.lang.RuntimeException: Unable to launch dunit VMS
> at dunit.standalone.DUnitLauncher.launchIfNeeded(DUnitLauncher.java:130)
> at dunit.DistributedTestCase.<init>(DistributedTestCase.java:416)
> at
> com.gemstone.gemfire.cache30.CacheTestCase.<init>(CacheTestCase.java:89)
> at
> com.gemstone.gemfire.cache30.RegionTestCase.<init>(RegionTestCase.java:110)
> at
> com.gemstone.gemfire.cache30.MultiVMRegionTestCase.<init>(MultiVMRegionTestCase.java:160)
> at
> com.gemstone.gemfire.cache30.GlobalRegionDUnitTest.<init>(GlobalRegionDUnitTest.java:39)
> at
> com.gemstone.gemfire.cache30.GlobalRegionCompressionDUnitTest.<init>(GlobalRegionCompressionDUnitTest.java:34)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at junit.framework.TestSuite.createTest(TestSuite.java:65)
> at junit.framework.TestSuite.addTestMethod(TestSuite.java:307)
> at junit.framework.TestSuite.addTestsFromTestCase(TestSuite.java:150)
> at junit.framework.TestSuite.<init>(TestSuite.java:129)
> at
> org.junit.internal.runners.JUnit38ClassRunner.<init>(JUnit38ClassRunner.java:74)
> at
> org.junit.internal.builders.JUnit3Builder.runnerForClass(JUnit3Builder.java:11)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
> at
> org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
> at
> org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:88)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
> at
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: VMs did not start up with 30 seconds
> at dunit.standalone.DUnitLauncher.launch(DUnitLauncher.java:181)
> at dunit.standalone.DUnitLauncher.launchIfNeeded(DUnitLauncher.java:128)
> ... 46 more
> )
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.TestCase.fail(TestCase.java:227)
> at junit.framework.TestSuite$1.runTest(TestSuite.java:97)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:252)
> at junit.framework.TestSuite.run(TestSuite.java:247)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
> at
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I was able to track down the stdout for this first test that failed. This
> looks like something that has been reported as a JDK bug when related to GC
> of some classes held in a WeakCache in the JDK -
> https://bugs.openjdk.java.net/browse/JDK-8087168
> {noformat}
> [locator]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [locator]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [vm_0]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [vm_0]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [vm_1]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [vm_1]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [vm_2]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [vm_2]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [vm_3]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [vm_3]Java HotSpot(TM) 64-Bit Server VM warning: ignoring option
> MaxPermSize=256M; support was removed in 8.0
> [vm_3]java.rmi.ServerException: RemoteException occurred in server thread;
> nested exception is:
> [vm_3] java.rmi.UnmarshalException: error unmarshalling arguments;
> nested exception is:
> [vm_3] java.io.InvalidClassException: Not a proxy
> [vm_3] at
> sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:420)
> [vm_3] at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:268)
> [vm_3] at sun.rmi.transport.Transport$1.run(Transport.java:200)
> [vm_3] at sun.rmi.transport.Transport$1.run(Transport.java:197)
> [vm_3] at java.security.AccessController.doPrivileged(Native Method)
> [vm_3] at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$241(TCPTransport.java:683)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$8/125090748.run(Unknown
> Source)
> [vm_3] at java.security.AccessController.doPrivileged(Native Method)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> [vm_3] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [vm_3] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [vm_3] at java.lang.Thread.run(Thread.java:745)
> [vm_3] at
> sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
> [vm_3] at
> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
> [vm_3] at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:379)
> [vm_3] at sun.rmi.registry.RegistryImpl_Stub.rebind(Unknown Source)
> [vm_3] at java.rmi.Naming.rebind(Naming.java:177)
> [vm_3] at dunit.standalone.ChildVM.main(ChildVM.java:52)
> [vm_3]Caused by: java.rmi.UnmarshalException: error unmarshalling arguments;
> nested exception is:
> [vm_3] java.io.InvalidClassException: Not a proxy
> [vm_3] at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source)
> [vm_3] at
> sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:410)
> [vm_3] at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:268)
> [vm_3] at sun.rmi.transport.Transport$1.run(Transport.java:200)
> [vm_3] at sun.rmi.transport.Transport$1.run(Transport.java:197)
> [vm_3] at java.security.AccessController.doPrivileged(Native Method)
> [vm_3] at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$241(TCPTransport.java:683)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$8/125090748.run(Unknown
> Source)
> [vm_3] at java.security.AccessController.doPrivileged(Native Method)
> [vm_3] at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> [vm_3] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [vm_3] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [vm_3] at java.lang.Thread.run(Thread.java:745)
> [vm_3]Caused by: java.io.InvalidClassException: Not a proxy
> [vm_3] at
> java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1562)
> [vm_3] at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515)
> [vm_3] at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
> [vm_3] at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> [vm_3] at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
> [vm_3] ... 16 more
> {noformat}
> This caused these tests to fail as well
> QueryUsingFunctionContextDUnitTest.initializationError
> QueryDataInconsistencyDUnitTest.initializationError
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
