[
https://issues.apache.org/jira/browse/GEODE-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139983#comment-15139983
]
Swapnil Bawaskar commented on GEODE-446:
----------------------------------------
One of the members was not able to talk to the locator for about 32 seconds.
The test failed after 30 seconds.
In the new log, I see a 15 second warning, followed by a IOException.
{noformat}
[vm_0][warn 2015/10/26 10:29:05.610 PDT <RMI TCP Connection(10)-10.118.33.207>
tid=0xf39] 15 seconds have elapsed while waiting for replies:
<UpdateAttributesProcessor$UpdateAttributesReplyProcessor 33155 waiting for 1
replies from [cc8-rh64(8770:locator)<v0>:42343]> on cc8-rh64(8782)<v257>:7795
whose current membership list is: [[cc8-rh64(8770:locator)<v0>:42343,
cc8-rh64(8800)<v250>:64079, cc8-rh64(8791)<v255>:12640,
cc8-rh64(8782)<v257>:7795, cc8-rh64(8795)<v249>:61553]]
[vm_0]
[locator][info 2015/10/26 10:29:20.766 PDT <Pooled High Priority Message
Processor 1> tid=0x41] Connection: shared=true ordered=false failed to connect
to peer cc8-rh64(8782)<v257>:7795 because: java.io.IOException: 30000 millis
timeout while waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[closed]
[locator]
{noformat}
The test fails after waiting for 30 seconds, and yet after 32 seconds, the
attempt to reconnect is successful and UpdateAttributesProcessor gets a reply:
{noformat}
[locator][info 2015/10/26 10:29:22.778 PDT <Pooled High Priority Message
Processor 1> tid=0x41] cc8-rh64(8770:locator)<v0>:42343: Successfully
reestablished connection to peer cc8-rh64(8782)<v257>:7795
[locator]
[vm_0][info 2015/10/26 10:29:22.778 PDT <P2P message reader for
cc8-rh64(8770:locator)<v0>:42343 shared unordered uid=129 port=55337>
tid=0x102e] Member cc8-rh64(8770:locator)<v0>:42343 is equivalent or in the
same redundancy zone.
[vm_0]
[vm_0][info 2015/10/26 10:29:22.779 PDT <RMI TCP Connection(10)-10.118.33.207>
tid=0xf39] UpdateAttributesProcessor$UpdateAttributesReplyProcessor wait for
replies completed
[vm_0]
{noformat}
This is rather odd as the two processes were running on the same machine.
Bruce, can you please take a look?
> CI failure:
> PartitionedRegionHAFailureAndRecoveryDUnitTest.testRecoveryOfSingleMemberFailure
> --------------------------------------------------------------------------------------------
>
> Key: GEODE-446
> URL: https://issues.apache.org/jira/browse/GEODE-446
> Project: Geode
> Issue Type: Bug
> Reporter: Kirk Lund
> Assignee: Darrel Schneider
> Labels: CI
> Attachments: GEODE-446-failure.log,
> com.gemstone.gemfire.internal.cache.PartitionedRegionHAFailureAndRecoveryDUnitTest.html
>
>
> {noformat}
> dunit.RMIException: While invoking
> com.gemstone.gemfire.internal.cache.PartitionedRegionHAFailureAndRecoveryDUnitTest$7.run
> in VM 1 running on Host angola.gemstone.com with 4 VMs
> at dunit.VM.invoke(VM.java:360)
> at dunit.VM.invoke(VM.java:303)
> at dunit.VM.invoke(VM.java:257)
> at
> com.gemstone.gemfire.internal.cache.PartitionedRegionHAFailureAndRecoveryDUnitTest.testRecoveryOfSingleMemberFailure(PartitionedRegionHAFailureAndRecoveryDUnitTest.java:458)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:252)
> at junit.framework.TestSuite.run(TestSuite.java:247)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:86)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:49)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
> at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
> at sun.reflect.GeneratedMethodAccessor176.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
> at
> org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:64)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: junit.framework.AssertionFailedError: Test waited more than 120
> seconds for redundancy recover
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.TestCase.fail(TestCase.java:227)
> at
> com.gemstone.gemfire.internal.cache.PartitionedRegionHAFailureAndRecoveryDUnitTest$7.run(PartitionedRegionHAFailureAndRecoveryDUnitTest.java:472)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at hydra.MethExecutor.executeObject(MethExecutor.java:258)
> at
> dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:68)
> at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$241(TCPTransport.java:683)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$1/868282079.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> ... 3 more
> {noformat}
> Previously run tests: [PartitionedRegionSingleHopDUnitTest,
> PartitionedRegionCacheXMLExampleDUnitTest, DeltaSizingDUnitTest,
> OffHeapEvictionStatsDUnitTest, Bug39079DUnitTest, PRBadToDataDUnitTest,
> PartitionedRegionBucketCreationDistributionDUnitTest,
> DeltaPropagationDUnitTest, DiskRegByteArrayDUnitTest, Bug41733DUnitTest,
> IteratorDUnitTest, BackupDUnitTest, ConcurrentDestroySubRegionDUnitTest,
> RemoteTransactionDUnitTest, EvictionObjectSizerDUnitTest,
> Bug42010StatsDUnitTest, PartitionedRegionInvalidateDUnitTest,
> PartitionedRegionHADUnitTest, CacheAdvisorDUnitTest,
> PartitionedRegionMultipleDUnitTest,
> PartitionedRegionDelayedRecoveryDUnitTest, ClientServerTransactionDUnitTest,
> P2PDeltaPropagationDUnitTest, PartitionedRegionLocalMaxMemoryDUnitTest,
> PartitionedRegionRedundancyZoneDUnitTest,
> PartitionedRegionHAFailureAndRecoveryDUnitTest]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
