[
https://issues.apache.org/jira/browse/GEODE-911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199989#comment-15199989
]
Jason Huynh commented on GEODE-911:
-----------------------------------
We have threads that linger after shutting down the SerialGatewaySenderImpl.
The stop() method sets the eventProcessor to null but when trying to retrieve
the queues to shut down, has an if check to return null if the eventProcessor
is null. This leads to the queues not shutting down and the batch removal
threads for each queue to linger. This eventually causes the out of memory
unable to start native thread issue.
This probably won't be encountered by a user as it is not expected to
continually start/shutdown gateway senders, however our tests do this quite
often in the same jvm.
Will work on a fix and test for this issue.
> CI failure:
> ConcurrentSerialGatewaySenderOperationsOffHeapDUnitTest.test_Bug44153_StopOneSender_StartAnotherSender_CheckQueueSize
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-911
> URL: https://issues.apache.org/jira/browse/GEODE-911
> Project: Geode
> Issue Type: Bug
> Reporter: Sai Boorlagadda
> Assignee: Jason Huynh
> Labels: CI
>
> {noformat}
> Error Message
> com.gemstone.gemfire.test.dunit.RMIException: While invoking
> com.gemstone.gemfire.internal.cache.wan.serial.SerialGatewaySenderOperationsDUnitTest$$Lambda$92/1788087112.run
> in VM 2 running on Host buildtest.gemstone.com with 8 VMs
> Stacktrace
> com.gemstone.gemfire.test.dunit.RMIException: While invoking
> com.gemstone.gemfire.internal.cache.wan.serial.SerialGatewaySenderOperationsDUnitTest$$Lambda$92/1788087112.run
> in VM 2 running on Host buildtest.gemstone.com with 8 VMs
> at com.gemstone.gemfire.test.dunit.VM.invoke(VM.java:370)
> at com.gemstone.gemfire.test.dunit.VM.invoke(VM.java:313)
> at com.gemstone.gemfire.test.dunit.VM.invoke(VM.java:267)
> at
> com.gemstone.gemfire.internal.cache.wan.serial.SerialGatewaySenderOperationsDUnitTest.test_Bug44153_StopOneSender_StartAnotherSender_CheckQueueSize(SerialGatewaySenderOperationsDUnitTest.java:434)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at junit.framework.TestCase.runBare(TestCase.java:141)
> at junit.framework.TestResult$1.protect(TestResult.java:122)
> at junit.framework.TestResult.runProtected(TestResult.java:142)
> at junit.framework.TestResult.run(TestResult.java:125)
> at junit.framework.TestCase.run(TestCase.java:129)
> at junit.framework.TestSuite.runTest(TestSuite.java:252)
> at junit.framework.TestSuite.run(TestSuite.java:247)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
> at
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
> at
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: junit.framework.AssertionFailedError: Event never occurred after
> 240000 ms: Expected region entries: 100 but actual entries: 90 present region
> keyset [40, 95, 67, 49, 31, 96, 20, 75, 103, 89, 68, 55, 62, 34, 19, 83, 13,
> 41, 48, 69, 33, 28, 97, 18, 25, 88, 61, 74, 56, 84, 70, 91, 81, 98, 46, 102,
> 101, 73, 108, 53, 90, 99, 36, 26, 45, 35, 63, 72, 54, 109, 82, 100, 51, 86,
> 44, 58, 64, 92, 16, 71, 10, 78, 43, 30, 23, 15, 38, 85, 29, 80, 107, 24, 106,
> 65, 79, 93, 77, 14, 21, 105, 60, 94, 66, 11, 39, 50, 104, 59, 76, 87]
> at junit.framework.Assert.fail(Assert.java:57)
> at junit.framework.TestCase.fail(TestCase.java:227)
> at
> com.gemstone.gemfire.test.dunit.DistributedTestCase.waitForCriterion(DistributedTestCase.java:1135)
> at
> com.gemstone.gemfire.internal.cache.wan.WANTestBase.validateRegionSize(WANTestBase.java:3691)
> at
> com.gemstone.gemfire.internal.cache.wan.serial.SerialGatewaySenderOperationsDUnitTest.lambda$test_Bug44153_StopOneSender_StartAnotherSender_CheckQueueSize$81c80a4a$15(SerialGatewaySenderOperationsDUnitTest.java:434)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at hydra.MethExecutor.executeObject(MethExecutor.java:267)
> at
> com.gemstone.gemfire.test.dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:84)
> at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$95(TCPTransport.java:683)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> ... 3 more
> Standard Output
> Previously run tests: [WANManagementDUnitTest, TestRemoteClusterDUnitTest,
> ClusterConfigurationDUnitTest, UpdateVersionDUnitTest,
> ConcurrentWANPropogation_2_DUnitTest,
> ConcurrentParallelGatewaySenderOperation_1_DUnitTest,
> ConcurrentSerialGatewaySenderOperationsOffHeapDUnitTest]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
