kirklund commented on pull request #5728:
URL: https://github.com/apache/geode/pull/5728#issuecomment-754230887
JmxServerReconnectDistributedTest hit a java-level deadlock which also shows
that DUnit RMI calls don't have a timeout. It hangs forever until the
DistributedTest job times out.
```
Found one Java-level deadlock:
=============================
"Management Task1":
waiting to lock monitor 0x00007fa59ba23f28 (object 0x00000007a10a3b98, a
org.apache.geode.management.internal.FederatingManager),
which is held by "RMI TCP Connection(8)-169.254.70.169"
"RMI TCP Connection(8)-169.254.70.169":
waiting to lock monitor 0x00007fa52eefecb8 (object 0x00000007a1077650, a
java.lang.Object),
which is held by "Management Task1"
Java stack information for the threads listed above:
===================================================
"Management Task1":
at
org.apache.geode.management.internal.FederatingManager.isRunning(FederatingManager.java:186)
- waiting to lock <0x00000007a10a3b98> (a
org.apache.geode.management.internal.FederatingManager)
at
org.apache.geode.management.internal.SystemManagementService.isManager(SystemManagementService.java:348)
at
org.apache.geode.management.internal.LocalManager.doManagementTask(LocalManager.java:310)
- locked <0x00000007a1077650> (a java.lang.Object)
at
org.apache.geode.management.internal.LocalManager.access$100(LocalManager.java:60)
at
org.apache.geode.management.internal.LocalManager$ManagementTask.run(LocalManager.java:373)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RMI TCP Connection(8)-169.254.70.169":
at
org.apache.geode.management.internal.LocalManager.unMarkForFederation(LocalManager.java:194)
- waiting to lock <0x00000007a1077650> (a java.lang.Object)
at
org.apache.geode.management.internal.SystemManagementService.unregisterMBean(SystemManagementService.java:343)
at
org.apache.geode.management.internal.beans.DistributedSystemBridge.removeRegion(DistributedSystemBridge.java:1508)
- locked <0x00000007a1045360> (a
java.util.concurrent.ConcurrentHashMap)
at
org.apache.geode.management.internal.beans.MBeanAggregator$RegionHandler.handleProxyRemoval(MBeanAggregator.java:324)
at
org.apache.geode.management.internal.beans.MBeanAggregator.afterRemoveProxy(MBeanAggregator.java:168)
at
org.apache.geode.management.internal.SystemManagementService.afterRemoveProxy(SystemManagementService.java:615)
at
org.apache.geode.management.internal.MBeanProxyFactory.removeProxy(MBeanProxyFactory.java:169)
at
org.apache.geode.management.internal.MBeanProxyFactory.removeAllProxies(MBeanProxyFactory.java:144)
at
org.apache.geode.management.internal.FederatingManager.removeMemberArtifacts(FederatingManager.java:516)
at
org.apache.geode.management.internal.FederatingManager.stopManagingActivity(FederatingManager.java:349)
at
org.apache.geode.management.internal.FederatingManager.stopManager(FederatingManager.java:181)
- locked <0x00000007a10a3b98> (a
org.apache.geode.management.internal.FederatingManager)
at
org.apache.geode.management.internal.SystemManagementService.close(SystemManagementService.java:206)
- locked <0x00000007a0ad18b0> (a java.util.HashMap)
at
org.apache.geode.management.internal.beans.ManagementAdapter.handleCacheRemoval(ManagementAdapter.java:717)
at
org.apache.geode.management.internal.beans.ManagementListener.handleEvent(ManagementListener.java:131)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.notifyResourceEventListeners(InternalDistributedSystem.java:2086)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.handleResourceEvent(InternalDistributedSystem.java:643)
at
org.apache.geode.internal.cache.GemFireCacheImpl.doClose(GemFireCacheImpl.java:2192)
- locked <0x00000007a07d27f0> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2151)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1559)
- locked <0x00000007a07d27f0> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1257)
at
org.apache.geode.test.dunit.Disconnect.disconnectFromDS(Disconnect.java:44)
at
org.apache.geode.test.dunit.rules.DistributedRule$TearDown.tearDownInVM(DistributedRule.java:237)
at
org.apache.geode.test.dunit.rules.DistributedRule$TearDown.lambda$doTearDown$bb17a952$1(DistributedRule.java:224)
at
org.apache.geode.test.dunit.rules.DistributedRule$TearDown$$Lambda$361/1785950851.run(Unknown
Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.geode.test.dunit.internal.MethodInvoker.executeObject(MethodInvoker.java:123)
at
org.apache.geode.test.dunit.internal.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:78)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$24/803288407.run(Unknown
Source)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
