[
https://issues.apache.org/jira/browse/GEODE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16837720#comment-16837720
]
Shelley Lynn Hughes-Godfrey edited comment on GEODE-4650 at 5/13/19 5:11 PM:
-----------------------------------------------------------------------------
Note that this hang reproduced in CI:
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UpgradeTestOpenJDK11/builds/707
Since this was fixed in 1.8, perhaps we should expect to see it in rolling
upgrade tests from older versions ... so just noting that it reproduced
(without reopening).
Hung Test:
2019-05-10 22:41:28.511 +0000
org.apache.geode.cache.wan.WANRollingUpgradeSecondaryEventsNotReprocessedAfterCurrentSiteMemberFailoverWithOldClient
testSecondaryEventsNotReprocessedAfterCurrentSiteMemberFailoverWithOldClient[from_v100]
Stack dump (from callstacks):
{noformat}
"RMI TCP Connection(3)-172.17.0.4" #35 daemon prio=5 os_prio=0 cpu=5492.78ms
elapsed=2867.65s tid=0x00007f23f8001800 nid=0x212 waiting on condition
[0x00007f244dab5000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000e0804d68> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1079)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1369)
at
java.util.concurrent.CountDownLatch.await([email protected]/CountDownLatch.java:278)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:736)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:812)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:789)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:879)
at
org.apache.geode.distributed.internal.locks.ElderInitProcessor.init(ElderInitProcessor.java:76)
at
org.apache.geode.distributed.internal.locks.ElderState.<init>(ElderState.java:57)
at
org.apache.geode.distributed.internal.DistributionManager.getElderStateWithTryLock(DistributionManager.java:3628)
at
org.apache.geode.distributed.internal.DistributionManager.getElderState(DistributionManager.java:3574)
at
org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.startElderCall(GrantorRequestProcessor.java:254)
at
org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.basicOp(GrantorRequestProcessor.java:377)
at
org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.basicOp(GrantorRequestProcessor.java:352)
at
org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.clearGrantor(GrantorRequestProcessor.java:340)
at
org.apache.geode.distributed.internal.locks.DLockService.clearGrantor(DLockService.java:885)
at
org.apache.geode.distributed.internal.locks.DLockGrantor.destroy(DLockGrantor.java:1274)
- locked <0x00000000e0b17d48> (a
org.apache.geode.distributed.internal.locks.DLockGrantor)
at
org.apache.geode.distributed.internal.locks.DLockService.nullLockGrantorId(DLockService.java:663)
at
org.apache.geode.distributed.internal.locks.DLockService.basicDestroy(DLockService.java:2606)
at
org.apache.geode.distributed.internal.locks.DLockService.destroyAndRemove(DLockService.java:2521)
- locked <0x00000000e0b17e78> (a java.lang.Object)
at
org.apache.geode.distributed.internal.locks.DLockService.destroyServiceNamed(DLockService.java:2420)
at
org.apache.geode.distributed.DistributedLockService.destroy(DistributedLockService.java:98)
at
org.apache.geode.internal.cache.GemFireCacheImpl.destroyGatewaySenderLockService(GemFireCacheImpl.java:1943)
at
org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2088)
- locked <0x00000000e0922ad8> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1862)
at
org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1858)
at
org.apache.geode.test.dunit.cache.internal.JUnit4CacheTestCase.closeCache(JUnit4CacheTestCase.java:327)
{noformat}
Test report artifacts from this job are available at:
{noformat}
http://files.apachegeode-ci.info/builds/apache-develop-main/1.10.0-SNAPSHOT.0269/test-artifacts/1557531731/upgradetestfiles-OpenJDK11-1.10.0-SNAPSHOT.0269.tgz
{noformat}
was (Author: lhughesgodfrey):
Re-opening as this hang reproduced in CI:
https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/UpgradeTestOpenJDK11/builds/707
Since this was fixed in 1.8, perhaps we should expect to see it in rolling
upgrade tests from older versions (but we need a marker for that for CI).
Hung Test:
2019-05-10 22:41:28.511 +0000
org.apache.geode.cache.wan.WANRollingUpgradeSecondaryEventsNotReprocessedAfterCurrentSiteMemberFailoverWithOldClient
testSecondaryEventsNotReprocessedAfterCurrentSiteMemberFailoverWithOldClient[from_v100]
Stack dump (from callstacks):
{noformat}
"RMI TCP Connection(3)-172.17.0.4" #35 daemon prio=5 os_prio=0 cpu=5492.78ms
elapsed=2867.65s tid=0x00007f23f8001800 nid=0x212 waiting on condition
[0x00007f244dab5000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000e0804d68> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1079)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1369)
at
java.util.concurrent.CountDownLatch.await([email protected]/CountDownLatch.java:278)
at
org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
at
org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:736)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:812)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:789)
at
org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:879)
at
org.apache.geode.distributed.internal.locks.ElderInitProcessor.init(ElderInitProcessor.java:76)
at
org.apache.geode.distributed.internal.locks.ElderState.<init>(ElderState.java:57)
at
org.apache.geode.distributed.internal.DistributionManager.getElderStateWithTryLock(DistributionManager.java:3628)
at
org.apache.geode.distributed.internal.DistributionManager.getElderState(DistributionManager.java:3574)
at
org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.startElderCall(GrantorRequestProcessor.java:254)
at
org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.basicOp(GrantorRequestProcessor.java:377)
at
org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.basicOp(GrantorRequestProcessor.java:352)
at
org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.clearGrantor(GrantorRequestProcessor.java:340)
at
org.apache.geode.distributed.internal.locks.DLockService.clearGrantor(DLockService.java:885)
at
org.apache.geode.distributed.internal.locks.DLockGrantor.destroy(DLockGrantor.java:1274)
- locked <0x00000000e0b17d48> (a
org.apache.geode.distributed.internal.locks.DLockGrantor)
at
org.apache.geode.distributed.internal.locks.DLockService.nullLockGrantorId(DLockService.java:663)
at
org.apache.geode.distributed.internal.locks.DLockService.basicDestroy(DLockService.java:2606)
at
org.apache.geode.distributed.internal.locks.DLockService.destroyAndRemove(DLockService.java:2521)
- locked <0x00000000e0b17e78> (a java.lang.Object)
at
org.apache.geode.distributed.internal.locks.DLockService.destroyServiceNamed(DLockService.java:2420)
at
org.apache.geode.distributed.DistributedLockService.destroy(DistributedLockService.java:98)
at
org.apache.geode.internal.cache.GemFireCacheImpl.destroyGatewaySenderLockService(GemFireCacheImpl.java:1943)
at
org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2088)
- locked <0x00000000e0922ad8> (a java.lang.Class for
org.apache.geode.internal.cache.GemFireCacheImpl)
at
org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1862)
at
org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1858)
at
org.apache.geode.test.dunit.cache.internal.JUnit4CacheTestCase.closeCache(JUnit4CacheTestCase.java:327)
{noformat}
Test report artifacts from this job are available at:
{noformat}
http://files.apachegeode-ci.info/builds/apache-develop-main/1.10.0-SNAPSHOT.0269/test-artifacts/1557531731/upgradetestfiles-OpenJDK11-1.10.0-SNAPSHOT.0269.tgz
{noformat}
> DLockService.clearGrantor can potentially hang
> ----------------------------------------------
>
> Key: GEODE-4650
> URL: https://issues.apache.org/jira/browse/GEODE-4650
> Project: Geode
> Issue Type: Bug
> Components: distributed lock service
> Reporter: Jason Huynh
> Priority: Major
> Labels: pull-request-available, swat
> Fix For: 1.8.0
>
> Attachments: callstacks-2018-02-10-05-25-15.txt,
> callstacks-2018-02-10-05-25-23.txt, callstacks-2018-02-10-05-25-30.txt
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> There was a test run in the precheckin pipeline that hung with the following
> stack:
>
> {code:java}
> "RMI TCP Connection(1)-172.17.0.3" #30 daemon prio=5 os_prio=0
> tid=0x00007f4560001800 nid=0x191 waiting on condition [0x00007f45771c0000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000000e082d298> (a
> java.util.concurrent.CountDownLatch$Sync)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
> at
> org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:64)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:715)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:790)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:766)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:853)
> at
> org.apache.geode.distributed.internal.locks.ElderInitProcessor.init(ElderInitProcessor.java:72)
> at
> org.apache.geode.distributed.internal.locks.ElderState.<init>(ElderState.java:56)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.getElderStateWithTryLock(ClusterDistributionManager.java:3359)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.getElderState(ClusterDistributionManager.java:3309)
> at
> org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.startElderCall(GrantorRequestProcessor.java:238)
> at
> org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.basicOp(GrantorRequestProcessor.java:347)
> at
> org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.basicOp(GrantorRequestProcessor.java:327)
> at
> org.apache.geode.distributed.internal.locks.GrantorRequestProcessor.clearGrantor(GrantorRequestProcessor.java:318)
> at
> org.apache.geode.distributed.internal.locks.DLockService.clearGrantor(DLockService.java:872)
> at
> org.apache.geode.distributed.internal.locks.DLockGrantor.destroy(DLockGrantor.java:1227)
> - locked <0x00000000e0837ff0> (a
> org.apache.geode.distributed.internal.locks.DLockGrantor)
> at
> org.apache.geode.distributed.internal.locks.DLockService.nullLockGrantorId(DLockService.java:646)
> at
> org.apache.geode.distributed.internal.locks.DLockService.basicDestroy(DLockService.java:2358)
> at
> org.apache.geode.distributed.internal.locks.DLockService.destroyAndRemove(DLockService.java:2276)
> - locked <0x00000000e05c7468> (a java.lang.Object)
> at
> org.apache.geode.distributed.internal.locks.DLockService.destroyServiceNamed(DLockService.java:2214)
> at
> org.apache.geode.distributed.DistributedLockService.destroy(DistributedLockService.java:84)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.destroyGatewaySenderLockService(GemFireCacheImpl.java:2043)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2180)
> - locked <0x00000000e04653e0> (a java.lang.Class for
> org.apache.geode.internal.cache.GemFireCacheImpl)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1960)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:1950)
> at
> org.apache.geode.test.junit.rules.ServerStarterRule.stopMember(ServerStarterRule.java:99)
> at
> org.apache.geode.test.junit.rules.MemberStarterRule.after(MemberStarterRule.java:81)
> at
> org.apache.geode.test.dunit.rules.ClusterStartupRule.stopElementInsideVM(ClusterStartupRule.java:412)
> at
> org.apache.geode.test.junit.rules.VMProvider.lambda$stopVM$fe0d42dc$1(VMProvider.java:35)
> at
> org.apache.geode.test.junit.rules.VMProvider$$Lambda$53/208982926.run(Unknown
> Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at hydra.MethExecutor.executeObject(MethExecutor.java:244)
> at
> org.apache.geode.test.dunit.standalone.RemoteDUnitVM.executeMethodOnObject(RemoteDUnitVM.java:70)
> at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
> at sun.rmi.transport.Transport$1.run(Transport.java:200)
> at sun.rmi.transport.Transport$1.run(Transport.java:197)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$7/1394836008.run(Unknown
> Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Locked ownable synchronizers:
> - <0x00000000e0332230> (a java.util.concurrent.ThreadPoolExecutor$Worker)
> - <0x00000000e08499b0> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> - <0x00000000e08520f0> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {code}
> It looks like the cache is shutting down and we are unable to destroy the
> lock service for the gateway sender.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
