[
https://issues.apache.org/jira/browse/GEODE-8440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180054#comment-17180054
]
Anilkumar Gingade commented on GEODE-8440:
------------------------------------------
The other observations are:
Just a few of the stuck threads in the hung server:
{noformat}
[warn 2020/07/30 22:31:39.619 PDT <ThreadsMonitor> tid=0x6fa] Thread <1902>
(0x76e) that was executed at <30 Jul 2020 22:30:55 PDT> has been stuck for
<44.225 seconds> and number of thread monitor iteration <1>
Thread Name <ResourceManagerRecoveryThread 1> state <TIMED_WAITING>
Waiting on <java.util.concurrent.Semaphore$NonfairSync@209d985>
Executor Group <OneTaskOnlyExecutor>
Monitored metric <ResourceManagerStats.numThreadsStuck>
Thread stack:
[email protected]/jdk.internal.misc.Unsafe.park(Native Method)
[email protected]/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
[email protected]/java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:592)
app//org.apache.geode.internal.cache.partitioned.rebalance.ParallelBucketOperator.waitForOperations(ParallelBucketOperator.java:154)
app//org.apache.geode.internal.cache.partitioned.rebalance.model.PartitionedRegionLoadModel.waitForOperations(PartitionedRegionLoadModel.java:786)
app//org.apache.geode.internal.cache.partitioned.rebalance.SatisfyRedundancy.nextStep(SatisfyRedundancy.java:52)
app//org.apache.geode.internal.cache.partitioned.rebalance.CompositeDirector.nextStep(CompositeDirector.java:100)
app//org.apache.geode.internal.cache.partitioned.PartitionedRegionRebalanceOp.execute(PartitionedRegionRebalanceOp.java:237)
app//org.apache.geode.internal.cache.PRHARedundancyProvider$1.run2(PRHARedundancyProvider.java:1500)
app//org.apache.geode.internal.cache.partitioned.RecoveryRunnable.run(RecoveryRunnable.java:60)
app//org.apache.geode.internal.OneTaskOnlyExecutor$DelegatingRunnable.run(OneTaskOnlyExecutor.java:142)
[email protected]/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[email protected]/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[email protected]/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[email protected]/java.lang.Thread.run(Thread.java:834)
...
[warn 2020/07/30 22:32:17.908 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:32:22.942 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:32:28.136 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:32:33.391 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:32:38.497 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:32:39.618 PDT <ThreadsMonitor> tid=0x6fa] Thread 1858
(0x742) is stuck
[warn 2020/07/30 22:32:39.619 PDT <ThreadsMonitor> tid=0x6fa] Thread <1858>
(0x742) that was executed at <30 Jul 2020 22:31:55 PDT> has been stuck for
<44.493 seconds> and number of thread monitor iteration <1>
Thread Name <Function Execution Processor2> state <TIMED_WAITING>
Executor Group <FunctionExecutionPooledExecutor>
Monitored metric <ResourceManagerStats.numThreadsStuck>
Thread stack:
[email protected]/java.lang.Thread.sleep(Native Method)
app//org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.getLuceneIndex(LuceneQueryFunction.java:197)
app//org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:129)
app//org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:81)
app//org.apache.geode.internal.cache.PartitionedRegionDataStore.executeOnDataStore(PartitionedRegionDataStore.java:2996)
app//org.apache.geode.internal.cache.partitioned.PartitionedRegionFunctionStreamingMessage.operateOnPartitionedRegion(PartitionedRegionFunctionStreamingMessage.java:101)
app//org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:333)
app//org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
app//org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:441)
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
app//org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:442)
app//org.apache.geode.distributed.internal.ClusterOperationExecutors.doFunctionExecutionThread(ClusterOperationExecutors.java:377)
app//org.apache.geode.distributed.internal.ClusterOperationExecutors$$Lambda$205/0x00000001008e3040.invoke(Unknown
Source)
app//org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
app//org.apache.geode.logging.internal.executors.LoggingThreadFactory$$Lambda$195/0x00000001008e0840.run(Unknown
Source)
[email protected]/java.lang.Thread.run(Thread.java:834)
...
[warn 2020/07/30 22:43:39.642 PDT <ThreadsMonitor> tid=0x6fa] Thread <1978>
(0x7ba) that was executed at <30 Jul 2020 22:32:12 PDT> has been stuck for
<686.701 seconds> and number of thread monitor iteration <11>
Thread Name <Function Execution Processor33> state <TIMED_WAITING>
Executor Group <FunctionExecutionPooledExecutor>
Monitored metric <ResourceManagerStats.numThreadsStuck>
Thread stack:
[email protected]/java.lang.Thread.sleep(Native Method)
app//org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.getLuceneIndex(LuceneQueryFunction.java:197)
app//org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:129)
app//org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:81)
app//org.apache.geode.internal.cache.PartitionedRegionDataStore.executeOnDataStore(PartitionedRegionDataStore.java:2996)
app//org.apache.geode.internal.cache.partitioned.PartitionedRegionFunctionStreamingMessage.operateOnPartitionedRegion(PartitionedRegionFunctionStreamingMessage.java:101)
app//org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:333)
app//org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
app//org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:441)
[email protected]/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[email protected]/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
app//org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:442)
app//org.apache.geode.distributed.internal.ClusterOperationExecutors.doFunctionExecutionThread(ClusterOperationExecutors.java:377)
app//org.apache.geode.distributed.internal.ClusterOperationExecutors$$Lambda$205/0x00000001008e3040.invoke(Unknown
Source)
app//org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
app//org.apache.geode.logging.internal.executors.LoggingThreadFactory$$Lambda$195/0x00000001008e0840.run(Unknown
Source)
[email protected]/java.lang.Thread.run(Thread.java:834)
[warn 2020/07/30 22:43:39.642 PDT <ThreadsMonitor> tid=0x6fa] There are 38
stuck threads in this node
[warn 2020/07/30 22:43:40.336 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:43:45.336 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:43:50.337 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:43:55.337 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:44:00.338 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:44:05.338 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:44:10.339 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
[warn 2020/07/30 22:44:15.339 PDT <Function Execution Processor1> tid=0x6ff] An
additional Function Execution Processor thread is being launched because all 32
thread pool threads are in use for greater than 5000 ms
{noformat}
This may be related to the fact that the current operations can't complete:
{noformat}
"Pooled High Priority Message Processor 2" #84 daemon prio=10 os_prio=0
cpu=328.79ms elapsed=1441.10s tid=0x00005590b258f000 nid=0x91c waiting on
condition [0x00007f8620dd0000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep([email protected]/Native Method)
at
org.apache.geode.distributed.internal.DistributionAdvisor$OperationMonitor.waitForCurrentOperations(DistributionAdvisor.java:1684)
at
org.apache.geode.distributed.internal.DistributionAdvisor$OperationMonitor.waitForCurrentOperations(DistributionAdvisor.java:1669)
at
org.apache.geode.distributed.internal.DistributionAdvisor$OperationMonitor.access$600(DistributionAdvisor.java:1580)
at
org.apache.geode.distributed.internal.DistributionAdvisor.waitForCurrentOperations(DistributionAdvisor.java:737)
at
org.apache.geode.internal.cache.StateFlushOperation$StateMarkerMessage.waitForCurrentOperations(StateFlushOperation.java:451)
at
org.apache.geode.internal.cache.StateFlushOperation$StateMarkerMessage.process(StateFlushOperation.java:408)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
at
org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:441)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
at
org.apache.geode.distributed.internal.ClusterOperationExecutors.runUntilShutdown(ClusterOperationExecutors.java:442)
at
org.apache.geode.distributed.internal.ClusterOperationExecutors.doHighPriorityThread(ClusterOperationExecutors.java:400)
at
org.apache.geode.distributed.internal.ClusterOperationExecutors$$Lambda$197/0x00000001008e1040.invoke(Unknown
Source)
at
org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:119)
at
org.apache.geode.logging.internal.executors.LoggingThreadFactory$$Lambda$195/0x00000001008e0840.run(Unknown
Source)
at java.lang.Thread.run([email protected]/Thread.java:834)
{noformat}
> Newly restarted node hang in creating region
> --------------------------------------------
>
> Key: GEODE-8440
> URL: https://issues.apache.org/jira/browse/GEODE-8440
> Project: Geode
> Issue Type: Bug
> Components: regions, wan
> Affects Versions: 1.14.0
> Reporter: Anilkumar Gingade
> Priority: Major
>
> In one of the internal test we observed test hanging with:
> "vm_9_thr_19_bridge5_host1_31792" #943 daemon prio=5 os_prio=0 cpu=1036.73ms
> elapsed=1026.10s tid=0x00007fc30c03f000 nid=0x5cbb waiting on condition
> [0x00007fc2d1ba2000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
> - parking to wait for <0x00000000e2394520> (a
> java.util.concurrent.CountDownLatch$Sync)
> at
> java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1079)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1369)
> at
> java.util.concurrent.CountDownLatch.await([email protected]/CountDownLatch.java:278)
> at
> org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:723)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:794)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:857)
> at
> org.apache.geode.internal.cache.persistence.MembershipFlushRequest.send(MembershipFlushRequest.java:63)
> at
> org.apache.geode.internal.cache.PRHARedundancyProvider.recoverPersistentBuckets(PRHARedundancyProvider.java:1598)
> at
> org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:1110)
> - locked <0x00000000ec1a8218> (a
> org.apache.geode.internal.cache.PartitionedRegion)
> at
> org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1220)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3081)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexImpl.createRegion(LuceneIndexImpl.java:274)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createRegion(LuceneIndexForPartitionedRegion.java:186)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createLuceneListenersAndFileChunkRegions(LuceneIndexForPartitionedRegion.java:123)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexImpl.initialize(LuceneIndexImpl.java:150)
> at
> org.apache.geode.cache.lucene.internal.LuceneServiceImpl.afterDataRegionCreated(LuceneServiceImpl.java:372)
> at
> org.apache.geode.cache.lucene.internal.LuceneRegionListener.afterCreate(LuceneRegionListener.java:116)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.invokeRegionAfter(GemFireCacheImpl.java:3146)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3123)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2975)
> at
> org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:237)
> at
> org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:675)
> at
> org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:594)
> at
> org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:339)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4180)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1601)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1427)
> at
> org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
> - locked <0x00000000e12f5db8> (a java.lang.Class for
> org.apache.geode.internal.cache.GemFireCacheImpl)
> - locked <0x00000000e12f5be8> (a java.lang.Class for
> org.apache.geode.internal.cache.InternalCacheBuilder)
> at
> org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
> at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)
> at hydra.CacheHelper.createCacheFromXml(CacheHelper.java:215)
> - locked <0x00000000e1302218> (a java.lang.Class for
> hydra.CacheHelper)
> at util.CacheUtil.createRegion(CacheUtil.java:376)
> at parReg.ParRegTest.HA_reinitializeRegion(ParRegTest.java:806)
> at parReg.ParRegTest.HAController(ParRegTest.java:2050)
> at parReg.ParRegTest.HydraTask_HAController(ParRegTest.java:1077)
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native
> Method)
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:62)
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
> at hydra.MethExecutor.execute(MethExecutor.java:173)
> at hydra.MethExecutor.execute(MethExecutor.java:141)
> at hydra.TestTask.execute(TestTask.java:197)
> at hydra.RemoteTestModule$1.run(RemoteTestModule.java:213)
> *** Stack changed
> "vm_9_thr_19_bridge5_host1_31792" #943 daemon prio=5 os_prio=0 cpu=1036.89ms
> elapsed=1036.12s tid=0x00007fc30c03f000 nid=0x5cbb waiting on condition
> [0x00007fc2d1ba2000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
> - parking to wait for <0x00000000e2394520> (a
> java.util.concurrent.CountDownLatch$Sync)
> at
> java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1079)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1369)
> at
> java.util.concurrent.CountDownLatch.await([email protected]/CountDownLatch.java:278)
> at
> org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:723)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:794)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:857)
> at
> org.apache.geode.internal.cache.persistence.MembershipFlushRequest.send(MembershipFlushRequest.java:63)
> at
> org.apache.geode.internal.cache.PRHARedundancyProvider.recoverPersistentBuckets(PRHARedundancyProvider.java:1598)
> at
> org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:1110)
> - locked <0x00000000ec1a8218> (a
> org.apache.geode.internal.cache.PartitionedRegion)
> at
> org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1220)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3081)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexImpl.createRegion(LuceneIndexImpl.java:274)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createRegion(LuceneIndexForPartitionedRegion.java:186)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createLuceneListenersAndFileChunkRegions(LuceneIndexForPartitionedRegion.java:123)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexImpl.initialize(LuceneIndexImpl.java:150)
> at
> org.apache.geode.cache.lucene.internal.LuceneServiceImpl.afterDataRegionCreated(LuceneServiceImpl.java:372)
> at
> org.apache.geode.cache.lucene.internal.LuceneRegionListener.afterCreate(LuceneRegionListener.java:116)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.invokeRegionAfter(GemFireCacheImpl.java:3146)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3123)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2975)
> at
> org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:237)
> at
> org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:675)
> at
> org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:594)
> at
> org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:339)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4180)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1601)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1427)
> at
> org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
> - locked <0x00000000e12f5db8> (a java.lang.Class for
> org.apache.geode.internal.cache.GemFireCacheImpl)
> - locked <0x00000000e12f5be8> (a java.lang.Class for
> org.apache.geode.internal.cache.InternalCacheBuilder)
> at
> org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
> at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)
> at hydra.CacheHelper.createCacheFromXml(CacheHelper.java:215)
> - locked <0x00000000e1302218> (a java.lang.Class for
> hydra.CacheHelper)
> at util.CacheUtil.createRegion(CacheUtil.java:376)
> at parReg.ParRegTest.HA_reinitializeRegion(ParRegTest.java:806)
> at parReg.ParRegTest.HAController(ParRegTest.java:2050)
> at parReg.ParRegTest.HydraTask_HAController(ParRegTest.java:1077)
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native
> Method)
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:62)
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
> at hydra.MethExecutor.execute(MethExecutor.java:173)
> at hydra.MethExecutor.execute(MethExecutor.java:141)
> at hydra.TestTask.execute(TestTask.java:197)
> at hydra.RemoteTestModule$1.run(RemoteTestModule.java:213)
> *** Stack changed
> "vm_9_thr_19_bridge5_host1_31792" #943 daemon prio=5 os_prio=0 cpu=1037.46ms
> elapsed=1075.21s tid=0x00007fc30c03f000 nid=0x5cbb waiting on condition
> [0x00007fc2d1ba2000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
> - parking to wait for <0x00000000e2394520> (a
> java.util.concurrent.CountDownLatch$Sync)
> at
> java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1079)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1369)
> at
> java.util.concurrent.CountDownLatch.await([email protected]/CountDownLatch.java:278)
> at
> org.apache.geode.internal.util.concurrent.StoppableCountDownLatch.await(StoppableCountDownLatch.java:72)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.basicWait(ReplyProcessor21.java:723)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:794)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:771)
> at
> org.apache.geode.distributed.internal.ReplyProcessor21.waitForRepliesUninterruptibly(ReplyProcessor21.java:857)
> at
> org.apache.geode.internal.cache.persistence.MembershipFlushRequest.send(MembershipFlushRequest.java:63)
> at
> org.apache.geode.internal.cache.PRHARedundancyProvider.recoverPersistentBuckets(PRHARedundancyProvider.java:1598)
> at
> org.apache.geode.internal.cache.PartitionedRegion.initPRInternals(PartitionedRegion.java:1110)
> - locked <0x00000000ec1a8218> (a
> org.apache.geode.internal.cache.PartitionedRegion)
> at
> org.apache.geode.internal.cache.PartitionedRegion.initialize(PartitionedRegion.java:1220)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3081)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexImpl.createRegion(LuceneIndexImpl.java:274)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createRegion(LuceneIndexForPartitionedRegion.java:186)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexForPartitionedRegion.createLuceneListenersAndFileChunkRegions(LuceneIndexForPartitionedRegion.java:123)
> at
> org.apache.geode.cache.lucene.internal.LuceneIndexImpl.initialize(LuceneIndexImpl.java:150)
> at
> org.apache.geode.cache.lucene.internal.LuceneServiceImpl.afterDataRegionCreated(LuceneServiceImpl.java:372)
> at
> org.apache.geode.cache.lucene.internal.LuceneRegionListener.afterCreate(LuceneRegionListener.java:116)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.invokeRegionAfter(GemFireCacheImpl.java:3146)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3123)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2975)
> at
> org.apache.geode.internal.cache.xmlcache.RegionCreation.createRoot(RegionCreation.java:237)
> at
> org.apache.geode.internal.cache.xmlcache.CacheCreation.initializeRegions(CacheCreation.java:675)
> at
> org.apache.geode.internal.cache.xmlcache.CacheCreation.create(CacheCreation.java:594)
> at
> org.apache.geode.internal.cache.xmlcache.CacheXmlParser.create(CacheXmlParser.java:339)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.loadCacheXml(GemFireCacheImpl.java:4180)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initializeDeclarativeCache(GemFireCacheImpl.java:1601)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.initialize(GemFireCacheImpl.java:1427)
> at
> org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:191)
> - locked <0x00000000e12f5db8> (a java.lang.Class for
> org.apache.geode.internal.cache.GemFireCacheImpl)
> - locked <0x00000000e12f5be8> (a java.lang.Class for
> org.apache.geode.internal.cache.InternalCacheBuilder)
> at
> org.apache.geode.internal.cache.CacheFactoryStatics.create(CacheFactoryStatics.java:61)
> at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:352)
> at hydra.CacheHelper.createCacheFromXml(CacheHelper.java:215)
> - locked <0x00000000e1302218> (a java.lang.Class for
> hydra.CacheHelper)
> at util.CacheUtil.createRegion(CacheUtil.java:376)
> at parReg.ParRegTest.HA_reinitializeRegion(ParRegTest.java:806)
> at parReg.ParRegTest.HAController(ParRegTest.java:2050)
> at parReg.ParRegTest.HydraTask_HAController(ParRegTest.java:1077)
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native
> Method)
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:62)
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke([email protected]/Method.java:566)
> at hydra.MethExecutor.execute(MethExecutor.java:173)
> at hydra.MethExecutor.execute(MethExecutor.java:141)
> at hydra.TestTask.execute(TestTask.java:197)
> at hydra.RemoteTestModule$1.run(RemoteTestModule.java:213)
> Stack for hung thread vm_9_thr_19_bridge5_host1_31792 was found 5 times and
> changed 4 times.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
