[
https://issues.apache.org/jira/browse/GEODE-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871516#comment-16871516
]
Barry Oglesby edited comment on GEODE-3718 at 6/24/19 4:56 PM:
---------------------------------------------------------------
Lynn and I have a test that reproduces this issue.
3 servers configured like:
server 1 - replicate region
server 2 - replicate persistent region
server 3 - replicate persistent region
Server 1 is configured like:
{noformat}
<cache ...>
<cache-server port="0"/>
<region name="UTLatest" refid="REPLICATE"/>
</cache>
{noformat}
Servers 2 and 3 are configured like:
{noformat}
<cache ...>
<cache-server port="0"/>
<region name="UTLatest" refid="REPLICATE_PERSISTENT"/>
</cache>
{noformat}
This matches the customer's proxy and data groups members. Server1 is in the
proxy group and servers 2 and 3 are in the data group.
Note: You must start the persistent servers first.
kill -9 one of the servers with a replicate persistent region.
When synchronization occurs (after maximumTimeBetweenPings - 60000ms), a
message like this will be logged in each member:
[info 2019/06/24 09:46:13.133 PDT <Timer-2> tid=0x2b] Region UTLatest is
requesting synchronization with 192.168.1.2(server3:51729)<ec><v20>:41002 for
192.168.1.2(server2:51722)<ec><v19>:41001
The member with the replicate region will also throw the ToDataException.
A couple work-arounds are:
- Instead of using a replicate region in server1, use a replicate proxy region
- Use replicate persistent regions in all members
was (Author: barry.oglesby):
Lynn and I have a test that reproduces this issue.
3 servers configured like:
server 1 - replicate region
server 2 - replicate persistent region
server 3 - replicate persistent region
Server 1 is configured like:
```
<cache ...>
<cache-server port="0"/>
<region name="UTLatest" refid="REPLICATE"/>
</cache>
```
Servers 2 and 3 are configured like:
```
<cache ...>
<cache-server port="0"/>
<region name="UTLatest" refid="REPLICATE_PERSISTENT"/>
</cache>
```
This matches the customer's proxy and data groups members. Server1 is in the
proxy group and servers 2 and 3 are in the data group.
Note: You must start the persistent servers first.
kill -9 one of the servers with a replicate persistent region.
When synchronization occurs (after maximumTimeBetweenPings - 60000ms), a
message like this will be logged in each member:
[info 2019/06/24 09:46:13.133 PDT <Timer-2> tid=0x2b] Region UTLatest is
requesting synchronization with 192.168.1.2(server3:51729)<ec><v20>:41002 for
192.168.1.2(server2:51722)<ec><v19>:41001
The member with the replicate region will also throw the ToDataException.
A couple work-arounds are:
- Instead of using a replicate region in server1, use a replicate proxy region
- Use replicate persistent regions in all members
> The InternalResourceManager fails to shutdown if a redundancy recovery task
> is scheduled but hasn't fired yet
> -------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-3718
> URL: https://issues.apache.org/jira/browse/GEODE-3718
> Project: Geode
> Issue Type: Bug
> Components: core
> Reporter: Barry Oglesby
> Assignee: Mario Ivanac
> Priority: Major
> Labels: needs-review, pull-request-available, recovery
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This happens with recovery-delay or startup-recovery-delay > 0.
> The thread gets stuck here:
> {noformat}
> "Thread-20" #133 prio=10 os_prio=31 tid=0x00007fa85b886000 nid=0x890b waiting
> on condition [0x000070001269e000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007bc408900> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
> at
> org.apache.geode.internal.cache.control.InternalResourceManager.stopExecutor(InternalResourceManager.java:375)
> at
> org.apache.geode.internal.cache.control.InternalResourceManager.close(InternalResourceManager.java:187)
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2161)
> - locked <0x00000007bc0bc520> (a java.lang.Class for
> org.apache.geode.internal.cache.GemFireCacheImpl)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1367)
> - locked <0x00000007bc0bc520> (a java.lang.Class for
> org.apache.geode.internal.cache.GemFireCacheImpl)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1017)
> at
> org.apache.geode.management.internal.beans.MemberMBeanBridge$1.run(MemberMBeanBridge.java:986)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The InternalResourceManager is waiting for the termination of its
> scheduledExecutor.
> The PRHARedundancyProvider initializes its recoveryExecutor using the
> InternalResourceManager's scheduledExecutor:
> {noformat}
> recoveryExecutor = new OneTaskOnlyExecutor(resourceManager.getExecutor(),
> new OneTaskOnlyExecutor.ConflatedTaskListener() {
> public void taskDropped() {
> InternalResourceManager.getResourceObserver().recoveryConflated(region);
> }
> });
> {noformat}
> The scheduleRedundancyRecovery method schedules a RecoveryRunnable if
> necessary.
> If that task hasn't fired yet, the InternalResourceManager doesn't close, and
> the JVM stays up.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
