[
https://issues.apache.org/jira/browse/GEODE-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871517#comment-16871517
]
Barry Oglesby commented on GEODE-6901:
--------------------------------------
Lynn and I have a test that reproduces this issue.
3 servers configured like:
server 1 - replicate region
server 2 - replicate persistent region
server 3 - replicate persistent region
Server 1 is configured like:
{noformat}
<cache ...>
<cache-server port="0"/>
<region name="UTLatest" refid="REPLICATE"/>
</cache>
{noformat}
Servers 2 and 3 are configured like:
{noformat}
<cache ...>
<cache-server port="0"/>
<region name="UTLatest" refid="REPLICATE_PERSISTENT"/>
</cache>
{noformat}
This matches the customer's proxy and data groups members. Server1 is in the
proxy group and servers 2 and 3 are in the data group.
Note: You must start the persistent servers first.
kill -9 one of the servers with a replicate persistent region.
When synchronization occurs (after maximumTimeBetweenPings - 60000ms), a
message like this will be logged in each member:
[info 2019/06/24 09:46:13.133 PDT <Timer-2> tid=0x2b] Region UTLatest is
requesting synchronization with 192.168.1.2(server3:51729)<ec><v20>:41002 for
192.168.1.2(server2:51722)<ec><v19>:41001
The member with the replicate region will also throw the ToDataException.
A couple work-arounds are:
- Instead of using a replicate region in server1, use a replicate proxy region
- Use replicate persistent regions in all members
> If a region is replicate and replicate persistent in different members and a
> replicate persistent member crashes, the replicate members throw a
> ToDataException attempting to synchronize the region
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-6901
> URL: https://issues.apache.org/jira/browse/GEODE-6901
> Project: Geode
> Issue Type: Bug
> Components: persistence, regions
> Reporter: Barry Oglesby
> Priority: Major
>
> If a region is replicate and replicate persistent in different members and a
> replicate persistent member crashes, the replicate members throw a
> ToDataException attempting to synchronize the region
> In this case, an exception like this is thrown in the replicate member:
> {noformat}
> [warn 2019/06/21 17:06:33.516 PDT <Timer-2> tid=0x2b] Timer task
> <org.apache.geode.internal.cache.DistributedRegion$1@6477fd5e> encountered
> exception
> org.apache.geode.ToDataException: class
> org.apache.geode.internal.cache.versions.VMRegionVersionVector
> at
> org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2331)
> at
> org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1492)
> at
> org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2067)
> at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2943)
> at
> org.apache.geode.internal.cache.InitialImageOperation$RequestImageMessage.toData(InitialImageOperation.java:2135)
> at
> org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2300)
> at
> org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1492)
> at
> org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:242)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:385)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:241)
> at
> org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:596)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1711)
> at
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1892)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2852)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:2779)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2816)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1526)
> at
> org.apache.geode.internal.cache.InitialImageOperation.synchronizeWith(InitialImageOperation.java:649)
> at
> org.apache.geode.internal.cache.DistributedRegion.synchronizeWith(DistributedRegion.java:1321)
> at
> org.apache.geode.internal.cache.DistributedRegion.synchronizeForLostMember(DistributedRegion.java:1310)
> at
> org.apache.geode.internal.cache.DistributedRegion.performSynchronizeForLostMemberTask(DistributedRegion.java:1295)
> at
> org.apache.geode.internal.cache.DistributedRegion$1.run2(DistributedRegion.java:1285)
> at
> org.apache.geode.internal.SystemTimer$SystemTimerTask.run(SystemTimer.java:445)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: java.lang.ClassCastException:
> org.apache.geode.internal.cache.persistence.DiskStoreID cannot be cast to
> org.apache.geode.distributed.internal.membership.InternalDistributedMember
> at
> org.apache.geode.internal.cache.versions.VMRegionVersionVector.writeMember(VMRegionVersionVector.java:31)
> at
> org.apache.geode.internal.cache.versions.RegionVersionVector.toData(RegionVersionVector.java:1204)
> at
> org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2300)
> ... 24 more
> {noformat}
> RegionVersionVector.java:1204 is here:
> {noformat}
> for (Map.Entry<T, RegionVersionHolder<T>> entry :
> this.memberToVersion.entrySet()) {
> -> writeMember(entry.getKey(), out);
> InternalDataSerializer.invokeToData(entry.getValue(), out);
> }
> {noformat}
> VMRegionVersionVector expects the entries of the memberToVersion to be keyed
> by InternalDistributedMembers:
> {noformat}
> protected void writeMember(InternalDistributedMember member, DataOutput out)
> throws IOException {
> {noformat}
> Logging in RegionVersionVector.toData shows the RegionVersionVector in this
> member is a VMRegionVersionVector and its memberToVersion map contains
> DiskStoreIDs. This causes the ClassCastException.
> {noformat}
> This RegionVersionVector's (class=VMRegionVersionVector) memberToVersion map
> contains the following 1 entries:
> member=402d383b29fa4c31-8597a3b72674bf5d; class=DiskStoreID
> {noformat}
> The documentation
> (https://gemfire.docs.pivotal.io/98/geode/managing/disk_storage/starting_system_with_disk_stores.html)
> makes it sound like this is a supported configuration:
> {noformat}
> For replicated regions, where you define persistence only in some of the
> region's host members, start the persistent replicate members prior to the
> non-persistent replicate members to make sure the data is recovered from disk.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
