[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597418#comment-13597418
]
Jeffrey Zhong commented on HBASE-7709:
--------------------------------------
I have another idea which IMHO is better. The basic idea is following:
1) We maintain a counter value called RD("replication distance") which
represents how far a WAL edit from a source cluster to current cluster like the
hop-counter mentioned in option 3.
2) Each replaying & receiving region server maintains an internal memory
ClusterDistanceMap <clusterId, MIN(RD)>. Every time, if it sees a WAL with RD
less than it currently has seen then just update the internal map with the
smaller RD value.
3) drop all WAL edits from a cluster with RD > the the one current region
server has in the ClusterDistanceMap
Initially we could duplicate data for first several WAL edits but it will be
corrected soon so we don't need to persistent any data for fail over scenario.
The above idea is similar to option 3 but without always double replicating
data on some clusters and maintaining the max-hop is human error-prone if we
forget to bump up the max hop-count value when more clusters join in
replication cycle.
Why it works? Loop detection: quick walker will catch up slow walker but travel
more.
When we have infinite loop replication as mentioned in the JIRA, the data from
a source must come from multiple ways to the destination with different RDs.
Because it's evolving some loops, the RD won't be same otherwise there is no
loop. Since the RD is different, we just need keep the data from the source
with min distance.
You may ask the diamond situation like following.
a->b->d
a->c->d
where the data from a will be replicated to d twice. This is we configure to
let d receive a's data twice. If there is loop involved and the loop-backed
data will be dropped by the above way.
This is general loop detection strategy so we can implement it in 0.96 or
above. For 0.94,
1) we can introduce a new version(3) in HLogKey
2) use top two bytes of UUID to store the RD value and the remaining 14 bytes
as a hash value of the 16 bytes length of origin UUID value without
compromising uniqueness because in most cases we have 10s clusters involved in
replication and the collision probability is less than 10(-18)
OR
using Ted's suggestion to overload the boolean byte.
3) we can introduce a configuration setting with default to true. When we want
to revert the new behavior, we can turn it off.
please let me how do you think? Assign the ticket to me firstly in case we
agree the implement the way I'm proposing.
Thanks,
-Jeffrey
> Infinite loop possible in Master/Master replication
> ---------------------------------------------------
>
> Key: HBASE-7709
> URL: https://issues.apache.org/jira/browse/HBASE-7709
> Project: HBase
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.95.0, 0.94.6
> Reporter: Lars Hofhansl
> Fix For: 0.95.0, 0.94.7
>
>
> We just discovered the following scenario:
> # Cluster A and B are setup in master/master replication
> # By accident we had Cluster C replicate to Cluster A.
> Now all edit originating from C will be bouncing between A and B. Forever!
> The reason is that when the edit come in from C the cluster ID is already set
> and won't be reset.
> We have a couple of options here:
> # Optionally only support master/master (not cycles of more than two
> clusters). In that case we can always reset the cluster ID in the
> ReplicationSource. That means that now cycles > 2 will have the data cycle
> forever. This is the only option that requires no changes in the HLog format.
> # Instead of a single cluster id per edit maintain a (unordered) set of
> cluster id that have seen this edit. Then in ReplicationSource we drop any
> edit that the sink has seen already. The is the cleanest approach, but it
> might need a lot of data stored per edit if there are many clusters involved.
> # Maintain a configurable counter of the maximum cycle side we want to
> support. Could default to 10 (even maybe even just). Store a hop-count in the
> WAL and the ReplicationSource increases that hop-count on each hop. If we're
> over the max, just drop the edit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
