I opened an umbrella for Replication v2 as HBASE-14379. At the moment it envisions the administration of cross-DC replication relationships and data access as the same as today. However, we do have an opportunity to reboot with a completely different approach. I thought it worth bringing up for discussion.
We could in theory reboot around timeline consistent region replicas. If you squint, region replicas have a similar theory of operation as cross-DC replication. What if we redefine administration and data access for Replication v2 as sets of region replica placements that can cross data center boundaries, with the client able to distinguish local locations from remote locations, and then choose based on policy? So if, for example, you may have three data centers, then instead of setting up three point-to-point replication peering relationships like today, you'd simply create a table that has a region replica placement policy in its schema with (logical) locations spanning all three data centers. Behind the scenes, each data center would have HBASE-10070 style primary-secondary relationships, and additionally: 1. the primary region will run something like today's replication source for each secondary location that is in a remote DC; 2. a primary region anywhere may receive change streams from remote DCs like today's replication sinks. On the client side we have some prior work in this regard: CSBT, and Ted Malaska's HBase.MCC. I mention CSBT but I don't think we want its partitioning or reliance on Zookeeper. HBase.MCC is more of a starting point. I'm not saying we should do this, only that we could do this. There are pros and cons. In some ways defining point-to-point replication relationships is easier for admins and users, e.g. the topology is built and managed explicitly. In some ways merging replicas and cross-DC replication is easier, e.g. it removes APIs, necessary tooling, cognitive load (cross-DC replication is no longer 'special'). -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
