[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222654#comment-14222654 ] ramkrishna.s.vasudevan commented on HBASE-11992: Thank you [~apurtell] for the tests and committing this to 0.98 branch. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Fix For: 0.98.9 Attachments: HBASE-11992-0.98.patch, HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch, HBASE-11992_0.98_3.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[
https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222040#comment-14222040
]
Andrew Purtell commented on HBASE-11992:
The latest patch contains two changes, this:
{code}
diff --git a/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/Repl
index 537fd60..33eaba3 100644
--- a/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/Replication
+++ b/hbase-client/src/main/java/org/apache/hadoop/hbase/replication/Replication
@@ -321,7 +321,6 @@ public class ReplicationPeerZKImpl implements ReplicationPee
public synchronized void nodeDataChanged(String path) {
if (path.equals(node)) {
super.nodeDataChanged(path);
-readTableCFsZnode();
}
}
}
{code}
and a change to the imports of ZooKeeperScanPolicyObserver in hbase-examples,
changing wildcards to class-by-class imports, etc. I'm going to drop that hunk.
Attaching updated patch.
The other change in HBASE-12136, to ReplicationPeersZKImpl, was previously
incorporated.
Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
-
Key: HBASE-11992
URL: https://issues.apache.org/jira/browse/HBASE-11992
Project: HBase
Issue Type: Task
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.9
Attachments: HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch,
HBASE-11992_0.98_3.patch, hbase-11367_0.98.patch
ReplicationSource tails the logs for each peer. HBASE-11367 introduces
ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is
run in the same RS process and instantiated per replication peer per region
server. Implementations of this interface handle the actual shipping of WAL
edits to the remote cluster.
This issue is for backporting HBASE-11367 to 0.98.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222047#comment-14222047 ] Andrew Purtell commented on HBASE-11992: Reran tests: - Cluster 1 to 2 unidirectional, load 1M rows on 1: All 1M rows OK on 1 and 2 - Clusters 1 and 2 bidirectional, load 1M rows on 1: All 1M rows OK on 1 and 2 - Clusters 1 and 2 bidirectional, load 1M rows on 1 and 2: All 2M rows OK on 1 and 2 - Cluster 1 to 2 unidirectional, load 1M rows on 1, kill RS on 1 and restart during load: All 1M rows OK on 1 and 2 - Cluster 1 to 2 unidirectional, load on 1, kill RS on 2 and restart during load: All 1M rows OK on 1 and 2 And a new one: - Cluster 1 to 2 unidirectional, load on 1, kill RS on 1 and restart, then kill RS on 2 and restart, repeat kills in sequence until load completes: All 1M rows OK on 1 and 2 Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Fix For: 0.98.9 Attachments: HBASE-11992-0.98.patch, HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch, HBASE-11992_0.98_3.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222153#comment-14222153 ] Hudson commented on HBASE-11992: FAILURE: Integrated in HBase-0.98 #695 (See [https://builds.apache.org/job/HBase-0.98/695/]) HBASE-11992 Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 (Ramkrishna S. Vasudevan) (apurtell: rev 25be14686f281949c0e9e16d0c3875a11e990518) * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeers.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerConfig.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseInterfaceAudience.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ScopeWALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationEndpoint.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationFactory.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java * hbase-protocol/src/main/protobuf/ZooKeeper.proto * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClient.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ChainWALEntryFilter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationTrackerZKImpl.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSinkManager.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/WALEntryFilter.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestPerTableCFReplication.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/SystemTableWALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerZKImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationTrackerZKImpl.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationStateZKBase.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseReplicationEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueueInfo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee:
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222169#comment-14222169 ] Hudson commented on HBASE-11992: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #662 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/662/]) HBASE-11992 Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 (Ramkrishna S. Vasudevan) (apurtell: rev 25be14686f281949c0e9e16d0c3875a11e990518) * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/WALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationTrackerZKImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationStateZKBase.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSinkManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueueInfo.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeer.java * hbase-protocol/src/main/protobuf/ZooKeeper.proto * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/BaseReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ScopeWALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ChainWALEntryFilter.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestPerTableCFReplication.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationTrackerZKImpl.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/replication/TestReplicationAdmin.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/replication/ReplicationAdmin.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeers.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationFactory.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ReplicationEndpoint.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerZKImpl.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSinkManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseInterfaceAudience.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeerConfig.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/SystemTableWALEntryFilter.java Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221665#comment-14221665 ] Andrew Purtell commented on HBASE-11992: Thanks for your patience [~ram_krish]. I tested the following scenarios with the latest patch on this issue rebased on top of 7750a1f (HBASE-12346 addendum): https://github.com/apurtell/hbase/commit/aca7161 - Cluster 1 to 2 unidirectional, load 1M rows on 1: All 1M rows OK on 1 and 2 - Clusters 1 and 2 bidirectional, load 1M rows on 1: All 1M rows OK on 1 and 2 - Clusters 1 and 2 bidirectional, load 1M rows on 1 and 2: All 2M rows OK on 1 and 2 - Cluster 1 to 2 unidirectional, load 1M rows on 1, kill RS on 1 and restart during load: All 1M rows OK on 1 and 2 - Cluster 1 to 2 unidirectional, load on 1, kill RS on 2 and restart during load: All 1M rows OK on 1 and 2 - Cluster 1 and 2 bidirectional, load 10M rows on 1 and 2: All 20M rows OK on 1 and 2 I'm going to commit this shortly to 0.98 unless objection. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Fix For: 0.98.9 Attachments: HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[
https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221692#comment-14221692
]
Virag Kothari commented on HBASE-11992:
---
Skimmed the patch and looks fine to me.
Only one problem I could notice in TableCFsTracker:
{code}
@Override
public synchronized void nodeDataChanged(String path) {
if (path.equals(node)) {
super.nodeDataChanged(path);
readTableCFsZnode();
}
}
{code}
This isn't right. Master branch doesn't have readTableCFsZnode() in this
method.
Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
-
Key: HBASE-11992
URL: https://issues.apache.org/jira/browse/HBASE-11992
Project: HBase
Issue Type: Task
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Fix For: 0.98.9
Attachments: HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch,
hbase-11367_0.98.patch
ReplicationSource tails the logs for each peer. HBASE-11367 introduces
ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is
run in the same RS process and instantiated per replication peer per region
server. Implementations of this interface handle the actual shipping of WAL
edits to the remote cluster.
This issue is for backporting HBASE-11367 to 0.98.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221708#comment-14221708 ] Andrew Purtell commented on HBASE-11992: bq. Only one problem I could notice in TableCFsTracker: Alright, holding off commit until [~ram_krish] can comment on this and/or provide an updated patch. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Fix For: 0.98.9 Attachments: HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14221857#comment-14221857 ] ramkrishna.s.vasudevan commented on HBASE-11992: This JIRA https://issues.apache.org/jira/browse/HBASE-12136 has done some change here. I will update the patch accordingly. Many thanks to @apurtell for the testing and to [~virag] for the reviews. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Fix For: 0.98.9 Attachments: HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217349#comment-14217349 ] ramkrishna.s.vasudevan commented on HBASE-11992: [~apurtell] Do you have time to check this now? HBASE-11639 is getting some feedback now and may be once we are coming to a closure on that we may need to apply the same on 0.98 over this JIRA. What do you think? Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Fix For: 0.98.9 Attachments: HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217395#comment-14217395 ] Andrew Purtell commented on HBASE-11992: Yes sir, the release is out so let's get this in. Will try it on two small clusters with chaos this week. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Fix For: 0.98.9 Attachments: HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14217443#comment-14217443 ] ramkrishna.s.vasudevan commented on HBASE-11992: Thanks a lot [~apurtell]. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Fix For: 0.98.9 Attachments: HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14194241#comment-14194241 ] ramkrishna.s.vasudevan commented on HBASE-11992: bq.fixed up to include HBASE-12386, Thanks [~apurtell]. This would ensure that we have the updated code in 0.98 also. Thanks for your time too. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, HBASE-11992_0.98_2.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14189325#comment-14189325 ] Andrew Purtell commented on HBASE-11992: So far I've confirmed the patch on RB contains the fix for HBASE-11963 and does *not* contain configuration changes in HBASE-11964 or other changes from defaults in 0.98. The TODO I mentioned above is still there but that's a trivial matter. Let me try this out on a cluster. I will also be back for another pass over the patch. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14184840#comment-14184840 ] ramkrishna.s.vasudevan commented on HBASE-11992: [~apurtell],[~apurt...@yahoo.com] Pls a take a look at the RB. Would be useful to get the feature dependent on this to 0.98. (HBASE-11639). Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172145#comment-14172145 ] ramkrishna.s.vasudevan commented on HBASE-11992: bq.Would it be possible for you to test again with two clusters where at least two RSes are replicating in each? Only upgrade the RSes, one at a time. I did this test as suggested. 1) Started two clusters with one master and 2 RSes each. 2) Created 3 tables in each cluster with REPLICATION_SCOPE and ensured that atleast one table is present in each RS. 3) Inserted data into all tables 4) Scanned the peer cluster and read back all the data from all the tables. 5) Stopped one RS in the main cluster. 6) Applied the patch and issued balance such that the newly started RS also has some table. 7) Inserted data into that table 8) Scan the peer cluster to read back all the data 9) Stop the RS in the peer cluster. 10) Apply the patch in the peer cluster for one RS. Read back all the data and do balancing such that we have tables in this RS also 11) Insert new data from the main cluster. 12) Scan the data in the peer cluster and ensure all the new data is also available 13) Apply patch on the other RS in the main cluster. 14) Balance the cluster and add new data to all the tables. 15) Scan the peer clusters and ensure all the data is available. I think the basic scenarios are working fine. What do you think [~apurtell],[~enis]? Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172542#comment-14172542 ] Andrew Purtell commented on HBASE-11992: Thanks, yes that sounds better. Please consider putting the patch up on RB, it's a large one. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172554#comment-14172554 ] Andrew Purtell commented on HBASE-11992: When doing the test above did you ensure/observe both the old version peer and new version peer were both picking up replication work at the same time in the same source cluster? Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172602#comment-14172602 ] ramkrishna.s.vasudevan commented on HBASE-11992: bq.When doing the test above did you ensure/observe both the old version peer and new version peer were both picking up replication work at the same time in the same source cluster? You mean both the peers were able to get the new edits? Yes. It was able to. Do you suspect any specific behaviour? Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172608#comment-14172608 ] ramkrishna.s.vasudevan commented on HBASE-11992: https://reviews.apache.org/r/26756 - RB link Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170835#comment-14170835 ] ramkrishna.s.vasudevan commented on HBASE-11992: Test restart scenarios before and after the patch. 1) Without patch started two servers and inserted some data. Ensure replication was on between these two servers. Read back the data from the peer cluster. Data was found. 2) Stopped the parent cluster region server and master. 3) Applied the patch on the parent cluster. 4) Read back the data from both the clusters. 5) Inserted new data in the parent cluster. 6) Read back the data from the peer cluster. New data was found. 7) Stopped the peer cluster -both master and region server. 8) Applied the patch on the peer cluster. 9) Read back the data from peer cluster. 10) Inserted new data from the parent cluster. 11) Read back the data from the peer cluster. New data was found. [~apurtell],[~enis] Anything else to be verified? Let me know. Thanks!! Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171068#comment-14171068 ] Andrew Purtell commented on HBASE-11992: bq. Tested upgrade scenarios before and after the patch. So you started two clusters, one master and one RS each? That won't test exactly the scenario of concern. Would it be possible for you to test again with two clusters where at least two RSes are replicating in each? Only upgrade the RSes, one at a time. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171162#comment-14171162 ] ramkrishna.s.vasudevan commented on HBASE-11992: bq.Would it be possible for you to test again with two clusters where at least two RSes are replicating in each? For now I don't have such a set up with me [~apurtell]. I will see what can I do. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: HBASE-11992_0.98_1.patch, hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169084#comment-14169084 ] ramkrishna.s.vasudevan commented on HBASE-11992: Created an udpated patch on this. Will run all the testcases before I address the review comments. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135733#comment-14135733 ] Andrew Purtell commented on HBASE-11992: Attached 0.98 patch from HBASE-11367 Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Attachments: hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135737#comment-14135737 ] ramkrishna.s.vasudevan commented on HBASE-11992: Thanks [~apurtell] for raising this. I will continue on this. In a couple of days or probably beginning of next week. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Attachments: hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[
https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135889#comment-14135889
]
Andrew Purtell commented on HBASE-11992:
Preliminary comments on the patch, or things that should be in it.
In hbase-client, the following classes don't have interface stability
annotations but should all be marked Private: ReplicationFactory,
ReplicationPeersZKImpl, ReplicationQueueInfo, ReplicationQueuesClient,
ReplicationQueuesClientZKImpl , ReplicationQueuesZKImpl,
ReplicationStateZKBase, ReplicationTrackerZKImpl. I don't think users should
have expected these be other than internal implementation. Changes to these
should be fine.
We don't have known users referencing these types. I checked Phoenix.
Remove this TODO? We're not going to do this in 0.98, right?
{code}
@@ -80,6 +86,8 @@ public class ReplicationAdmin implements Closeable {
.toString(HConstants.REPLICATION_SCOPE_GLOBAL);
private final HConnection connection;
+ // TODO: replication should be managed by master. All the classes except
ReplicationAdmin should
+ // be moved to hbase-server. Resolve it in HBASE-11392.
private final ReplicationQueuesClient replicationQueuesClient;
private final ReplicationPeers replicationPeers;
{code}
Synchronize methods that call reconnect() in the new class
HBaseReplicationEndpoint:
{code}
diff --git
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
new file mode 100644
index 000..4b9a28f
--- /dev/null
+++
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java
[...]
+ @Override
+ public UUID getPeerUUID() {
+UUID peerUUID = null;
+try {
+ peerUUID = ZKClusterId.getUUIDForCluster(zkw);
+} catch (KeeperException ke) {
--- + reconnect(ke);
+}
+return peerUUID;
+ }
[...]
+ public ListServerName getRegionServers() {
+try {
+ setRegionServers(fetchSlavesAddresses(this.getZkw()));
+} catch (KeeperException ke) {
+ if (LOG.isDebugEnabled()) {
+LOG.debug(Fetch salves addresses failed., ke);
+ }
--- + reconnect(ke);
+}
+return regionServers;
+ }
[...]
{code}
The additional state in the ReplicationPeer znode pbuf introduces
mixed-version-compatibility issues:
{code}
diff --git hbase-protocol/src/main/protobuf/ZooKeeper.proto
hbase-protocol/src/main/protobuf/ZooKeeper.proto
index 37816da..598385c 100644
--- hbase-protocol/src/main/protobuf/ZooKeeper.proto
+++ hbase-protocol/src/main/protobuf/ZooKeeper.proto
@@ -119,6 +119,9 @@ message ReplicationPeer {
// clusterkey is the concatenation of the slave cluster's
//
hbase.zookeeper.quorum:hbase.zookeeper.property.clientPort:zookeeper.znode.parent
required string clusterkey = 1;
+ optional string replicationEndpointImpl = 2;
+ repeated BytesBytesPair data = 3;
+ repeated NameStringPair configuration = 4;
}
{code}
Here, will this be enough for a mixed version deployment where add_peer might
be executed on an older version and won't write the new fields of
ReplicationState into the ZK node for the peer? Should we be checking a site
file setting also?
{code}
@@ -372,9 +381,32 @@ public class ReplicationSourceManager implements
ReplicationListener {
LOG.warn(Passed replication source implementation throws errors, +
defaulting to ReplicationSource, e);
src = new ReplicationSource();
+}
+ReplicationEndpoint replicationEndpoint = null;
+try {
+ String replicationEndpointImpl = peerConfig.getReplicationEndpointImpl();
+ if (replicationEndpointImpl == null) {
+// Default to HBase inter-cluster replication endpoint
+replicationEndpointImpl =
HBaseInterClusterReplicationEndpoint.class.getName();
+ }
+ @SuppressWarnings(rawtypes)
+ Class c = Class.forName(replicationEndpointImpl);
+ replicationEndpoint = (ReplicationEndpoint) c.newInstance();
+} catch (Exception e) {
+ LOG.warn(Passed replication endpoint implementation throws errors, e);
+ throw new IOException(e);
}
{code}
Likewise, what happens when you add a peer with a custom replication endpoint
impl setting but the work gets picked up by an older server? We can't ask a
user or admin to disable replication while performing a rolling upgrade of
either the source or destination clusters so I think it's necessary to document
that customized replication endpoints are not supported in mixed version
deployments, the fleet must be up to date before customized peer settings can
be put in place. If we could ask older servers that cannot honor the new
customization settings to stop replicating and let their work be transferred to
the newer servers, that would also work, but we do not have this
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135891#comment-14135891 ] Andrew Purtell commented on HBASE-11992: Ping [~lhofhansl], [~jesse_yates] Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135909#comment-14135909 ] Andrew Purtell commented on HBASE-11992: I also wonder about major service migration. In order to stage in customized replication endpoints do we require remove_peer then add_peer, instead of an online migration and update of existing replication sources? The former implies service downtime or some edits that should have been replicated will be missed between remove and (re)add, the latter does not. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11992) Backport HBASE-11367 (Pluggable replication endpoint) to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136399#comment-14136399 ] Enis Soztutar commented on HBASE-11992: --- bq. I think it's necessary to document that customized replication endpoints are not supported in mixed version deployments, the fleet must be up to date before customized peer settings can be put in place I think this is reasonable. Usually, admins should finish for any rolling upgrade to finish before using a particular feature. Backport HBASE-11367 (Pluggable replication endpoint) to 0.98 - Key: HBASE-11992 URL: https://issues.apache.org/jira/browse/HBASE-11992 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Attachments: hbase-11367_0.98.patch ReplicationSource tails the logs for each peer. HBASE-11367 introduces ReplicationEndpoint which is customizable per peer. ReplicationEndpoint is run in the same RS process and instantiated per replication peer per region server. Implementations of this interface handle the actual shipping of WAL edits to the remote cluster. This issue is for backporting HBASE-11367 to 0.98. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
