[jira] [Commented] (HBASE-16415) Replication in different namespace

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165680#comment-16165680
 ] 

Guanghao Zhang commented on HBASE-16415:


[~jan.kun...@gmail.com] Good job. Leave some comments on review board. And the 
config example should be add to add_peer.rb, too. Thanks.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415-#2.patch, HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165601#comment-16165601
 ] 

Guanghao Zhang commented on HBASE-16415:


[~jan.kun...@gmail.com] [~yuzhih...@gmail.com] Sorry for late. I will take a 
look about the patch.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415-#2.patch, HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164616#comment-16164616
 ] 

Ted Yu commented on HBASE-16415:


[~zghaobac]:
Can you take a look at the patch ?

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415-#2.patch, HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164410#comment-16164410
 ] 

Jan Kunigk commented on HBASE-16415:


Hello,

Thanks [~busbey] for kicking of the precommit job.

Since I am new to the process: Are there more steps by me required at this 
point?
Would the reviewers like me to perform additional tests?
Are there more questions, or some questions from before which I should answer 
in more detail?

Best, Jan 

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415-#2.patch, HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143991#comment-16143991
 ] 

Hadoop QA commented on HBASE-16415:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
36m 53s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 
40s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 59s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-16415 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12882560/HBASE-16415-%232.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 7dfe1bc0f199 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / a7270cc |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/8362/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/8362/console |
| Powered by | Apache Yetus 0.4.0   http://yetus.apache.org |


This message was automatically generated.



> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415-#2.patch, HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to 

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143828#comment-16143828
 ] 

Sean Busbey commented on HBASE-16415:
-

I've restarted the precommit job for this change. let's see if 
TestSplitTableRegionProcedure has gone bad.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415-#2.patch, HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133134#comment-16133134
 ] 

Hadoop QA commented on HBASE-16415:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
30m 50s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha4. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 21s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}161m  7s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:bdc94b1 |
| JIRA Issue | HBASE-16415 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12882560/HBASE-16415-%232.patch
 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 7dcdabf7e838 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 7fee03e |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC3 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/8149/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/8149/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/8149/console |
| Powered by | Apache Yetus 0.4.0   http://yetus.apache.org |


This message was automatically generated.



> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New 

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088629#comment-16088629
 ] 

Ted Yu commented on HBASE-16415:


>From populating tableRedirectionsMap, apart from init(), you use utilize the 
>following (see BaseReplicationEndpoint):
{code}
  public void peerConfigUpdated(ReplicationPeerConfig rpc){
{code}

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088616#comment-16088616
 ] 

Jan Kunigk commented on HBASE-16415:


Hello, sorry for the delay and thanks for the feedback. I have created 
https://reviews.apache.org/r/60896/

With regards to the above questions (not addressed the posting on reviewboard 
yet):

> please move the javadoc starting with...
Changed

> Please add annotation for audience for 
> RedirectingInterClusterReplicationEndpoint (Private).
done

> For redirectEntries(), tableRedirectionsMap is populated. This action can be 
> lifted outside redirectEntries() 
Can you point me to a good place where it should go? Are you thinking to 
override init() with super.init() and populate it in there? The way I 
understand it init may not suffice. The mapping may be changed in between, 
correct? 

> IllegalArgumentException can come from parsing val - adjust message 
> accordingly. This shouldn't be at INFO level.
Agree, changed to LOG.warn



> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077340#comment-16077340
 ] 

Hadoop QA commented on HBASE-16415:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green} 1{color} | {color:green} hbaseanti {color} | {color:green}  0m  
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green} 1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green} 1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 3s{color} | {color:green} master passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green}  1m 
23s{color} | {color:green} master passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green}  1m 
26s{color} | {color:green} master passed {color} |
| {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
27s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  6m 
13s{color} | {color:red} hbase-server in master has 10 extant Findbugs 
warnings. {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green} 1{color} | {color:green} hadoopcheck {color} | {color:green} 
83m 24s{color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 11m 
27s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}266m 10s{color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green} 1{color} | {color:green} asflicense {color} | {color:green}  0m 
40s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}387m 27s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleWAL |
|   | hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient |
|   | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
|   | hadoop.hbase.security.access.TestCoprocessorWhitelistMasterObserver |
|   | hadoop.hbase.master.procedure.TestProcedureAdmin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Issue | HBASE-16415 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12875952/HBASE-16415.patch |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 85f86c18e3e0 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016 
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 75d2eca |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC3 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7542/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7542/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/7542/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076926#comment-16076926
 ] 

Ted Yu commented on HBASE-16415:


For RedirectingInterClusterReplicationEndpoint.java, please move the javadoc 
starting with 'A {@link 
org.apache.hadoop.hbase.replication.ReplicationEndpoint}' to just above 'public 
class RedirectingInterClusterReplicationEndpoint'

Please add annotation for audience for 
RedirectingInterClusterReplicationEndpoint (Private).

For redirectEntries(), tableRedirectionsMap is populated. This action can be 
lifted outside redirectEntries() - redirection rules wouldn't change that often.
{code}
   } catch (IllegalArgumentException e) {
 LOG.info("Found unknown configuration key "   key   "in 
ReplicationPeerConfiguration");
{code}
IllegalArgumentException can come from parsing val - adjust message 
accordingly. This shouldn't be at INFO level.
{code}
 int n = Math.min(Math.min(this.maxThreads, entries.size()/100 1), 
numSinks);
{code}
How is 100 determined ?
{code}
   int index = 
Math.abs(Bytes.hashCode(e.getKey().getEncodedRegionName())%n);
{code}
Why is abs() needed ?

Please put patch on review board.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076870#comment-16076870
 ] 

Jan Kunigk commented on HBASE-16415:


I should also say that this has been tested with the aforementioned test class 
and on the hbase shell (in Pseudo Distributed more) up to now.

J 

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
> Attachments: HBASE-16415.patch
>
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071292#comment-16071292
 ] 

Jan Kunigk commented on HBASE-16415:


Thanks, I also just realised, that with Option 2 we could have a user configure 
two redirections with the same source, which is not something that the 
redirection code supports in this initial patch.

With Option 1, we can rule out such misconfiguration via the the Map Interfaces 
invariant, that it may never contain duplicate keys.

J 

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069541#comment-16069541
 ] 

Guanghao Zhang commented on HBASE-16415:


Option 1 may be better. And they are only used in 
RedirectingInterClusterReplicationEndpoint, I thought it is ok to use 
ns:tablename as the config key and value. Thanks.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064699#comment-16064699
 ] 

Jan Kunigk commented on HBASE-16415:


Thanks Guanghao,

I have an implementation working that successfully redirects replication in 
Pseudo-Distributed mode, albeit providing a hardcoded redirection configuration 
(in-code).
Now I am implementing proper configuration for the redirection.
I can think of two options to provide an external configuration for this:

Option 1) is to simply pass the redirections themselves as config keys and have 
the RedirectingInterClusterReplicationEndpoint attempt to (also) parse each 
config key as a ns:tablename spec. 
Map configuration = new Map();
configuration.put("ns1:t1", "ns2:t1");
configuration.put("ns1:tA", "ns1:tB");
This would translate into the following option to the add_peer command from the 
HBase shell: CONFIG => { "ns1:t1" => "ns2:t1", "ns1:tA" => "ns1:tB" }

Option 2) is to explicitly name the redirections config items via a name:
Map configuration = new Map();
configuration.put("redirections", "ns1:t1>>ns2:t1,ns1:tA>>ns1:tB");
This would translate into the following option to the add_peer command from the 
HBase shell: CONFIG => { "redirections" => "ns1:t1>>ns2:t1,ns1:tA>>ns1:tB"}

Option 1) has the advantage of avoiding special parsing and characters in the 
configuration specification.
Option 2) has the advantage of allowing unambiguous keys for the configuration, 
I am not sure that "namespacexyz:tableabc" (Option1) is a good choice for a 
configuration key. It implies that anything that is not a well-defined key is a 
redirection rule... Doesn't sound wise, but I did not want to exclude it...

Opinions? other proposals?

J


> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055644#comment-16055644
 ] 

Guanghao Zhang commented on HBASE-16415:


bq. making this a general interface vs. encapsulating this functionality in the 
new endpoint entirely
We don't need design too early. If we need this interface in the future, we can 
abstract it to the parent class in the future. Now encapsulating this 
functionality in the new endpoint entirely is enough. Thanks.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055630#comment-16055630
 ] 

Jan Kunigk commented on HBASE-16415:


I mean that, adding the method to ReplicationEndpoint would define a general 
interface for redirecting entries.
Other Endpoints (RegionReplicaReplicationEndpoint, or new future ones) could 
implement a redirectEntries() method too if they want to provide this, maybe 
with slightly different semantics.
Maybe those Endpoints would choose slightly different semantics for their own 
implementation of redirectEntries().
If they do not provide an implementation, then the empty default implementation 
from the interface will kick in.

If we implement redirection within 
RedirectingInterClusterReplicationEndpoint.createBatches(), the concept of 
redirection would not be exposed beyond 
RedirectingInterClusterReplicationEndpoint.

I do not have enough experience with the HBase codebase to weigh the merits of 
making this a general interface vs. encapsulating this functionality in the new 
endpoint entirely and will rely on your guidance.

I hope I could clarify my idea though.

J




> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055440#comment-16055440
 ] 

Guanghao Zhang commented on HBASE-16415:


bq. however then notion of redirection is locked into the 
RedirectingInterClusterReplicationEndpoint.
Can you explain more about this?
bq. Again, happy to override createBatches() as well.
If override createBatches() is enough to implement redirect, why need add a 
method to ReplicationEndpoint?

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055420#comment-16055420
 ] 

Jan Kunigk commented on HBASE-16415:


Hi Guanghao,

thanks for the comment. Happy to put it into createBatches() as well, however 
then notion of redirection is locked into the 
RedirectingInterClusterReplicationEndpoint.
May be the behaviour of redirecting entries is functionality, fairly common to 
a range of endpoints?

Currently I have actually added a 
{code}
  default int redirectEntries(ReplicateContext context) {
// This method should be overridden by subclasses which support redirection 
of source
// TableNames in the target
...
{code}
to ReplicationEndpoint.java

The invocation would not have to occur in the parent class of, but could be a 
general "trait" of the replication logic as part of the shipper thread shortly 
before replicate is called:
{code}
// Redirect the edits to another table in the target if the endpoint 
implements it and if any
// redirections are configured for the current batch of edits
// Returns zero if no implementation is provided
int redirected = 
source.getReplicationEndpoint().redirectEntries(replicateContext);
{code}

This could give (maybe future) endpoints other than "InterCluster" type 
endpoints the ability to implement the same behaviour in a different context.

I'd be interested about your opinion on the above points.
Again, happy to override createBatches() as well.
Either way, looks like we're very close to finalizing the discussion before I 
implement a patch :)

Best, J





> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054827#comment-16054827
 ] 

Guanghao Zhang commented on HBASE-16415:


Why not extends HBaseInterClusterReplicationEndpoint and only override 
createBatches method?

bq. It seems option 3 is better.
redirect() is only a behavior of RedirectingInterClusterReplicationEndpoint. So 
it is not good to add it to the parent class? Thanks.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054113#comment-16054113
 ] 

Ted Yu commented on HBASE-16415:


It seems option 3 is better. It is Okay to have an empty (default in Java 8) 
abstract method.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053870#comment-16053870
 ] 

Jan Kunigk commented on HBASE-16415:


Thanks Ted.

I am in the process of implementing the logic of entry redirection in 
RedirectingInterClusterReplicationEndpoint, but wonder how I should combine 
that class' replicate implementation with redirection.
I would very much appreciate the community once more, while I see several 
options:

1. Embed, the logic of redirections into the replicate() method, which would 
mean
a) Overriding replicate of HBaseInterClusterReplicationEndpoint, but also
b) duplicating the actual code in 
HBaseInterClusterReplicationEndpoint.replicate(), which I don't think is a good 
implementation

2. Implement the logic of redirections in a completely new method redirect(), 
which needs to be called from within instances of 
HBaseInterClusterReplicationEndpoint or 
RedirectingInterClusterReplicationEndpoint
We could wrap the content of replicate() into a doReplicate() method in 
HBaseInterClusterReplicationEndpoint.
HBaseInterClusterReplicationEndpoint would then only call doReplicate() in 
replicate()
RedirectingInterClusterReplicationEndpoint would call redirect(); doReplicate() 
as part of replicate()
However, this solution puts the functionality of redirection way down into 
HBaseInterClusterReplicationEndpoint, which may not be a good idea.

3. Implement redirect() as an abstract method in HBaseReplicationEndpoint 
(which is the abstract base class for HBaseInterClusterReplicationEndpoint and 
RegionReplicaReplicationEndpoint.
The redirect() implementation of both HBaseInterClusterReplicationEndpoint and 
RegionReplicaReplicationEndpoint would not do anything (optionally logs this). 
RedirectingInterClusterReplicationEndpoint.replicate() overrides the empty 
implementation with actually redirecting entries.
While this would lead to "empty" abstract method implementations, it appears to 
me as the cleanest and most straightforward way to achieve the redirection 
logic to occur in the ReplicationEndpoint.

4. Copy the behaviour of getWALEntryfilter() in BaseReplicationEndpoint (This 
is more involved, but avoids empty abstract method implementations):
return filters.isEmpty() ? null : new ChainWALEntryFilter(filters);
1. Implement a getRedirector() function in BaseReplicationEnpoint, which 
returns an instance of a novel Redirector class, implementing the redirections 
logic.
2. If no explicit Redirectors are set in the endpoint, return an empty 
redirector (NullRedirector, etc.)
3. Provide a redirectEntries() function in BaseReplicationEndpoint which 
will invoke getRedirector() and Redirector.redirect() _if_ getRedirector() != 
null
4. The version in BaseReplicationEnpoint will return an empty redirector 
(or null) for getRedirector(), which means that direct subclasses 
(HBaseInterClusterReplicationEndpoint or RegionReplicaReplicationEndpoint) will 
also return null and not redirect anything
5. The version in RedirectingInterClusterReplicationEndpoint will overwrite 
getRedirector() and return a Redirector, which implements 
6. Embed redirectEntries() into the replicate() method of 
HBaseInterClusterReplicationEndpoint (or all other classes who want to reserve 
the option option of redirection) 


Again, appreciate your feedback, J

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051159#comment-16051159
 ] 

Ted Yu commented on HBASE-16415:


Looking at the existing classes, not all of them start with HBase.
e.g. RegionReplicaReplicationEndpoint

I think RedirectingInterClusterReplicationEndpoint should be good enough.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050980#comment-16050980
 ] 

Jan Kunigk commented on HBASE-16415:


Yes, that makes sense, thanks for clarifying.
I would extend {code} HBaseInterClusterReplicationEndpoint {code}.

Any restriction / guidance on the naming? I would use {code} 
HBaseRedirectingInterClusterReplicationEndpoint {code} for the new subclass.

J

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050708#comment-16050708
 ] 

Guanghao Zhang commented on HBASE-16415:


I thought you don't need change the replication shipper thread. You only need 
to add a new ReplicationEndpoint and implement replicate() method. You can 
embed the redirections information there. Thanks.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16050466#comment-16050466
 ] 

Jan Kunigk commented on HBASE-16415:


Hi Guanghao, thanks for your feedback.
Yes, I agree it would be better to use a new ReplicationEndpoint.

The ReplicationEndpoint is passed into the run() method of ReplicaitonSource:
```
// get the WALEntryFilter from ReplicationEndpoint and add it to default 
filters
ArrayList filters = Lists.newArrayList(
  (WALEntryFilter)new SystemTableWALEntryFilter());
WALEntryFilter filterFromEndpoint = 
this.replicationEndpoint.getWALEntryfilter();
```
then at the end of this method a new attempt for starting the shipper thread 
(i.e. ReplicationSourceWALReaderThread) is launched:
```
  tryStartNewShipperThread(walGroupId, queue);
```
tryStartNewShipperThread() is invoking startNewWALReaderThread, which returns a 
ReplicationSourceWALReaderThread and also applies all filters to it:
```
   ChainWALEntryFilter readerFilter = new ChainWALEntryFilter(filters);

ReplicationSourceWALReaderThread walReader = new 
ReplicationSourceWALReaderThread(manager,
replicationQueueInfo, queue, startPosition, fs, conf, readerFilter, 
metrics);
```
I agree, that embedding the redirections information into a new 
ReplicationEndpoint just like we do with the filters makes sense.
But, the inspection of the individual WAL entries would still have to occur in 
the Shipper Threads itself
(like
```
entry = filterEntry(entry);
entry = redirectEntry(entry);
```
)

Do you agree? Or am I missing something? 

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046157#comment-16046157
 ] 

Guanghao Zhang commented on HBASE-16415:


Now the replication source read the wal entries by reader thread and filter wal 
entries by the filter config (replication scope, perr table-cfs, peer 
namespaces, etc.). Then ship them by shipper thread. They were write to peer 
cluster by replication endpoint. And we already suppose the pluggable 
replication endpoint. So I thought we should implement this feature 
(replication in different namespace) by implementing a new replication 
endpoint. And use the config now for replication endpoint. Your proposal will 
make the replication source more complicate. IMO it is not a good idea.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045637#comment-16045637
 ] 

Ted Yu commented on HBASE-16415:


Before posting your patch, see if you can incorporate Yu's suggestion.

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>Assignee: Jan Kunigk
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044353#comment-16044353
 ] 

Jan Kunigk commented on HBASE-16415:


Can someone assign me please? I do not seem to have that privilege.
 

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jan Kunigk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044348#comment-16044348
 ] 

Jan Kunigk commented on HBASE-16415:


Hi, I would like to make the following
contribution to this, looking very much forward to feedback and comments:

ReplicationSourceWALReaderThread continuously follows WALEntries to be
replicated for a specified WAL via WAL.Reader's next() method and adds them
to WALEntryBatches

As far as I can see, those WALEntries are copies of the originally
persisted local WALs. In order to direct these Entries to TableNames,
different to the source, I propose to intercept the copied WALEntries on
the source cluster and probe if they belong to a TableName, which is to be
re-written.

If such a probe is successful, then the WALKey of any such WALEntry needs
to be changed accordingly. WALKey provides a getTableName() method, but
currently not a setTableName() method, which would simply have to be added
to change the private TableName member.

I propose to intercept the entries via a new method redirectEntry(), which
is invoked shortly before the entry is added to its WALEntryBatch and
immediately after the entry has been filtered by filterEntry() like so:

Entry entry = entryStream.next();
if (updateSerialReplPos(batch, entry)) {
  batch.lastWalPosition = entryStream.getPosition();
  break;
}
entry = filterEntry(entry);
entry = redirectEntry(entry); // <--
if (entry != null) {
  WALEdit edit = entry.getEdit();
  if (edit != null && !edit.isEmpty()) {
long entrySize = getEntrySize(entry);
batch.addEntry(entry);

redirectEntry() bases its decisions on a 'Map
redirections', where the keys are the source table name and the values the
destination table name. The Map would be included in the
ReplicationPeerConfig, which can be obtained from within
ReplicationSourceWALReaderThread via the instance of
ReplicationSourceManager, which is in turn passed as an argument to both
available constructors.

When a TableName object from a WALKey from the WALEntryStream matches the
key of any of the entries in the redirections map, that WALKey's TableName
is replaced by the the value of that entry.

The rationale for intercepting on the sending side is that the setup and
peer management is performed on the source today already and there is no
mechanism I can see which would carry the redirection rules themselves
across.

Similarly to the way that the hbase shell allows to specify the tables and
column families to be replicated (set_peer_table_CFs), I propose a new
command (also on the sending side) 'set_peer_table_redirections', which
accepts a map of Strings, corresponding to the required final specification
of the redirections as TableNames:

set_peer_redirections['ns_source1:table_source1' : 'ns_dest1:table_dest1',
'ns_source2:table_source2' : 'ns_dest2:table_dest2', ...
'ns_sourcen:table_sourcen' : 'ns_destn:table_destn', ]

Best, J

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16044168#comment-16044168
 ] 

Yu Li commented on HBASE-16415:
---

FWIW, taking usage of {{NamespaceGroupingStrategy}} introduced in HBASE-14456 
together with multiwal may make the design simpler 

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HBASE-16415) Replication in different namespace

[Jean-Marc Spaggiari (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025323#comment-16025323
 ] 

Jean-Marc Spaggiari commented on HBASE-16415:
-

Indeed, it will be very nice! Anyone want to take this?

> Replication in different namespace
> --
>
> Key: HBASE-16415
> URL: https://issues.apache.org/jira/browse/HBASE-16415
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: Christian Guegi
>
> It would be nice to replicate tables from one namespace to another namespace.
> Example:
> Master cluster, namespace=default, table=bar
> Slave cluster, namespace=dr, table=bar
> Replication happens in class ReplicationSink:
>   public void replicateEntries(List entries, final CellScanner 
> cells, ...){
> ...
> TableName table = 
> TableName.valueOf(entry.getKey().getTableName().toByteArray());
> ...
> addToHashMultiMap(rowMap, table, clusterIds, m);
> ...
> for (Entry> entry : 
> rowMap.entrySet()) {
>   batch(entry.getKey(), entry.getValue().values());
> }
>}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)