subject:"\[jira\] \[Commented\] \(HBASE\-7709\) Infinite loop possible in Master\/Master replication"

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Jean-Daniel Cryans (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752757#comment-13752757
]

Jean-Daniel Cryans commented on HBASE-7709:
---

Infinite loop possible in Master/Master replication
---

Key: HBASE-7709
URL: https://issues.apache.org/jira/browse/HBASE-7709
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 0.94.6, 0.95.1
Reporter: Lars Hofhansl
Assignee: Vasu Mariyala
Fix For: 0.98.0, 0.94.12, 0.96.0

Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch,
0.95-trunk-rev2.patch, 0.95-trunk-rev3.patch, 0.95-trunk-rev4.patch,
HBASE-7709.patch, HBASE-7709-rev1.patch, HBASE-7709-rev2.patch,
HBASE-7709-rev3.patch, HBASE-7709-rev4.patch, HBASE-7709-rev5.patch

We just discovered the following scenario:
# Cluster A and B are setup in master/master replication
# By accident we had Cluster C replicate to Cluster A.
Now all edit originating from C will be bouncing between A and B. Forever!
The reason is that when the edit come in from C the cluster ID is already set
and won't be reset.
We have a couple of options here:
# Optionally only support master/master (not cycles of more than two
clusters). In that case we can always reset the cluster ID in the
ReplicationSource. That means that now cycles 2 will have the data cycle
forever. This is the only option that requires no changes in the HLog format.
# Instead of a single cluster id per edit maintain a (unordered) set of
cluster id that have seen this edit. Then in ReplicationSource we drop any
edit that the sink has seen already. The is the cleanest approach, but it
might need a lot of data stored per edit if there are many clusters involved.
# Maintain a configurable counter of the maximum cycle side we want to
support. Could default to 10 (even maybe even just). Store a hop-count in the
WAL and the ReplicationSource increases that hop-count on each hop. If we're
over the max, just drop the edit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread stack (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752762#comment-13752762
]

stack commented on HBASE-7709:
--

Applied to 0.95 and to trunk. Want this in 0.94 [~lhofhansl]?
[~vasu.mariy...@gmail.com] Thanks boss. Any chance of a release note on this
issue?

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Lars Hofhansl (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752818#comment-13752818
]

Lars Hofhansl commented on HBASE-7709:
--

Yeah, will sync up with Vasu off line and probably commit to 0.94 soon.

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Lars Hofhansl (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752841#comment-13752841
]

Lars Hofhansl commented on HBASE-7709:
--

Will commit to 0.94 later today if there are no objections.

Infinite loop possible in Master/Master replication
---

Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch,
0.95-trunk-rev2.patch, 0.95-trunk-rev3.patch, 0.95-trunk-rev4.patch,
7709-0.94-rev6.txt, HBASE-7709.patch, HBASE-7709-rev1.patch,
HBASE-7709-rev2.patch, HBASE-7709-rev3.patch, HBASE-7709-rev4.patch,
HBASE-7709-rev5.patch

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752858#comment-13752858
]

Hadoop QA commented on HBASE-7709:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12600457/7709-0.94-rev6.txt
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6953//console

This message is automatically generated.

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752911#comment-13752911
 ] 

Hudson commented on HBASE-7709:
---

SUCCESS: Integrated in hbase-0.95 #500 (See 
[https://builds.apache.org/job/hbase-0.95/500/])
HBASE-7709 Infinite loop possible in Master/Master replication (stack: rev 
1518334)
* 
/hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* 
/hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
* /hbase/branches/0.95/hbase-protocol/src/main/protobuf/WAL.proto
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BaseRowProcessor.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowProcessor.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotLogSplitter.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


 Infinite loop possible in Master/Master replication
 ---

 Key: HBASE-7709
 URL: https://issues.apache.org/jira/browse/HBASE-7709
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6, 0.95.1
Reporter: Lars Hofhansl
Assignee: Vasu Mariyala
 Fix For: 0.98.0, 0.94.12, 0.96.0

 Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch, 
 0.95-trunk-rev2.patch, 0.95-trunk-rev3.patch, 0.95-trunk-rev4.patch, 
 7709-0.94-rev6.txt, HBASE-7709.patch, HBASE-7709-rev1.patch, 
 HBASE-7709-rev2.patch, HBASE-7709-rev3.patch, HBASE-7709-rev4.patch, 
 HBASE-7709-rev5.patch


  We just discovered the following scenario:
 # Cluster A and B are setup in master/master replication
 # By accident we had Cluster C replicate to Cluster A.
 Now all edit originating from C will be bouncing between A and B. Forever!
 The reason is that when the edit come in from C the cluster ID is already set 
 and won't be reset.
 We have a couple of options here:
 # Optionally only support master/master (not cycles of more than two 
 clusters). In that case we can always reset the cluster ID in the 
 ReplicationSource. That means that now cycles  2 will have the data cycle 
 forever. This is the only option that requires no changes in the HLog format.
 # Instead of a single cluster id per edit maintain a (unordered) set of 
 cluster id that have seen this edit. Then in ReplicationSource we drop any 
 edit that the sink has seen already. The is the cleanest approach, but it 
 might need a lot of data stored per edit if there are many clusters involved.
 # Maintain a configurable counter of the maximum cycle side we want to 
 support. Could default to 10 (even maybe even just). Store a hop-count in the 
 WAL and the ReplicationSource increases that hop-count on each hop. If we're 
 over the max, just drop the edit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13752920#comment-13752920
 ] 

Hudson commented on HBASE-7709:
---

SUCCESS: Integrated in HBase-TRUNK #4441 (See 
[https://builds.apache.org/job/HBase-TRUNK/4441/])
HBASE-7709 Infinite loop possible in Master/Master replication (stack: rev 
1518335)
* 
/hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* 
/hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
* /hbase/trunk/hbase-protocol/src/main/protobuf/WAL.proto
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BaseRowProcessor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowProcessor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotLogSplitter.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


 Infinite loop possible in Master/Master replication
 ---

 Key: HBASE-7709
 URL: https://issues.apache.org/jira/browse/HBASE-7709
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6, 0.95.1
Reporter: Lars Hofhansl
Assignee: Vasu Mariyala
 Fix For: 0.98.0, 0.94.12, 0.96.0

 Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch, 
 0.95-trunk-rev2.patch, 0.95-trunk-rev3.patch, 0.95-trunk-rev4.patch, 
 7709-0.94-rev6.txt, HBASE-7709.patch, HBASE-7709-rev1.patch, 
 HBASE-7709-rev2.patch, HBASE-7709-rev3.patch, HBASE-7709-rev4.patch, 
 HBASE-7709-rev5.patch


  We just discovered the following scenario:
 # Cluster A and B are setup in master/master replication
 # By accident we had Cluster C replicate to Cluster A.
 Now all edit originating from C will be bouncing between A and B. Forever!
 The reason is that when the edit come in from C the cluster ID is already set 
 and won't be reset.
 We have a couple of options here:
 # Optionally only support master/master (not cycles of more than two 
 clusters). In that case we can always reset the cluster ID in the 
 ReplicationSource. That means that now cycles  2 will have the data cycle 
 forever. This is the only option that requires no changes in the HLog format.
 # Instead of a single cluster id per edit maintain a (unordered) set of 
 cluster id that have seen this edit. Then in ReplicationSource we drop any 
 edit that the sink has seen already. The is the cleanest approach, but it 
 might need a lot of data stored per edit if there are many clusters involved.
 # Maintain a configurable counter of the maximum cycle side we want to 
 support. Could default to 10 (even maybe even just). Store a hop-count in the 
 WAL and the ReplicationSource increases that hop-count on each hop. If we're 
 over the max, just drop the edit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753119#comment-13753119
 ] 

Hudson commented on HBASE-7709:
---

SUCCESS: Integrated in hbase-0.95-on-hadoop2 #276 (See 
[https://builds.apache.org/job/hbase-0.95-on-hadoop2/276/])
HBASE-7709 Infinite loop possible in Master/Master replication (stack: rev 
1518334)
* 
/hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* 
/hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
* /hbase/branches/0.95/hbase-protocol/src/main/protobuf/WAL.proto
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BaseRowProcessor.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowProcessor.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
/hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotLogSplitter.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
* 
/hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


 Infinite loop possible in Master/Master replication
 ---

 Key: HBASE-7709
 URL: https://issues.apache.org/jira/browse/HBASE-7709
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6, 0.95.1
Reporter: Lars Hofhansl
Assignee: Vasu Mariyala
 Fix For: 0.98.0, 0.94.12, 0.96.0

 Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch, 
 0.95-trunk-rev2.patch, 0.95-trunk-rev3.patch, 0.95-trunk-rev4.patch, 
 7709-0.94-rev6.txt, HBASE-7709.patch, HBASE-7709-rev1.patch, 
 HBASE-7709-rev2.patch, HBASE-7709-rev3.patch, HBASE-7709-rev4.patch, 
 HBASE-7709-rev5.patch


  We just discovered the following scenario:
 # Cluster A and B are setup in master/master replication
 # By accident we had Cluster C replicate to Cluster A.
 Now all edit originating from C will be bouncing between A and B. Forever!
 The reason is that when the edit come in from C the cluster ID is already set 
 and won't be reset.
 We have a couple of options here:
 # Optionally only support master/master (not cycles of more than two 
 clusters). In that case we can always reset the cluster ID in the 
 ReplicationSource. That means that now cycles  2 will have the data cycle 
 forever. This is the only option that requires no changes in the HLog format.
 # Instead of a single cluster id per edit maintain a (unordered) set of 
 cluster id that have seen this edit. Then in ReplicationSource we drop any 
 edit that the sink has seen already. The is the cleanest approach, but it 
 might need a lot of data stored per edit if there are many clusters involved.
 # Maintain a configurable counter of the maximum cycle side we want to 
 support. Could default to 10 (even maybe even just). Store a hop-count in the 
 WAL and the ReplicationSource increases that hop-count on each hop. If we're 
 over the max, just drop the edit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753140#comment-13753140
 ] 

Hudson commented on HBASE-7709:
---

SUCCESS: Integrated in HBase-0.94-security #274 (See 
[https://builds.apache.org/job/HBase-0.94-security/274/])
HBASE-7709 Infinite loop possible in Master/Master replication (Vasu Mariyala) 
(larsh: rev 1518410)
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


 Infinite loop possible in Master/Master replication
 ---

 Key: HBASE-7709
 URL: https://issues.apache.org/jira/browse/HBASE-7709
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6, 0.95.1
Reporter: Lars Hofhansl
Assignee: Vasu Mariyala
 Fix For: 0.98.0, 0.94.12, 0.96.0

 Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch, 
 0.95-trunk-rev2.patch, 0.95-trunk-rev3.patch, 0.95-trunk-rev4.patch, 
 7709-0.94-rev6.txt, HBASE-7709.patch, HBASE-7709-rev1.patch, 
 HBASE-7709-rev2.patch, HBASE-7709-rev3.patch, HBASE-7709-rev4.patch, 
 HBASE-7709-rev5.patch


  We just discovered the following scenario:
 # Cluster A and B are setup in master/master replication
 # By accident we had Cluster C replicate to Cluster A.
 Now all edit originating from C will be bouncing between A and B. Forever!
 The reason is that when the edit come in from C the cluster ID is already set 
 and won't be reset.
 We have a couple of options here:
 # Optionally only support master/master (not cycles of more than two 
 clusters). In that case we can always reset the cluster ID in the 
 ReplicationSource. That means that now cycles  2 will have the data cycle 
 forever. This is the only option that requires no changes in the HLog format.
 # Instead of a single cluster id per edit maintain a (unordered) set of 
 cluster id that have seen this edit. Then in ReplicationSource we drop any 
 edit that the sink has seen already. The is the cleanest approach, but it 
 might need a lot of data stored per edit if there are many clusters involved.
 # Maintain a configurable counter of the maximum cycle side we want to 
 support. Could default to 10 (even maybe even just). Store a hop-count in the 
 WAL and the ReplicationSource increases that hop-count on each hop. If we're 
 over the max, just drop the edit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753164#comment-13753164
 ] 

Hudson commented on HBASE-7709:
---

SUCCESS: Integrated in HBase-0.94 #1128 (See 
[https://builds.apache.org/job/HBase-0.94/1128/])
HBASE-7709 Infinite loop possible in Master/Master replication (Vasu Mariyala) 
(larsh: rev 1518410)
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEdit.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


 Infinite loop possible in Master/Master replication
 ---

 Key: HBASE-7709
 URL: https://issues.apache.org/jira/browse/HBASE-7709
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6, 0.95.1
Reporter: Lars Hofhansl
Assignee: Vasu Mariyala
 Fix For: 0.98.0, 0.94.12, 0.96.0

 Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch, 
 0.95-trunk-rev2.patch, 0.95-trunk-rev3.patch, 0.95-trunk-rev4.patch, 
 7709-0.94-rev6.txt, HBASE-7709.patch, HBASE-7709-rev1.patch, 
 HBASE-7709-rev2.patch, HBASE-7709-rev3.patch, HBASE-7709-rev4.patch, 
 HBASE-7709-rev5.patch


  We just discovered the following scenario:
 # Cluster A and B are setup in master/master replication
 # By accident we had Cluster C replicate to Cluster A.
 Now all edit originating from C will be bouncing between A and B. Forever!
 The reason is that when the edit come in from C the cluster ID is already set 
 and won't be reset.
 We have a couple of options here:
 # Optionally only support master/master (not cycles of more than two 
 clusters). In that case we can always reset the cluster ID in the 
 ReplicationSource. That means that now cycles  2 will have the data cycle 
 forever. This is the only option that requires no changes in the HLog format.
 # Instead of a single cluster id per edit maintain a (unordered) set of 
 cluster id that have seen this edit. Then in ReplicationSource we drop any 
 edit that the sink has seen already. The is the cleanest approach, but it 
 might need a lot of data stored per edit if there are many clusters involved.
 # Maintain a configurable counter of the maximum cycle side we want to 
 support. Could default to 10 (even maybe even just). Store a hop-count in the 
 WAL and the ReplicationSource increases that hop-count on each hop. If we're 
 over the max, just drop the edit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-28 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13753204#comment-13753204
 ] 

Hudson commented on HBASE-7709:
---

FAILURE: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #700 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/700/])
HBASE-7709 Infinite loop possible in Master/Master replication (stack: rev 
1518335)
* 
/hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
* 
/hbase/trunk/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/WALProtos.java
* /hbase/trunk/hbase-protocol/src/main/protobuf/WAL.proto
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BaseRowProcessor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RowProcessor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotLogSplitter.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


 Infinite loop possible in Master/Master replication
 ---

 Key: HBASE-7709
 URL: https://issues.apache.org/jira/browse/HBASE-7709
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6, 0.95.1
Reporter: Lars Hofhansl
Assignee: Vasu Mariyala
 Fix For: 0.98.0, 0.94.12, 0.96.0

 Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch, 
 0.95-trunk-rev2.patch, 0.95-trunk-rev3.patch, 0.95-trunk-rev4.patch, 
 7709-0.94-rev6.txt, HBASE-7709.patch, HBASE-7709-rev1.patch, 
 HBASE-7709-rev2.patch, HBASE-7709-rev3.patch, HBASE-7709-rev4.patch, 
 HBASE-7709-rev5.patch


  We just discovered the following scenario:
 # Cluster A and B are setup in master/master replication
 # By accident we had Cluster C replicate to Cluster A.
 Now all edit originating from C will be bouncing between A and B. Forever!
 The reason is that when the edit come in from C the cluster ID is already set 
 and won't be reset.
 We have a couple of options here:
 # Optionally only support master/master (not cycles of more than two 
 clusters). In that case we can always reset the cluster ID in the 
 ReplicationSource. That means that now cycles  2 will have the data cycle 
 forever. This is the only option that requires no changes in the HLog format.
 # Instead of a single cluster id per edit maintain a (unordered) set of 
 cluster id that have seen this edit. Then in ReplicationSource we drop any 
 edit that the sink has seen already. The is the cleanest approach, but it 
 might need a lot of data stored per edit if there are many clusters involved.
 # Maintain a configurable counter of the maximum cycle side we want to 
 support. Could default to 10 (even maybe even just). Store a hop-count in the 
 WAL and the ReplicationSource increases that hop-count on each hop. If we're 
 over the max, just drop the edit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-27 Thread Ted Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13751447#comment-13751447
]

Ted Yu commented on HBASE-7709:
---

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-26 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13750192#comment-13750192
]

Hadoop QA commented on HBASE-7709:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12599751/HBASE-7709-rev5.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6896//console

This message is automatically generated.

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-26 Thread Vasu Mariyala (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13750289#comment-13750289
]

Vasu Mariyala commented on HBASE-7709:
--

The patch HBASE-7709-rev5.patch is on the top of 0.94 and hence the hadoop qa
would always fail while applying this patch on trunk. Can any one please run
the hadoop qa build for the patch 0.95-trunk-rev4.patch (which is the trunk
and 0.95 patch)?

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-26 Thread Ted Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13750297#comment-13750297
]

Ted Yu commented on HBASE-7709:
---

Please attach 0.95-trunk-rev4.patch one more time - Hadoop QA picks up the
latest attachment

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-26 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13750559#comment-13750559
]

Hadoop QA commented on HBASE-7709:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1250/0.95-trunk-rev4.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.

{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.

{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}. The applied patch generated 2 release
audit warnings (more than the trunk's current 0 warnings).

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6902//console

This message is automatically generated.

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-26 Thread Vasu Mariyala (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13750617#comment-13750617
]

Vasu Mariyala commented on HBASE-7709:
--

The release audit warnings are not related to the patch. This has to do with
the missing licenses in the below files. After correcting the license info in
these files, the release audit is successful

{code}
***

Unapproved licenses:

/home/vmariyala/bigdata-dev/testhbase/hbase-server/src/main/resources/hbase-webapps/static/css/bootstrap-theme.min.css

/home/vmariyala/bigdata-dev/testhbase/hbase-server/src/main/resources/hbase-webapps/static/css/bootstrap-theme.css

***
{code}

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-26 Thread Jeffrey Zhong (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13750727#comment-13750727
]

Jeffrey Zhong commented on HBASE-7709:
--

I reviewed 0.94 and trunk patch. They both looks good to me! +1 from me.
Thanks.
In the trunk, we currently carry all clusterIds in the replication path and we
could optimize this later when there is a need.

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-25 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749550#comment-13749550
]

Hadoop QA commented on HBASE-7709:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12599751/HBASE-7709-rev5.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6866//console

This message is automatically generated.

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-24 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749520#comment-13749520
]

Hadoop QA commented on HBASE-7709:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12599751/HBASE-7709-rev5.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6865//console

This message is automatically generated.

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-23 Thread Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13748685#comment-13748685
 ] 

Ted Yu commented on HBASE-7709:
---

{code}
+  public void setClusters(SetUUID clusterIds) {
{code}
Would setClusterIds() be better name for the above method ?
{code}
+   * @return the set of clusters that have consumed the mutation
{code}
'set of clusters' - 'set of cluster Ids'
{code}
+  public SetUUID getClusters() {
{code}
getClusters - getClusterIds
{code}
-private UUID clusterId;
+private SetUUID clusters;
{code}
clusters - clusterIds

If you agree with the above comments, please modify names in other places as 
well.

 Infinite loop possible in Master/Master replication
 ---

 Key: HBASE-7709
 URL: https://issues.apache.org/jira/browse/HBASE-7709
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6, 0.95.1
Reporter: Lars Hofhansl
Assignee: Vasu Mariyala
 Fix For: 0.98.0, 0.94.12, 0.96.0

 Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch, 
 0.95-trunk-rev2.patch, HBASE-7709.patch, HBASE-7709-rev1.patch, 
 HBASE-7709-rev2.patch, HBASE-7709-rev3.patch


  We just discovered the following scenario:
 # Cluster A and B are setup in master/master replication
 # By accident we had Cluster C replicate to Cluster A.
 Now all edit originating from C will be bouncing between A and B. Forever!
 The reason is that when the edit come in from C the cluster ID is already set 
 and won't be reset.
 We have a couple of options here:
 # Optionally only support master/master (not cycles of more than two 
 clusters). In that case we can always reset the cluster ID in the 
 ReplicationSource. That means that now cycles  2 will have the data cycle 
 forever. This is the only option that requires no changes in the HLog format.
 # Instead of a single cluster id per edit maintain a (unordered) set of 
 cluster id that have seen this edit. Then in ReplicationSource we drop any 
 edit that the sink has seen already. The is the cleanest approach, but it 
 might need a lot of data stored per edit if there are many clusters involved.
 # Maintain a configurable counter of the maximum cycle side we want to 
 support. Could default to 10 (even maybe even just). Store a hop-count in the 
 WAL and the ReplicationSource increases that hop-count on each hop. If we're 
 over the max, just drop the edit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-23 Thread Jeffrey Zhong (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13748982#comment-13748982
]

Jeffrey Zhong commented on HBASE-7709:
--

I reviewed the trunk patch. One thing I noticed that the trunk patch deprecats
clusterId and related code. I think we should still keep it around. The reason
is that one of the semantics of clusterId is the Original ClusterId where the
changes are generated. This information will be very useful when we build
monitoring dashboard to show how many edits from each source cluster. Similarly
we could combine the original cluster Id and write time to know replication
latency from source to current cluster.
The rest looks good. Thanks.

Infinite loop possible in Master/Master replication
---

Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch,
0.95-trunk-rev2.patch, 0.95-trunk-rev3.patch, HBASE-7709.patch,
HBASE-7709-rev1.patch, HBASE-7709-rev2.patch, HBASE-7709-rev3.patch,
HBASE-7709-rev4.patch

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-23 Thread Vasu Mariyala (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749081#comment-13749081
]

Vasu Mariyala commented on HBASE-7709:
--

[~jeffreyz] This cluster information is only stored as part of the HLog and it
gets rolled. So do you think it is the place from where we read the information
about the originating cluster to build such metrics?

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-23 Thread Lars Hofhansl (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749166#comment-13749166
]

Lars Hofhansl commented on HBASE-7709:
--

This does raise a good point. Maybe we should store the cluster ids in order of
traversal. That would later allow us to reconstruct the replication path
between clusters and display it in the shell.

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-23 Thread Ted Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13749171#comment-13749171
]

Ted Yu commented on HBASE-7709:
---

bq. we should store the cluster ids in order of traversal
+1

Infinite loop possible in Master/Master replication
---

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-22 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13748263#comment-13748263
]

Hadoop QA commented on HBASE-7709:
--

{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12599256/0.95-trunk-rev2.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.

{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.

{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6850//console

This message is automatically generated.

Infinite loop possible in Master/Master replication
---

Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch,
0.95-trunk-rev2.patch, HBASE-7709.patch, HBASE-7709-rev1.patch,
HBASE-7709-rev2.patch, HBASE-7709-rev3.patch

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-21 Thread Lars Hofhansl (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746586#comment-13746586
]

Lars Hofhansl commented on HBASE-7709:
--

The 0.94 patch looks good. Bit large, but then again this is a bad bug to have
(when it hits you you'll useless load on your cluster forever, throwing your
versions off, etc).
Nice refactoring of the replication test.

Few nits:
* PREFIX_CLUSTER_KEY in WALEdit could just be '_', right? No need to store that
longer prefix everywhere.
* Similarly maybe make PREFIX_CONSUMED_CLUSTER_IDS in Mutation just _cs.id
* The comment for scopes in WALEdit could be a bit more explicit that we're
overloading scopes with the cluster id for backwards compatibility.

+1 otherwise (assuming the full 0.94 test suite passes)

Looking at trunk patch now.

Infinite loop possible in Master/Master replication
---

Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch,
HBASE-7709.patch, HBASE-7709-rev1.patch, HBASE-7709-rev2.patch

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-21 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746611#comment-13746611
 ] 

Lars Hofhansl commented on HBASE-7709:
--

In trunk:
* should repeated UUID clusters = 8 in WAL.proto? Otherwise we can't read old 
log entries. But maybe that's not a problem...?
* in Import:
{code}
+clusters = new HashSetUUID();
+clusters.add(ZKClusterId.getUUIDForCluster(zkw));
{code}
Can be written as {{cluster = 
Collections.Collections.singleton(ZKClusterId.getUUIDForCluster(zkw))}}
* Is this right?
{code}
+  for(UUID clusterId : key.getClusters()) {
 uuidBuilder.setLeastSigBits(clusterId.getLeastSignificantBits());
 uuidBuilder.setMostSigBits(clusterId.getMostSignificantBits());
+keyBuilder.addClusters(uuidBuilder.build());
{code}
addClusters expects a Set.
* Where is HlogKey.PREFIX_CLUSTER_KEY used? Just to read old versions of 
WALEdits? Need to discuss if that is necessary. [~stack]? This has to do with 
upgrading WALEdits from pre 0.95.

Otherwise looks great.

 Infinite loop possible in Master/Master replication
 ---

 Key: HBASE-7709
 URL: https://issues.apache.org/jira/browse/HBASE-7709
 Project: HBase
  Issue Type: Bug
  Components: Replication
Affects Versions: 0.94.6, 0.95.1
Reporter: Lars Hofhansl
Assignee: Vasu Mariyala
 Fix For: 0.98.0, 0.94.12, 0.96.0

 Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch, 
 HBASE-7709.patch, HBASE-7709-rev1.patch, HBASE-7709-rev2.patch


  We just discovered the following scenario:
 # Cluster A and B are setup in master/master replication
 # By accident we had Cluster C replicate to Cluster A.
 Now all edit originating from C will be bouncing between A and B. Forever!
 The reason is that when the edit come in from C the cluster ID is already set 
 and won't be reset.
 We have a couple of options here:
 # Optionally only support master/master (not cycles of more than two 
 clusters). In that case we can always reset the cluster ID in the 
 ReplicationSource. That means that now cycles  2 will have the data cycle 
 forever. This is the only option that requires no changes in the HLog format.
 # Instead of a single cluster id per edit maintain a (unordered) set of 
 cluster id that have seen this edit. Then in ReplicationSource we drop any 
 edit that the sink has seen already. The is the cleanest approach, but it 
 might need a lot of data stored per edit if there are many clusters involved.
 # Maintain a configurable counter of the maximum cycle side we want to 
 support. Could default to 10 (even maybe even just). Store a hop-count in the 
 WAL and the ReplicationSource increases that hop-count on each hop. If we're 
 over the max, just drop the edit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-21 Thread Vasu Mariyala (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746781#comment-13746781
]

Vasu Mariyala commented on HBASE-7709:
--

Attached the patches for 0.94 (HBASE-7709-rev3.patch) and 0.95,
trunk(0.95-trunk-rev2.patch) which addresses the nits mentioned by Lars

0.94

a) Changed PREFIX_CLUSTER_KEY to '.' (period as the column family names
can't start with it)

b) PREFIX_CONSUMED_CLUSTER_IDS changed to _cs.id

c) A comment has been added in WALEdit mentioning that it is done for
backwards compatibility and has been removed in 0.95.2+ releases

trunk/0.95

a) From protobuf documentation

repeated: this field can be repeated any number of times (including zero)
in a well-formed message. The order of the repeated values will be preserved..
optional: a well-formed message can have zero or one of this field (but
not more than one).

So does repeated imply it is optional? Also, from the WALProtos.java the
clusters list is initialized to empty list in the initFields() method so we
would not get any NullPointerException. May be, I would do more research on
this.

b) clusters in Import has been changed to use singleton

c) addClusters has a method public Builder
addClusters(org.apache.hadoop.hbase.protobuf.generated.HBaseProtos.UUID value)
which takes the UUID as the parameter.

d) Yes, this is used only to read the older log entries when migrating from
0.94 to 0.95.2.

Infinite loop possible in Master/Master replication
---

Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch,
0.95-trunk-rev2.patch, HBASE-7709.patch, HBASE-7709-rev1.patch,
HBASE-7709-rev2.patch, HBASE-7709-rev3.patch

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-21 Thread Himanshu Vashishtha (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746818#comment-13746818
]

Himanshu Vashishtha commented on HBASE-7709:

bq. + repeated UUID clusters = 8;
/*
- optional CustomEntryType custom_entry_type = 8;
-
+ optional CustomEntryType custom_entry_type = 9;

This re-ordering good because 0.96.0 is not released yet?

I think we should have the flexibility to read older edits as a clean shutdown
is a stringent requirement (especially for larger clusters). Also when
replication is enabled, there may be some old logs left to replicate.

Infinite loop possible in Master/Master replication
---

Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch,
0.95-trunk-rev2.patch, HBASE-7709.patch, HBASE-7709-rev1.patch,
HBASE-7709-rev2.patch, HBASE-7709-rev3.patch

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-21 Thread Vasu Mariyala (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746838#comment-13746838
]

Vasu Mariyala commented on HBASE-7709:
--

[~v.himanshu] There is no re-ordering done with the patch. The entry
custom_entry_type is and was a commented one. I changed the number to 9 just
incase if some one un-comments it in the future. Please let me know if I miss
anything

Infinite loop possible in Master/Master replication
---

Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch,
0.95-trunk-rev2.patch, HBASE-7709.patch, HBASE-7709-rev1.patch,
HBASE-7709-rev2.patch, HBASE-7709-rev3.patch

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-20 Thread Vasu Mariyala (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13745573#comment-13745573
]

Vasu Mariyala commented on HBASE-7709:
--

0.95-trunk-rev1.patch contains the javadoc fix and is the latest patch for the
trunk and 0.95 branches. HBASE-7709-rev2.patch is the updated patch on the top
of 0.94 which addresses the comments made by [~lhofhansl], [~jdcryans] and
[~jeffreyz]

Infinite loop possible in Master/Master replication
---

Attachments: 095-trunk.patch, 0.95-trunk-rev1.patch,
HBASE-7709.patch, HBASE-7709-rev1.patch, HBASE-7709-rev2.patch

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-16 Thread Vasu Mariyala (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742367#comment-13742367
 ] 

Vasu Mariyala commented on HBASE-7709:
--

I ran all the test cases on my local machine with the trunk patch and they are 
successful. But everytime it is run on jenkins, it throws 

FATAL: Unable to delete script file /tmp/hudson5964600500647866956.sh
hudson.util.IOException2: remote file operation failed: 
/tmp/hudson5964600500647866956.sh at hudson.remoting.Channel@5ce45886:hadoop1
at hudson.FilePath.act(FilePath.java:902)
at hudson.FilePath.act(FilePath.java:879)
at hudson.FilePath.delete(FilePath.java:1288)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:101)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
at hudson.model.Run.execute(Run.java:1597)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:247)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:516)
at hudson.remoting.Request.call(Request.java:129)
at hudson.remoting.Channel.call(Channel.java:714)
at hudson.FilePath.act(FilePath.java:895)
... 13 more
Caused by: java.io.IOException: Unexpected termination of the channel
at 
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at 
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1316)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
at hudson.remoting.Command.readFrom(Command.java:92)
at 
hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:72)
at 
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected 
termination of the channel
hudson.remoting.RequestAbortedException: 
hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected 
termination of the channel
at 
hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
at 
hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
at hudson.remoting.Request.call(Request.java:174)
at hudson.remoting.Channel.call(Channel.java:714)
at 
hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:167)
at com.sun.proxy.$Proxy40.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:925)
at hudson.Launcher$ProcStarter.join(Launcher.java:360)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
at hudson.model.Run.execute(Run.java:1597)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:247)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: 
Unexpected termination of the channel
at hudson.remoting.Request.abort(Request.java:299)
at hudson.remoting.Channel.terminate(Channel.java:774)
at 
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.io.IOException: Unexpected termination of the channel
at 
hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
Caused by: java.io.EOFException
at 
java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1316)

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-16 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742560#comment-13742560
]

Hadoop QA commented on HBASE-7709:
--

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12598500/095-trunk.patch
against trunk revision .

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.

{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.

{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.

{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestAdmin

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/6788//console

This message is automatically generated.

Infinite loop possible in Master/Master replication
---

Attachments: 095-trunk.patch, HBASE-7709.patch,
HBASE-7709-rev1.patch, HBASE-7709-rev2.patch

[jira] [Commented] (HBASE-7709) Infinite loop possible in Master/Master replication

2013-08-16 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-7709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742640#comment-13742640
]

Hadoop QA commented on HBASE-7709:
--

{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12598525/0.95-trunk-rev1.patch
against trunk revision .