subject:"\[jira\] \[Commented\] \(HBASE\-5986\) Clients can see holes in the META table when regions are being split"

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-06-01 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13287236#comment-13287236
 ] 

Hudson commented on HBASE-5986:
---

Integrated in HBase-0.92-security #109 (See 
[https://builds.apache.org/job/HBase-0.92-security/109/])
HBASE-5986 Clients can see holes in the META table when regions are being 
split (Enis) (Revision 1342866)

 Result = SUCCESS
tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.92.2, 0.96.0, 0.94.1

 Attachments: 5986-v2.txt, HBASE-5986-0.92.patch, 
 HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-26 Thread Zhihong Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283936#comment-13283936
]

Zhihong Yu commented on HBASE-5986:
---

I verified that the 3 failed tests for 0.94 passed.

Integrated patch for 0.94 to 0.94 branch.

Clients can see holes in the META table when regions are being split

Key: HBASE-5986
URL: https://issues.apache.org/jira/browse/HBASE-5986
Project: HBase
Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.96.0

Attachments: 5986-v2.txt, HBASE-5986-0.92.patch,
HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch

We found this issue when running large scale ingestion tests for HBASE-5754.
The problem is that the .META. table updates are not atomic while splitting a
region. In SplitTransaction, there is a time lap between the marking the
parent offline, and adding of daughters to the META table. This can result in
clients using MetaScanner, of HTable.getStartEndKeys (used by the
TableInputFormat) missing regions which are made just offline, but the
daughters are not added yet.
This is also related to HBASE-4335.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-26 Thread Zhihong Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283942#comment-13283942
]

Zhihong Yu commented on HBASE-5986:
---

I verified the failed tests for 0.92 passed locally.

Integrated to 0.92 branch.

Thanks for the patches, Enis.

Clients can see holes in the META table when regions are being split

Attachments: 5986-v2.txt, HBASE-5986-0.92.patch,
HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-26 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283947#comment-13283947
 ] 

Hudson commented on HBASE-5986:
---

Integrated in HBase-0.94 #218 (See 
[https://builds.apache.org/job/HBase-0.94/218/])
HBASE-5986 Clients can see holes in the META table when regions are being 
split (Enis) (Revision 1342865)

 Result = FAILURE
tedyu : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.96.0

 Attachments: 5986-v2.txt, HBASE-5986-0.92.patch, 
 HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-26 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283968#comment-13283968
 ] 

Hudson commented on HBASE-5986:
---

Integrated in HBase-0.92 #422 (See 
[https://builds.apache.org/job/HBase-0.92/422/])
HBASE-5986 Clients can see holes in the META table when regions are being 
split (Enis) (Revision 1342866)

 Result = FAILURE
tedyu : 
Files : 
* /hbase/branches/0.92/CHANGES.txt
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* 
/hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.96.0

 Attachments: 5986-v2.txt, HBASE-5986-0.92.patch, 
 HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-26 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283990#comment-13283990
 ] 

Hudson commented on HBASE-5986:
---

Integrated in HBase-0.94-security #32 (See 
[https://builds.apache.org/job/HBase-0.94-security/32/])
HBASE-5986 Clients can see holes in the META table when regions are being 
split (Enis) (Revision 1342865)

 Result = SUCCESS
tedyu : 
Files : 
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* 
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* 
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.96.0

 Attachments: 5986-v2.txt, HBASE-5986-0.92.patch, 
 HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-25 Thread Zhihong Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283708#comment-13283708
 ] 

Zhihong Yu commented on HBASE-5986:
---

@Enis:
Did you have a chance to run the backports through respective test suite ?

Thanks

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.96.0

 Attachments: 5986-v2.txt, HBASE-5986-0.92.patch, 
 HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-25 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283735#comment-13283735
]

Enis Soztutar commented on HBASE-5986:
--

@Ted, I did run TestEndToEndSplitTransaction, but not the whole suite. Let me
do that.

Clients can see holes in the META table when regions are being split

Attachments: 5986-v2.txt, HBASE-5986-0.92.patch,
HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-25 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283831#comment-13283831
 ] 

Hudson commented on HBASE-5986:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #16 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/16/])
HBASE-5986 Clients can see holes in the META table when regions are being 
split (Revision 1342099)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicStatistics.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.96.0

 Attachments: 5986-v2.txt, HBASE-5986-0.92.patch, 
 HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-25 Thread Enis Soztutar (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13283881#comment-13283881
 ] 

Enis Soztutar commented on HBASE-5986:
--

Here are the test results for 0.94: 
{code}
Tests run: 551, Failures: 0, Errors: 0, Skipped: 0
...
Tests run: 932, Failures: 1, Errors: 2, Skipped: 9

Failed tests:   
testShutdownSimpleFixup(org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster):
 expected:1 but was:0

Tests in error: 
  
testDelayedRpcImmediateReturnValue(org.apache.hadoop.hbase.ipc.TestDelayedRpc): 
Call to /127.0.0.1:53586 failed on socket timeout exception: 
java.net.SocketTimeoutException: 1000 millis timeout while waiting for channel 
to be ready for read. ch : java.nio.channels.SocketChannel[connected 
local=/127.0.0.1:53623 remote=/127.0.0.1:53586]
  testLocalHBaseCluster(org.apache.hadoop.hbase.TestLocalHBaseCluster): Master 
not initialized after 200 seconds
{code}

I rerun the tests locally with success, except TestLocalHBaseCluster. But it 
fails on 0.94 HEAD as well for me. 

For 0.92:
{code}

Results :

Failed tests:   
testMultipleResubmits(org.apache.hadoop.hbase.master.TestSplitLogManager)
  testcomputeHDFSBlocksDistribution(org.apache.hadoop.hbase.util.TestFSUtils)

Tests in error:
  testClusterRestart(org.apache.hadoop.hbase.master.TestRestartCluster): 
org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
  
testWholesomeSplit(org.apache.hadoop.hbase.regionserver.TestSplitTransaction): 
Failed delete of 
/homes/hortonde/enis/code/hbase-0.92/target/test-data/af023188-0b23-4f9d-a9bc-a074e94e57f8/org.apache.hadoop.hbase.regionserver.TestSplitTransaction/table/7c59b6677ad46bf3f652a83de1e62bcb
  testRollback(org.apache.hadoop.hbase.regionserver.TestSplitTransaction): 
Target HLog directory already exists: 
/homes/hortonde/enis/code/hbase-0.92/target/test-data/af023188-0b23-4f9d-a9bc-a074e94e57f8/org.apache.hadoop.hbase.regionserver.TestSplitTransaction/logs
  testRollback(org.apache.hadoop.hbase.regionserver.TestSplitTransaction)
  loadTest[0](org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential): test 
timed out after 12 milliseconds
  loadTest[0](org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel): test 
timed out after 12 milliseconds

Tests run: 1135, Failures: 2, Errors: 6, Skipped: 8
{code} 

Also run those failed tests locally with success. It seems we can go ahead with 
0.92 and 0.94 if you don't have any concerns. 

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.96.0

 Attachments: 5986-v2.txt, HBASE-5986-0.92.patch, 
 HBASE-5986-0.94.patch, HBASE-5986-test_v1.patch, HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-23 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282036#comment-13282036
]

Hadoop QA commented on HBASE-5986:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528806/HBASE-5986_v3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 34 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.master.TestAssignmentManager
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.replication.TestMasterReplication

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1972//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1972//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1972//console

This message is automatically generated.

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-23 Thread Zhihong Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282039#comment-13282039
 ] 

Zhihong Yu commented on HBASE-5986:
---

I ran TestAssignmentManager with patch v3 and it passed:
{code}
Running org.apache.hadoop.hbase.master.TestAssignmentManager
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.703 sec
...
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 17.004s
{code}

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch, 
 HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-23 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282079#comment-13282079
 ] 

stack commented on HBASE-5986:
--

Thanks for running tests Ted.

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.96.0

 Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch, 
 HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-23 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282117#comment-13282117
 ] 

Hudson commented on HBASE-5986:
---

Integrated in HBase-TRUNK #2919 (See 
[https://builds.apache.org/job/HBase-TRUNK/2919/])
HBASE-5986 Clients can see holes in the META table when regions are being 
split (Revision 1342099)

 Result = FAILURE
stack : 
Files : 
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicStatistics.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
* 
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java


 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.96.0

 Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch, 
 HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-23 Thread Zhihong Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282138#comment-13282138
]

Zhihong Yu commented on HBASE-5986:
---

This JIRA is marked as bug.
The reason behind not choosing approach #3 is to facilitate backporting to 0.92
and 0.94

I think 0.92.2 and 0.94.1 should carry this fix.

Clients can see holes in the META table when regions are being split

Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch,
HBASE-5986_v3.patch

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-23 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13282144#comment-13282144
 ] 

Lars Hofhansl commented on HBASE-5986:
--

+1 for 0.94, can't speak for 0.92.

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Fix For: 0.96.0

 Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch, 
 HBASE-5986_v3.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-17 Thread Zhihong Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13278142#comment-13278142
 ] 

Zhihong Yu commented on HBASE-5986:
---

@Stack:
What do you think of patch v2 ?

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: 5986-v2.txt, HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-16 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277025#comment-13277025
]

jirapos...@reviews.apache.org commented on HBASE-5986:
--

bq. On 2012-05-16 05:20:29, ramkrishna vasudevan wrote:
bq. src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 402
bq. https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line402
bq.
bq. Blocking time out is one factor on which we will be waiting. So
this blockingtimeout should be in lieu with 'fileSplitTimeout'?

bq. From my understanding, fileSplitTimeout is a regionserver property, while
the timeout parameter in this patch is a client side property, since the
blocking will happen on the client. I think hbase.client.operation.timeout
can serve us well. HConstants state that it is Default HBase client operation
timeout, which is tantamount to a blocking call.

- enis

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7928
---

On 2012-05-16 01:53:09, enis wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/5133/
bq. ---
bq.
bq. (Updated 2012-05-16 01:53:09)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. We found this issue when running large scale ingestion tests for
HBASE-5754. The problem is that the .META. table updates are not atomic while
splitting a region. In SplitTransaction, there is a time lap between the
marking the parent offline, and adding of daughters to the META table. This can
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the
TableInputFormat) missing regions which are made just offline, but the
daughters are not added yet.
bq.
bq. This patch is the approach 2 mentioned in the issue comments, mainly
during META scan, if we detect that the region is split, we block until the
information for the child regions are available in META and manually feed those
rows to the MetaScanner. Although approach 3 (using local region transactions)
seems cleaner, they are not available under branch 0.92, which I think should
also incorporate this fix. I'll provide ports once we are clear for trunk.
bq.
bq. Also this patch does not fix MetaReader (see
https://issues.apache.org/jira/browse/HBASE-3475).
bq.
bq.
bq. This addresses bug HBASE-5986.
bq. https://issues.apache.org/jira/browse/HBASE-5986
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
a8091e6
bq.
bq. Diff: https://reviews.apache.org/r/5133/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. added extensive tests under TestEndToEndSplitTranscation, and ran existing
unit tests.
bq.
bq.
bq. Thanks,
bq.
bq. enis
bq.
bq.

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-16 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277024#comment-13277024
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--



bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 391
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line391
bq.  
bq.   null is always returned by getRegionResultBlocking() in case of time 
out.
bq.   How do we deal with that ?
bq.  

we are checking for null return value for resultA, and resultB. But I'll also 
add a LOG warning. 


bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 390
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line390
bq.  
bq.   Do we have to instantiate HTable every time ?

This is a bit tricky. Ideally we should not. But there is no close() on 
MetaScannerVisitor, so we cannot close the HTable properly if we reuse the 
HTable across calls to processRow(). We can add a close() method and call it, 
or obtain the HTable from the context, but that would imply changing the class 
signature for MetaScannerVisitor. I assumed since we are reusing HConnection's 
HTable creation is cheap, is that not the case, wdyt? 


bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 360
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line360
bq.  
bq.   Can we add a config param for blockingTimeout ?

please see below


bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 398
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line398
bq.  
bq.   We shouldn't be passing blockingTimeout here.
bq.   We need to consider the amount of time spent in the call @ line 388.

agreed


bq.  On 2012-05-16 02:52:04, Ted Yu wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 430
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line430
bq.  
bq.   Please restore interrupted state of the thread.

agreed


- enis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7925
---


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.  Summary
bq.  ---
bq.  
bq.  We found this issue when running large scale ingestion tests for 
HBASE-5754. The problem is that the .META. table updates are not atomic while 
splitting a region. In SplitTransaction, there is a time lap between the 
marking the parent offline, and adding of daughters to the META table. This can 
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the 
TableInputFormat) missing regions which are made just offline, but the 
daughters are not added yet.
bq.  
bq.  This patch is the approach 2 mentioned in the issue comments, mainly 
during META scan, if we detect that the region is split, we block until the 
information for the child regions are available in META and manually feed those 
rows to the MetaScanner. Although approach 3 (using local region transactions) 
seems cleaner, they are not available under branch 0.92, which I think should 
also incorporate this fix. I'll provide ports once we are clear for trunk. 
bq.  
bq.  Also this patch does not fix MetaReader (see 
https://issues.apache.org/jira/browse/HBASE-3475). 
bq.  
bq.  
bq.  This addresses bug HBASE-5986.
bq.  https://issues.apache.org/jira/browse/HBASE-5986
bq.  
bq.  
bq.  Diffs
bq.  -
bq.  
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512 
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4 
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999 
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 a8091e6 
bq.  
bq.  Diff: https://reviews.apache.org/r/5133/diff
bq.  
bq.  
bq.  Testing
bq.  ---
bq.  
bq.  added extensive tests under TestEndToEndSplitTranscation, and ran existing 
unit tests. 
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  enis
bq.  
bq.



 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL:

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-16 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277031#comment-13277031
]

jirapos...@reviews.apache.org commented on HBASE-5986:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7938
---

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17261

I was asking how do we convey the fact that MetaScanner has timed out to
client.

I don't see exception being thrown which contains timeout information.

- Ted

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-16 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277035#comment-13277035
 ] 

stack commented on HBASE-5986:
--

Regards 1. and 2., what guarantees do we have that the daughter will not have 
split by the time we go into our wait on the daughters to come online?

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-16 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277161#comment-13277161
]

jirapos...@reviews.apache.org commented on HBASE-5986:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7939
---

Looks great Enis. Thanks for having a go at this. Loads of white space
introduced which you might fix in v2. Otherwise a few little items below

src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
https://reviews.apache.org/r/5133/#comment17262

Bit of javadoc to say this is best-effort.

Also, does this belong in MetaReader (Won't hold you to it... these two
classes, a MetaReader vs MetaEditor are kinda silly... this whole catalog
package needs killing).

src/main/java/org/apache/hadoop/hbase/client/HTable.java
https://reviews.apache.org/r/5133/#comment17264

Why you remove this? We don't return these any more? Offline I think is
'dead', unused now. Split not.

src/main/java/org/apache/hadoop/hbase/client/HTable.java
https://reviews.apache.org/r/5133/#comment17265

This seems like a good refactoring.

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17266

Good

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17267

Should this be public? Should it remain internal to the HTable hidden?

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17268

Yeah, it could be a long time before a region comes on line if its in a GC
or some such.

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17269

Yeah, what Ted says... Can you close when done? See MetaEditor/MetaReader.
They do this a bunch. Closing means for sure the zk and connection resources
will be cleaned up afterward. Its reference counting so keepign around an
HTable could mess it up.

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17270

Yeah, so, what happens if daughter has split by the time I get here?

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17271

So if interrupted or we don't find it by the time the blocking time has
passed, we just return null? What you reckon? We should at least complain?

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17272

Does the scan of meta start at first table region?

- Michael

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-16 Thread jirapos...@reviews.apache.org (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277236#comment-13277236
 ] 

jirapos...@reviews.apache.org commented on HBASE-5986:
--



bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 473
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109116#file109116line473
bq.  
bq.   Why you remove this?  We don't return these any more?  Offline I 
think is 'dead', unused now.  Split not.

If we discover that a region has been split in META, than, it is past 
point-of-no-return and the region cannot be seen un-split anymore, even though 
concurrent rs failures.  for getStartEndKeys() we are returning whatever the 
getRegionLocation() provides. getRegionLocations() ignores offline regions, but 
returns daughter regions for split-parents (which are offline as well).


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 351
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line351
bq.  
bq.   Should this be public?  Should it remain internal to the HTable 
hidden?

MetaScanner is confusing in its visibility. It's javadoc states Although 
public visibility, this is not a public-facing API and may evolve in minor 
releases., but it is annotated with @InterfaceAudience.Public 
@InterfaceStability.Evolving. I think for BlockingMetaScannerVisitor, we should 
set the visibility the same as MetaScannerVisitor. I think, MetaScannerVisitor 
itself is of little use without the BlockingMetaScannerVisitor functionality 
due to this issue. 
Shall we change MetaScanner, MSV and BMSV to be @InterfaceAudience.Private, 
wdyt? 


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java, line 310
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109115#file109115line310
bq.  
bq.   Bit of javadoc to say this is best-effort.
bq.   
bq.   Also, does this belong in MetaReader (Won't hold you to it... these 
two classes, a MetaReader vs MetaEditor are kinda silly... this whole catalog 
package needs killing).

agreed


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 395
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line395
bq.  
bq.   Yeah, so, what happens if daughter has split by the time I get here?

The scanner provides the regions results in sorted order, and since the 
daughters are sorted after the parent, we always process parent first. When we 
process the daughter regions manually (processRow(resultA)), we also add them 
to the daughterRegions set. If the scanner also sees them, they are just 
skipped (line 373)


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 434
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line434
bq.  
bq.   So if interrupted or we don't find it by the time the blocking time 
has passed, we just return null?   What you reckon?  We should at least 
complain?

yeah, I think we can throw RegionOfflineException upon timeout and interrupt. 
The operation can be retried by the client upon timeout.


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 465
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line465
bq.  
bq.   Does the scan of meta start at first table region?

MetaScanner.metaScan() ensures that the scan starts at the first table region. 


bq.  On 2012-05-16 21:32:56, Michael Stack wrote:
bq.   src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java, line 390
bq.   https://reviews.apache.org/r/5133/diff/1/?file=109117#file109117line390
bq.  
bq.   Yeah, what Ted says... Can you close when done?  See 
MetaEditor/MetaReader.  They do this a bunch.  Closing means for sure the zk 
and connection resources will be cleaned up afterward.  Its reference counting 
so keepign around an HTable could mess it up.

We are already closing the HTable's. But let me see how we can best reuse 
Htable instances. 


- enis


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7939
---


On 2012-05-16 01:53:09, enis wrote:
bq.  
bq.  ---
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/5133/
bq.  ---
bq.  
bq.  (Updated 2012-05-16 01:53:09)
bq.  
bq.  
bq.  Review request for hbase.
bq.  
bq.  
bq.

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-16 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277306#comment-13277306
]

jirapos...@reviews.apache.org commented on HBASE-5986:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/
---

(Updated 2012-05-17 00:58:21.965488)

Review request for hbase.

Changes
---

v2:
- addressed review comments
- when timeout and interrupt throws RegionOfflineException.
- reuse HTable instances in BMSV.
- adds close() to MSV assuming it is indeed not public, evolving but private.
we should decide the visibility.

Summary
---

We found this issue when running large scale ingestion tests for HBASE-5754.
The problem is that the .META. table updates are not atomic while splitting a
region. In SplitTransaction, there is a time lap between the marking the parent
offline, and adding of daughters to the META table. This can result in clients
using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat)
missing regions which are made just offline, but the daughters are not added
yet.

This patch is the approach 2 mentioned in the issue comments, mainly during
META scan, if we detect that the region is split, we block until the
information for the child regions are available in META and manually feed those
rows to the MetaScanner. Although approach 3 (using local region transactions)
seems cleaner, they are not available under branch 0.92, which I think should
also incorporate this fix. I'll provide ports once we are clear for trunk.

Also this patch does not fix MetaReader (see
https://issues.apache.org/jira/browse/HBASE-3475).

This addresses bug HBASE-5986.
https://issues.apache.org/jira/browse/HBASE-5986

Diffs (updated)
-

src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512
src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4
src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999
src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2

src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
a8091e6

Diff: https://reviews.apache.org/r/5133/diff

Testing
---

added extensive tests under TestEndToEndSplitTranscation, and ran existing unit
tests.

Thanks,

enis

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-16 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277494#comment-13277494
]

jirapos...@reviews.apache.org commented on HBASE-5986:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7948
---

v2 looks much better.

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17287

We may need to mark this JIRA an incompatible change.

src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
https://reviews.apache.org/r/5133/#comment17283

'with' - 'by'

src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
https://reviews.apache.org/r/5133/#comment17284

This class can be private.

src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
https://reviews.apache.org/r/5133/#comment17285

The interval can be made shorter.

src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
https://reviews.apache.org/r/5133/#comment17286

Name this method verifyRegionsOfHTable ?

- Ted

On 2012-05-17 00:58:21, enis wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/5133/
bq. ---
bq.
bq. (Updated 2012-05-17 00:58:21)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. We found this issue when running large scale ingestion tests for
HBASE-5754. The problem is that the .META. table updates are not atomic while
splitting a region. In SplitTransaction, there is a time lap between the
marking the parent offline, and adding of daughters to the META table. This can
result in clients using MetaScanner, of HTable.getStartEndKeys (used by the
TableInputFormat) missing regions which are made just offline, but the
daughters are not added yet.
bq.
bq. This patch is the approach 2 mentioned in the issue comments, mainly
during META scan, if we detect that the region is split, we block until the
information for the child regions are available in META and manually feed those
rows to the MetaScanner. Although approach 3 (using local region transactions)
seems cleaner, they are not available under branch 0.92, which I think should
also incorporate this fix. I'll provide ports once we are clear for trunk.
bq.
bq. Also this patch does not fix MetaReader (see
https://issues.apache.org/jira/browse/HBASE-3475).
bq.
bq.
bq. This addresses bug HBASE-5986.
bq. https://issues.apache.org/jira/browse/HBASE-5986
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512
bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 5d4be3f
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
5cac9af
bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4
bq.src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 0ad9b18
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
bq.
src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
a8091e6
bq.
bq. Diff: https://reviews.apache.org/r/5133/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. added extensive tests under TestEndToEndSplitTranscation, and ran existing
unit tests.
bq.
bq.
bq. Thanks,
bq.
bq. enis
bq.
bq.

Clients can see holes in the META table when regions are being split

--
This message is automatically generated by JIRA.
If you think it was

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-16 Thread Hadoop QA (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13277516#comment-13277516
]

Hadoop QA commented on HBASE-5986:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12527760/5986-v2.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

-1 findbugs. The patch appears to introduce 32 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/1903//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/1903//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/1903//console

This message is automatically generated.

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-15 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276385#comment-13276385
]

jirapos...@reviews.apache.org commented on HBASE-5986:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/
---

Review request for hbase.

Summary
---

We found this issue when running large scale ingestion tests for HBASE-5754.
The problem is that the .META. table updates are not atomic while splitting a
region. In SplitTransaction, there is a time lap between the marking the parent
offline, and adding of daughters to the META table. This can result in clients
using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat)
missing regions which are made just offline, but the daughters are not added
yet.

This addresses bug HBASE-5986.
https://issues.apache.org/jira/browse/HBASE-5986

Diffs
-

src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java 8873512
src/main/java/org/apache/hadoop/hbase/client/HTable.java b8290e4
src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java f404999

src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
a8091e6

Diff: https://reviews.apache.org/r/5133/diff

Testing
---

added extensive tests under TestEndToEndSplitTranscation, and ran existing unit
tests.

Thanks,

enis

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-15 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276386#comment-13276386
]

jirapos...@reviews.apache.org commented on HBASE-5986:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/
---

(Updated 2012-05-16 01:53:09.861265)

Review request for hbase.

Summary (updated)
---

We found this issue when running large scale ingestion tests for HBASE-5754.
The problem is that the .META. table updates are not atomic while splitting a
region. In SplitTransaction, there is a time lap between the marking the parent
offline, and adding of daughters to the META table. This can result in clients
using MetaScanner, of HTable.getStartEndKeys (used by the TableInputFormat)
missing regions which are made just offline, but the daughters are not added
yet.

Also this patch does not fix MetaReader (see
https://issues.apache.org/jira/browse/HBASE-3475).

This addresses bug HBASE-5986.
https://issues.apache.org/jira/browse/HBASE-5986

Diffs
-

src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
a8091e6

Diff: https://reviews.apache.org/r/5133/diff

Testing
---

added extensive tests under TestEndToEndSplitTranscation, and ran existing unit
tests.

Thanks,

enis

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-15 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276399#comment-13276399
]

jirapos...@reviews.apache.org commented on HBASE-5986:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7925
---

src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java
https://reviews.apache.org/r/5133/#comment17245

Remove white space.

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17246

Can we add a config param for blockingTimeout ?

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17247

Do we have to instantiate HTable every time ?

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17250

null is always returned by getRegionResultBlocking() in case of time out.
How do we deal with that ?

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17249

We shouldn't be passing blockingTimeout here.
We need to consider the amount of time spent in the call @ line 388.

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17248

Please restore interrupted state of the thread.

- Ted

Clients can see holes in the META table when regions are being split

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-15 Thread jirapos...@reviews.apache.org (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13276477#comment-13276477
]

jirapos...@reviews.apache.org commented on HBASE-5986:
--

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5133/#review7928
---

src/main/java/org/apache/hadoop/hbase/client/MetaScanner.java
https://reviews.apache.org/r/5133/#comment17254

Blocking time out is one factor on which we will be waiting. So this
blockingtimeout should be in lieu with 'fileSplitTimeout'?

- ramkrishna

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-10 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272918#comment-13272918
]

Enis Soztutar commented on HBASE-5986:
--

Possible fixes I can think of:
1. Keep MetaScanner/MetaReader as non-consistent (as it is), but allow for a
consistent view for getting table regions. Since single row puts are atomic,
when the parent region is mutated to be offline, the HRI for daughters are
added to the row. So on MetaScanner.allTableRegions and similar calls, we can
keep track of daughter regions from split parents and return them to the
client.
2. Make MetaScanner consistent, in that, whenever it sees a split parent, it
blocks until the daughters are available.
3. We have region-local transactions now, so if we ensure that the rows for
parent and daughters will be served from the same META region, then we can
update all three rows atomically. Maybe we can come up with a META-specific
split policy to ensure split-regions go to the same META region.
Thoughts?

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-10 Thread Lars Hofhansl (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272948#comment-13272948
]

Lars Hofhansl commented on HBASE-5986:
--

I think we are assuming in many other places that META only has a single region.

Is there another alternative, such as adding the daughter regions first, and
then have HTable disentangle conflicts?

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-10 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272952#comment-13272952
]

Enis Soztutar commented on HBASE-5986:
--

bq. I think we are assuming in many other places that META only has a single
region.
The ROOT is -by-design- one region, but META is not, right?

bq. Is there another alternative, such as adding the daughter regions first,
and then have HTable disentangle conflicts?
I have thought about this as well, but then there is a time window in which you
have both the parent and daughter regions online, and parent not marked as
split. So the client again has to resolve that the returned regions are
overlapping.

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-10 Thread Lars Hofhansl (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272966#comment-13272966
]

Lars Hofhansl commented on HBASE-5986:
--

True, META is not single region by design, but effectively it is (not really
suggesting that we add more code that relies on this). I think you're right we
could add a split policy that ensures that parent and daughters are always in
the same region.

As for the alternative:
If the client has to recheck in these (rare) cases that seems OK, as long as
the client can reliably tell that it does have to recheck.

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-10 Thread Zhihong Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272974#comment-13272974
 ] 

Zhihong Yu commented on HBASE-5986:
---

Can MetaScanner.allTableRegions() return special exception so that client knows 
it needs to check back ?

 Clients can see holes in the META table when regions are being split
 

 Key: HBASE-5986
 URL: https://issues.apache.org/jira/browse/HBASE-5986
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.92.1, 0.96.0, 0.94.1
Reporter: Enis Soztutar
Assignee: Enis Soztutar
 Attachments: HBASE-5986-test_v1.patch


 We found this issue when running large scale ingestion tests for HBASE-5754. 
 The problem is that the .META. table updates are not atomic while splitting a 
 region. In SplitTransaction, there is a time lap between the marking the 
 parent offline, and adding of daughters to the META table. This can result in 
 clients using MetaScanner, of HTable.getStartEndKeys (used by the 
 TableInputFormat) missing regions which are made just offline, but the 
 daughters are not added yet. 
 This is also related to HBASE-4335. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-10 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13272987#comment-13272987
]

Enis Soztutar commented on HBASE-5986:
--

I have implemented approach 1 by adding split daughters to the returned map
from MetaScanner.allTableRegions(). But then the problem is that, we are
returning regions which does not yet exists in the META table, so any
subsequent getRegion call will fail.

Thinking a bit more about 3, I think we already guarantee that the region split
parent, and daughters fall into the same META region. Let's say we have two
regions region1 and region2, with start keys start_key*, and timestamps ts*
respectively.

Before split:
{code}
table start_key1 ts1 encoded_name1
table start_key2 ts2 encoded_name2
{code}

Now, if we split region1, daughters will be sorted after region1, and before
region2:
{code}
table start_key1 ts1 encoded_name1 offline split
table start_key1 ts3 encoded_name1
table mid_key1 ts3 encoded_name1
table start_key2 ts2 encoded_name2
{code}

we know this since we have the invariants ts3 ts1
(SplitTransaction.getDaughterRegionIdTimestamp()) and start_key1 mid_key1
start_key2. Even if we have a region boundary between start_key1 and start_key2
in the META table, the daughters will be co-located with the parent. The only
exception is that while the user table is split, we have a concurrent split for
the META table, and the new region boundary is chosen to be between the parent
and daughters. With some effort, we can prevent this, but it seems to be very
highly unlikely.

So, if my analysis is correct, that means option 3 seems like the best choice,
since this will not complicate the meta scan code. The problem is that, there
is no internal API to do multi-row transcations other than using the
coprocessor. Should we think of allowing that w/o coprocessors?

@Lars, does HRegion.mutateRowsWithLock() guarantee that a concurrent scanner
won't see partial changes?

Clients can see holes in the META table when regions are being split

[jira] [Commented] (HBASE-5986) Clients can see holes in the META table when regions are being split

2012-05-10 Thread Lars Hofhansl (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13273045#comment-13273045
]

Lars Hofhansl commented on HBASE-5986:
--

@Enis: yes HRegion.mutateRowsWithLock() has correct MVCC semantics even across
multi row scans.

The region boundary could be between start_key1 and mid_key1 if there rows
inserted and deleted before... Still unlikely. To be safe we could just have a
splitKeyPolicy that never splits the table prefix (i.e. all keys for the same
table are guaranteed to be in the region).

Clients can see holes in the META table when regions are being split

37 matches

Mail list logo