[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush
[ https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139548#comment-13139548 ] Hadoop QA commented on HBASE-4695: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501494/hbase-4695-0.92.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/103//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/103//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/103//console This message is automatically generated. WAL logs get deleted before region server can fully flush - Key: HBASE-4695 URL: https://issues.apache.org/jira/browse/HBASE-4695 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.90.4 Reporter: jack levin Assignee: gaojinchao Priority: Blocker Fix For: 0.90.5 Attachments: HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt To replicate the problem do the following: 1. check /hbase/.logs/ directory to see if you have WAL logs for the region server you are shutting down. 2. executing kill pid (where pid is a regionserver pid) 3. Watch the regionserver log to start flushing, you will see how many regions are left to flush: 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 489 regions to close 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 116 regions to close 4. Check /hbase/.logs/ -- you will notice that it has dissapeared. 5. Check namenode logs: 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=root ip=/10.101.1.5 cmd=delete src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749 Note that, if you kill -9 the RS now, and it crashes on flush, you won't have any WAL logs to replay. We need to make sure that logs are deleted or moved out only when RS has fully flushed. Otherwise its possible to lose data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Attachment: 4583.txt Here a *sketch* of a patch. May need some clean up but should work. Basically the KVs of Append and Increment are applied to memstore through a call to the standard applyFamilyMapToMemstore(...); then duplicates are removed after the rwcc is moved forward. Still a lot of boilerplate shared between Append and Increment. I think the dup removal does not need to hold the regions updatesLock.readLock (just the lock.readLock), but I am not entirely sure. Comments/suggestions/flames/praise are welcome. :) Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139551#comment-13139551 ] Lars Hofhansl commented on HBASE-4583: -- Also changes some calls to take ? extends Collection rather than List, so that I can use guava's MultiMap. If all callers would be changed the ? extends could be removed. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush
[ https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139553#comment-13139553 ] gaojinchao commented on HBASE-4695: --- @Ted Thanks for your reveiw. I considered if this.fsOk is false , calling closeWAL() makes no sense. Your approach is same as original logic, that is ok, I will verify next monday. WAL logs get deleted before region server can fully flush - Key: HBASE-4695 URL: https://issues.apache.org/jira/browse/HBASE-4695 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.90.4 Reporter: jack levin Assignee: gaojinchao Priority: Blocker Fix For: 0.90.5 Attachments: HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt To replicate the problem do the following: 1. check /hbase/.logs/ directory to see if you have WAL logs for the region server you are shutting down. 2. executing kill pid (where pid is a regionserver pid) 3. Watch the regionserver log to start flushing, you will see how many regions are left to flush: 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 489 regions to close 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 116 regions to close 4. Check /hbase/.logs/ -- you will notice that it has dissapeared. 5. Check namenode logs: 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=root ip=/10.101.1.5 cmd=delete src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749 Note that, if you kill -9 the RS now, and it crashes on flush, you won't have any WAL logs to replay. We need to make sure that logs are deleted or moved out only when RS has fully flushed. Otherwise its possible to lose data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139551#comment-13139551 ] Lars Hofhansl edited comment on HBASE-4583 at 10/30/11 7:08 AM: Also changes some calls to take ? extends Collection rather than List, so that I can use guava's MultiMap. If all callers would be changed the ? extends could be removed. was (Author: lhofhansl): Also changes some calls to take ? extends Collection rather than List, so that I can use guava's MultiMap. If all callers would be changed the ? extends could be removed. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139550#comment-13139550 ] Lars Hofhansl edited comment on HBASE-4583 at 10/30/11 7:06 AM: Here's a *sketch* of a patch. May need some clean up but should work. Basically the KVs of Append and Increment are applied to memstore through a call to the standard applyFamilyMapToMemstore(...); then duplicates are removed after the rwcc is moved forward. Still a lot of boilerplate shared between Append and Increment. I think the dup removal does not need to hold the regions updatesLock.readLock (just the lock.readLock), but I am not entirely sure. Comments/suggestions/flames/praise are welcome. :) was (Author: lhofhansl): Here a *sketch* of a patch. May need some clean up but should work. Basically the KVs of Append and Increment are applied to memstore through a call to the standard applyFamilyMapToMemstore(...); then duplicates are removed after the rwcc is moved forward. Still a lot of boilerplate shared between Append and Increment. I think the dup removal does not need to hold the regions updatesLock.readLock (just the lock.readLock), but I am not entirely sure. Comments/suggestions/flames/praise are welcome. :) Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure
[ https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139554#comment-13139554 ] Hadoop QA commented on HBASE-4690: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501496/4690-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/104//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/104//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/104//console This message is automatically generated. Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure Key: HBASE-4690 URL: https://issues.apache.org/jira/browse/HBASE-4690 Project: HBase Issue Type: Test Affects Versions: 0.92.0 Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4690-trunk.txt See https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/ Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region server. One fix for this issue is to spin up MiniCluster with 1 region server so that we don't need to search for the region server where first region is hosted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4696) HRegionThriftServer' might have to indefinitely do redirtects
[ https://issues.apache.org/jira/browse/HBASE-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139564#comment-13139564 ] dhruba borthakur commented on HBASE-4696: - +1 code looks good to me. HRegionThriftServer' might have to indefinitely do redirtects - Key: HBASE-4696 URL: https://issues.apache.org/jira/browse/HBASE-4696 Project: HBase Issue Type: Improvement Affects Versions: 0.94.0 Reporter: Prakash Khemani Assignee: Jonathan Gray Fix For: 0.94.0 Attachments: HBASE-4696-v1.patch HRegionThriftServer.getRowWithColumnsTs() redirects the request to the correct region server if it has landed on the wrong region-server. With this approach the smart-client will never get a NotServingRegionException and it will never be able to invalidate its cache. It will indefinitely send the request to the wrong region server and the wrong region server will always be redirecting it. Either redirects should be turned off and the client should react to NotServingRegionExceptions. Or somehow a flag should be set in the response telling the client to refresh its cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure
[ https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139627#comment-13139627 ] Ted Yu commented on HBASE-4690: --- Except for TestAdmin#testEnableDisableAddColumnDeleteColumn, the other test failures were due to 'Too many open files' I ran TestAdmin#testEnableDisableAddColumnDeleteColumn a few times and didn't encounter test failure. Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure Key: HBASE-4690 URL: https://issues.apache.org/jira/browse/HBASE-4690 Project: HBase Issue Type: Test Affects Versions: 0.92.0 Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4690-trunk.txt See https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/ Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region server. One fix for this issue is to spin up MiniCluster with 1 region server so that we don't need to search for the region server where first region is hosted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4695) WAL logs get deleted before region server can fully flush
[ https://issues.apache.org/jira/browse/HBASE-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139630#comment-13139630 ] Ted Yu commented on HBASE-4695: --- The test failures reported by HadoopQA were due to 'Too many open files' WAL logs get deleted before region server can fully flush - Key: HBASE-4695 URL: https://issues.apache.org/jira/browse/HBASE-4695 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.90.4 Reporter: jack levin Assignee: gaojinchao Priority: Blocker Fix For: 0.90.5 Attachments: HBASE-4695_branch90_trial.patch, hbase-4695-0.92.txt To replicate the problem do the following: 1. check /hbase/.logs/ directory to see if you have WAL logs for the region server you are shutting down. 2. executing kill pid (where pid is a regionserver pid) 3. Watch the regionserver log to start flushing, you will see how many regions are left to flush: 09:36:54,665 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 489 regions to close 09:56:35,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting on 116 regions to close 4. Check /hbase/.logs/ -- you will notice that it has dissapeared. 5. Check namenode logs: 09:26:41,607 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=root ip=/10.101.1.5 cmd=delete src=/hbase/.logs/rdaa5.prod.imageshack.com,60020,1319749 Note that, if you kill -9 the RS now, and it crashes on flush, you won't have any WAL logs to replay. We need to make sure that logs are deleted or moved out only when RS has fully flushed. Otherwise its possible to lose data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[
https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139641#comment-13139641
]
Ted Yu commented on HBASE-4583:
---
Publishing on reviewboard would make reviewing much easier.
Patch looks good overall.
A few minor comments.
newFamilyMap.asMap() is called more than once in append() and increment(). We
can use a variable to hold its return and reuse the variable in place of the
second call.
Line 3948 can be omitted:
{code}
flush = isFlushSize(size);
{code}
because we have the same call at line 3964.
Same argument can be made for the call at line 3824.
In MemStore.java:
{code}
* For each KeyValue if the keyValue did already exist, with a
{code}
Should read 'if the KeyValue does already ...'
This comment in Store.java can be refined:
{code}
* qualifier exists in MemStore with a memstoreTS the passed KV, it will be
removed.
{code}
because we only pass keyvalues to this.memstore.removeDups().
Integrate RWCC with Append and Increment operations
---
Key: HBASE-4583
URL: https://issues.apache.org/jira/browse/HBASE-4583
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.94.0
Attachments: 4583.txt
Currently Increment and Append operations do not work with RWCC and hence a
client could see the results of multiple such operation mixed in the same
Get/Scan.
The semantics might be a bit more interesting here as upsert adds and removes
to and from the memstore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4519) 25s sleep when expiring sessions in tests
[
https://issues.apache.org/jira/browse/HBASE-4519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139643#comment-13139643
]
nkeywal commented on HBASE-4519:
TestZooKeeper#testClientSessionExpired() has a very similar piece of code, with
a 15s timeout 'only'.
But when i tried this value, I had a failure in TestFromClientSide... to be
continued
25s sleep when expiring sessions in tests
-
Key: HBASE-4519
URL: https://issues.apache.org/jira/browse/HBASE-4519
Project: HBase
Issue Type: Improvement
Affects Versions: 0.90.4
Reporter: Jean-Daniel Cryans
Assignee: nkeywal
Fix For: 0.92.0
There's a hardcoded 25 seconds sleep in HBaseTestingUtility.expireSession:
{code}
int sessionTimeout = 5 * 1000; // 5 seconds
...
final long sleep = sessionTimeout * 5L;
LOG.info(ZK Closed Session 0x + Long.toHexString(sessionID) +
; sleeping= + sleep);
Thread.sleep(sleep);
{code}
I'm pretty sure this can be lowered at lot, and it would speed up a couple of
tests. The only thing I'm afraid of is if this was made to accomodate flaky
tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139660#comment-13139660 ] Ted Yu commented on HBASE-4583: --- bq. Still a lot of boilerplate shared between Append and Increment. I think we should take this opportunity to reduce duplicate code. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4703) Improvements in tests
Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - tmp dir management: to allow multiple tests using dir to run on the same machine without conflicting, the tmp dir is set once per JVM. This should fix HBASE-3672 as well, because the map reduce cluster is configured to use the hadoop tmp dir - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4703) Improvements in tests
[ https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4703: --- Description: Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility was: Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - tmp dir management: to allow multiple tests using dir to run on the same machine without conflicting, the tmp dir is set once per JVM. This should fix HBASE-3672 as well, because the map reduce cluster is configured to use the hadoop tmp dir - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable
[jira] [Updated] (HBASE-4703) Improvements in tests
[ https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4703: --- Attachment: 20111030_4703_all.patch Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111030_4703_all.patch Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139709#comment-13139709
]
Jesse Yates commented on HBASE-4605:
@Ted: ok, fair enough. I'm just trying to work through the possibilities and
get feedback (thanks btw!).
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139721#comment-13139721 ] Lars Hofhansl commented on HBASE-4583: -- The reason why I am calling isFlushSize again is because the duplicate removal might fail and that should not impact the operation. So I call it first in the required part and then again in the part that might fail. If the 2nd part fails, the first value should be used. Agree on the boiler plate removal. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4703) Improvements in tests
[ https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139730#comment-13139730 ] Ted Yu commented on HBASE-4703: --- Currently patch testing uses -p0 to apply patches. From Todd: Try git show --no-prefix or git diff --no-prefix instead. That way it will apply with patch -p0 and should work in test-patch Also, the observations in this JIRA should be broadcast and receive wider discussion so that we can establish well-accepted guidelines. Otherwise the issues corrected by this effort would get back into hbase codebase. Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111030_4703_all.patch Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139734#comment-13139734 ] Ted Yu commented on HBASE-4583: --- @Lars: removeDupsInMemstore() isn't declared to throw exception. So I didn't expect failure in that part of the code. If runtime exception comes out of it, would requestFlush() still be executed ? Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4703) Improvements in tests
[ https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-4703: --- Attachment: 20111030_4703_all.v2.patch taking into account ted's comment (with --no-prefix). Ok for your other comments, I will write a general mail. Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[Differential] [Updated, 52 lines] D111: [jira] [HBASE-4698] Let the HFile Pretty Printer print all the key values for a specific row.
Liyin updated the revision [jira] [HBASE-4698] Let the HFile Pretty Printer
print all the key values for a specific row..
Reviewers: mbautin, Kannan, jgray, gqchen, nspiegelberg, JIRA
Address Mikhail's comments.
REVISION DETAIL
https://reviews.facebook.net/D111
AFFECTED FILES
src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
Index: src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
===
--- src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
+++ src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java
@@ -41,7 +41,6 @@
import org.apache.hadoop.hbase.HRegionInfo;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.io.hfile.HFile.FileInfo;
-import org.apache.hadoop.hbase.io.hfile.HFileBlockIndex.BlockIndexReader;
import org.apache.hadoop.hbase.regionserver.TimeRangeTracker;
import org.apache.hadoop.hbase.util.BloomFilter;
import org.apache.hadoop.hbase.util.BloomFilterFactory;
@@ -68,7 +67,9 @@
private boolean printBlocks;
private boolean checkRow;
private boolean checkFamily;
-
+ private boolean isSeekToRow = false;
+ // the row that the user want to specify and only print kv for.
+ private byte[] row = null;
private Configuration conf;
private ListPath files = new ArrayListPath();
@@ -92,6 +93,8 @@
options.addOption(a, checkfamily, false, Enable family check);
options.addOption(f, file, true,
File to scan. Pass full-path; e.g. hdfs://a:9000/hbase/.META./12/34);
+options.addOption(s, seekToRow, true,
+Seek to this row and print all the kvs for this row only);
options.addOption(r, region, true,
Region to scan. Pass region name; e.g. '.META.,,1');
}
@@ -125,6 +128,19 @@
files.add(new Path(cmd.getOptionValue(f)));
}
+if (cmd.hasOption(s)) {
+ String key = cmd.getOptionValue(s);
+ if (key!= null key.length() != 0) {
+row = key.getBytes();
+isSeekToRow = true;
+System.out.println(Only print the kv for the row: +
+ Bytes.toString(row));
+ } else {
+System.err.println(Invalid row is specified.);
+System.exit(-1);
+ }
+}
+
if (cmd.hasOption(r)) {
String regionName = cmd.getOptionValue(r);
byte[] rn = Bytes.toBytes(regionName);
@@ -203,8 +219,13 @@
if (printKey || checkRow || checkFamily) {
// scan over file and read key/value's, performing any requested checks
HFileScanner scanner = reader.getScanner(false, false, false);
- scanner.seekTo();
- scanKeyValues(file, scanner);
+ if (this.isSeekToRow) {
+// seek to the first kv on this row
+scanner.seekTo(KeyValue.createFirstOnRow(this.row).getKey());
+ } else {
+scanner.seekTo();
+ }
+ scanKeyValues(file, scanner, row);
}
// print meta data
@@ -220,7 +241,16 @@
reader.close();
}
- private void scanKeyValues(Path file, HFileScanner scanner)
+ /**
+ *
+ * @param file The path of the file
+ * @param scanner The HFileScanner for the file
+ * @param row Seek to the specific row and print all the kvs for this row.
+ * if Row is null, it means no row is specified and it will
+ * print all the rows in this file.
+ * @throws IOException
+ */
+ private void scanKeyValues(Path file, HFileScanner scanner, byte[] row)
throws IOException {
KeyValue pkv = null;
boolean first = true;
@@ -230,6 +260,18 @@
System.out.print([);
do {
KeyValue kv = scanner.getKeyValue();
+ if (row != null row.length != 0) {
+int result = Bytes.compareTo(kv.getRow(), row);
+if (result 0) {
+ System.out.println(All the data for row +
+ Bytes.toString(row) + has been printed out);
+ break;
+} else if (result 0) {
+ System.out.println(Start to scan for the row: +
+ Bytes.toString(row));
+ continue;
+}
+ }
// check if rows are in order
if (checkRow pkv != null) {
if (Bytes.compareTo(pkv.getRow(), kv.getRow()) 0) {
[jira] [Updated] (HBASE-4703) Improvements in tests
[ https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4703: - Status: Patch Available (was: Open) Submitting test to run it against patch build Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4703) Improvements in tests
[ https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139747#comment-13139747 ] stack commented on HBASE-4703: -- I'm not sure there is much to discuss about the recommendations made above by N; they seem commonsensical. N, we should make sure your patterns make it out into documentation -- there is a dev section in the manual that we should add them to. Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139750#comment-13139750 ] Lars Hofhansl commented on HBASE-4583: -- New patch on rb: https://reviews.apache.org/r/2633/ Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139751#comment-13139751 ] Lars Hofhansl commented on HBASE-4583: -- Some amount of boilerplate is necessary, because of the locking requirements (i.e. cannot have an upsert method in HRegion, since the setup and the first part of upsert need to have the same locks held). Could have a boilerplate method and pass in a command object, but that would not necessarily make it more readable. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure
[ https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139753#comment-13139753 ] stack commented on HBASE-4690: -- +1 on trying the patch. Commit it I'd say. We can resolve later if we go a period w/o these failures. Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure Key: HBASE-4690 URL: https://issues.apache.org/jira/browse/HBASE-4690 Project: HBase Issue Type: Test Affects Versions: 0.92.0 Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4690-trunk.txt See https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/ Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region server. One fix for this issue is to spin up MiniCluster with 1 region server so that we don't need to search for the region server where first region is hosted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4703) Improvements in tests
[ https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139755#comment-13139755 ] nkeywal commented on HBASE-4703: Yes, I was thinking about writing a small doc, I will have a look at the existing section and propose some modifications. Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK
[
https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139757#comment-13139757
]
stack commented on HBASE-4298:
--
The TDLS failed because of:
{code}
2011-10-30 00:16:16,142 ERROR
[org.apache.hadoop.hdfs.server.datanode.DataXceiver@15bc53]
datanode.DataXceiver(131): DatanodeRegistration(127.0.0.1:50070,
storageID=DS-1489265021-67.195.138.20-50070-1319933763131, infoPort=37761,
ipcPort=47077):DataXceiver
java.net.SocketException: Too many open files
{code}
Support to drain RS nodes through ZK
Key: HBASE-4298
URL: https://issues.apache.org/jira/browse/HBASE-4298
Project: HBase
Issue Type: Improvement
Components: master
Affects Versions: 0.90.4
Environment: all
Reporter: Aravind Gottipati
Priority: Critical
Labels: patch
Fix For: 0.92.0, 0.90.5
Attachments: 4298-trunk-v2.txt, 90_hbase.patch, trunk_hbase.patch
HDFS currently has a way to exclude certain datanodes and prevent them from
getting new blocks. HDFS goes one step further and even drains these nodes
for you. This enhancement is a step in that direction.
The idea is that we mark nodes in zookeeper as draining nodes. This means
that they don't get any more new regions. These draining nodes look exactly
the same as the corresponding nodes in /rs, except they live under /draining.
Eventually, support for draining them can be added. I am submitting two
patches for review - one for the 0.90 branch and one for trunk (in git).
Here are the two patches
0.90 -
https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2
trunk -
https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5
I have tested both these patches and they work as advertised.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4603) Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers
[ https://issues.apache.org/jira/browse/HBASE-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139758#comment-13139758 ] stack commented on HBASE-4603: -- @nkeywal Want me to commit this? Its committable. The TDLS failure was because of too many open files. Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers --- Key: HBASE-4603 URL: https://issues.apache.org/jira/browse/HBASE-4603 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all. Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111017_4603_MiniHBaseCluster.patch, 4603-v2.txt This functions waits for at least 2 times hbase.master.wait.on.regionservers.interval, defaulted at 3 seconds, i.e. 6 seconds for every mini hbase cluster starts. In the context of a mini cluster, it's not useful, as the regions servers are created locally. Changing this to a lower value such as 100ms gives 5.8 second per HBase cluser start. It should lower the build time on the apache server by more than 8%. Beeing more aggressive (removing all the wait time) could be possible as well. To be studied later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4677) Remove old single bulkLoadHFile method
[ https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139760#comment-13139760 ] stack commented on HBASE-4677: -- Comment from RB: +1 on patch Jon but we didn't deprecate this in 0.90 so removing it would be a little sudden? Should we deprecate it in 0.92 and apply this patch to TRUNK (0.94)? Remove old single bulkLoadHFile method -- Key: HBASE-4677 URL: https://issues.apache.org/jira/browse/HBASE-4677 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4677.patch In review for HBASE-4649, there is some debate as whether to remove, deprecate, or leave the HRegionServer.bulkLoadHFile method. https://reviews.apache.org/r/2545/ . This jira will take care of that for the 0.92 and trunk releases, and allow the same patch to remain for an optional 0.90.x patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4703) Improvements in tests
[ https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139763#comment-13139763 ] Ted Yu commented on HBASE-4703: --- bq. 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility I am not sure if we can remove sleep based synchronisation. As the analysis in HBASE-4690 showed, the timing of operations in cluster for different hardware / OS combinations would be different. I guess we don't want to make test code too complicated so that test code can be perfectly synchronous with minicluster. Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4603) Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers
[ https://issues.apache.org/jira/browse/HBASE-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139762#comment-13139762 ] nkeywal commented on HBASE-4603: Yes, it should decrease the total test duration by 5 minutes. Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers --- Key: HBASE-4603 URL: https://issues.apache.org/jira/browse/HBASE-4603 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all. Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111017_4603_MiniHBaseCluster.patch, 4603-v2.txt This functions waits for at least 2 times hbase.master.wait.on.regionservers.interval, defaulted at 3 seconds, i.e. 6 seconds for every mini hbase cluster starts. In the context of a mini cluster, it's not useful, as the regions servers are created locally. Changing this to a lower value such as 100ms gives 5.8 second per HBase cluser start. It should lower the build time on the apache server by more than 8%. Beeing more aggressive (removing all the wait time) could be possible as well. To be studied later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3680) Publish more metrics about mslab
[ https://issues.apache.org/jira/browse/HBASE-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3680: - Status: Open (was: Patch Available) Publish more metrics about mslab Key: HBASE-3680 URL: https://issues.apache.org/jira/browse/HBASE-3680 Project: HBase Issue Type: Improvement Affects Versions: 0.90.1 Reporter: Jean-Daniel Cryans Assignee: Todd Lipcon Fix For: 0.92.0 Attachments: hbase-3680.txt We have been using mslab on all our clusters for a while now and it seems it tends to OOME or send us into GC loops of death a lot more than it used to. For example, one RS with mslab enabled and 7GB of heap died out of OOME this afternoon; it had .55GB in the block cache and 2.03GB in the memstores which doesn't account for much... but it could be that because of mslab a lot of space was lost in those incomplete 2MB blocks and without metrics we can't really tell. Compactions were running at the time of the OOME and I see block cache activity. The average load on that cluster is 531. We should at least publish the total size of all those blocks and maybe even take actions based on that (like force flushing). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3680) Publish more metrics about mslab
[ https://issues.apache.org/jira/browse/HBASE-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3680: - Status: Patch Available (was: Open) Trying patch build. Publish more metrics about mslab Key: HBASE-3680 URL: https://issues.apache.org/jira/browse/HBASE-3680 Project: HBase Issue Type: Improvement Affects Versions: 0.90.1 Reporter: Jean-Daniel Cryans Assignee: Todd Lipcon Fix For: 0.92.0 Attachments: hbase-3680.txt We have been using mslab on all our clusters for a while now and it seems it tends to OOME or send us into GC loops of death a lot more than it used to. For example, one RS with mslab enabled and 7GB of heap died out of OOME this afternoon; it had .55GB in the block cache and 2.03GB in the memstores which doesn't account for much... but it could be that because of mslab a lot of space was lost in those incomplete 2MB blocks and without metrics we can't really tell. Compactions were running at the time of the OOME and I see block cache activity. The average load on that cluster is 531. We should at least publish the total size of all those blocks and maybe even take actions based on that (like force flushing). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3939) Some crossports of Hadoop IPC fixes
[ https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3939: - Status: Open (was: Patch Available) Some crossports of Hadoop IPC fixes --- Key: HBASE-3939 URL: https://issues.apache.org/jira/browse/HBASE-3939 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Fix For: 0.92.0 Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939.txt A few fixes from Hadoop IPC that we should probably cross-port into our copy: - HADOOP-7227: remove the protocol version check at call time - HADOOP-7146: fix a socket leak in server - HADOOP-7121: fix behavior when response serialization throws an exception - HADOOP-7346: send back nicer error response when client is using an out of date IPC version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3939) Some crossports of Hadoop IPC fixes
[ https://issues.apache.org/jira/browse/HBASE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3939: - Status: Patch Available (was: Open) Some crossports of Hadoop IPC fixes --- Key: HBASE-3939 URL: https://issues.apache.org/jira/browse/HBASE-3939 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Fix For: 0.92.0 Attachments: 3939-v2.txt, 3939-v3.txt, 3939-v4.txt, 3939.txt A few fixes from Hadoop IPC that we should probably cross-port into our copy: - HADOOP-7227: remove the protocol version check at call time - HADOOP-7146: fix a socket leak in server - HADOOP-7121: fix behavior when response serialization throws an exception - HADOOP-7346: send back nicer error response when client is using an out of date IPC version -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3680) Publish more metrics about mslab
[ https://issues.apache.org/jira/browse/HBASE-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139768#comment-13139768 ] stack commented on HBASE-3680: -- +1 Publish more metrics about mslab Key: HBASE-3680 URL: https://issues.apache.org/jira/browse/HBASE-3680 Project: HBase Issue Type: Improvement Affects Versions: 0.90.1 Reporter: Jean-Daniel Cryans Assignee: Todd Lipcon Fix For: 0.92.0 Attachments: hbase-3680.txt We have been using mslab on all our clusters for a while now and it seems it tends to OOME or send us into GC loops of death a lot more than it used to. For example, one RS with mslab enabled and 7GB of heap died out of OOME this afternoon; it had .55GB in the block cache and 2.03GB in the memstores which doesn't account for much... but it could be that because of mslab a lot of space was lost in those incomplete 2MB blocks and without metrics we can't really tell. Compactions were running at the time of the OOME and I see block cache activity. The average load on that cluster is 531. We should at least publish the total size of all those blocks and maybe even take actions based on that (like force flushing). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4603) Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers
[ https://issues.apache.org/jira/browse/HBASE-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4603: - Resolution: Fixed Fix Version/s: 0.92.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Applied to trunk and 0.92. Thanks for patch N. Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers --- Key: HBASE-4603 URL: https://issues.apache.org/jira/browse/HBASE-4603 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all. Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.92.0 Attachments: 20111017_4603_MiniHBaseCluster.patch, 4603-v2.txt This functions waits for at least 2 times hbase.master.wait.on.regionservers.interval, defaulted at 3 seconds, i.e. 6 seconds for every mini hbase cluster starts. In the context of a mini cluster, it's not useful, as the regions servers are created locally. Changing this to a lower value such as 100ms gives 5.8 second per HBase cluser start. It should lower the build time on the apache server by more than 8%. Beeing more aggressive (removing all the wait time) could be possible as well. To be studied later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4703) Improvements in tests
[
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139774#comment-13139774
]
nkeywal commented on HBASE-4703:
I don't have a generic solution. In some cases, it's possible to replace
something like this:
{noformat}
for (i1; inbTries; ++i){
if (cond())
break;
Thread.sleep(1s);
}
{noformat}
by
{noformat}
while (!timeout(5s) !cond()){
Thread.sleep(0.1s);
}
{noformat}
On average the second solution takes 0.45s less than the first. Basically, in
the tests we should be more aggressive with a 3 nodes cluster than in
production with a 100 nodes one. I.e.: it makes sense to test a condition 10
times per second on the tests and 1 time per second in prod. Ideally, there
should be no sleep at all, just a wait for the right event.
In tha patch, in most cases, the modifications are of the first type; just
increasing the number of tests per second, but not changing the timeout.
Improvements in tests
-
Key: HBASE-4703
URL: https://issues.apache.org/jira/browse/HBASE-4703
Project: HBase
Issue Type: Improvement
Components: test
Affects Versions: 0.92.0
Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
Global:
- when possible, make the test using the default cluster configuration for
the number of region (1 instead of 2 or 3). This allows a faster stop/start,
and is a step toward a shared cluster configuration.
- 'sleep': lower or remove the sleep based synchronisation in the tests (in
HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin,
TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache,
TestServerCustomProtocol, TestReplicationSink)
- Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big
or in a loop. Not done for tests that rely on the WAL.
Local issues:
- TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on
tearDown, that makes it impossible to use in // with another test using this
directory
- TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15
seconds to make it a part of the small subset
- TestMemoryBoundedLogMessageBuffer useless System.out.println
- io.hfile.TestReseekTo useless System.out.println
- TestTableInputFormat does not shutdown the cluster
- testGlobalMemStore does not shutdown the cluster
- rest.client.TestRemoteAdmin: simplified, does not use local admin, single
test instead of two.
- HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one
server, should start the number of missing server instead.
- TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure
[ https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139779#comment-13139779 ] Ted Yu commented on HBASE-4690: --- Integrate to 0.92 and TRUNK. Thanks for the review Stack. Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure Key: HBASE-4690 URL: https://issues.apache.org/jira/browse/HBASE-4690 Project: HBase Issue Type: Test Affects Versions: 0.92.0 Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4690-trunk.txt See https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/ Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region server. One fix for this issue is to spin up MiniCluster with 1 region server so that we don't need to search for the region server where first region is hosted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139788#comment-13139788 ] Ted Yu commented on HBASE-4583: --- bq. and then when the rowlock and the updatesLock.readLock are released, but after the rwcc is forwarded, the old rows are removed from the memstore. I think the above description would be more readable if expressed this way: after the rwcc is forwarded and, the rowlock and the updatesLock.readLock are released, the old rows are removed from the memstore. Correct me if the above is wrong. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139804#comment-13139804 ] Lars Hofhansl commented on HBASE-4583: -- Sure, that is more to the point. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Attachment: 4583-v2.txt Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Status: Patch Available (was: Open) Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4703) Improvements in tests
[ https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139817#comment-13139817 ] Hadoop QA commented on HBASE-4703: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501528/20111030_4703_all.v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 186 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.TestFullLogReconstruction org.apache.hadoop.hbase.master.TestMasterFailover Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/105//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/105//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/105//console This message is automatically generated. Improvements in tests - Key: HBASE-4703 URL: https://issues.apache.org/jira/browse/HBASE-4703 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch Global: - when possible, make the test using the default cluster configuration for the number of region (1 instead of 2 or 3). This allows a faster stop/start, and is a step toward a shared cluster configuration. - 'sleep': lower or remove the sleep based synchronisation in the tests (in HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin, TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache, TestServerCustomProtocol, TestReplicationSink) - Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big or in a loop. Not done for tests that rely on the WAL. Local issues: - TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on tearDown, that makes it impossible to use in // with another test using this directory - TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15 seconds to make it a part of the small subset - TestMemoryBoundedLogMessageBuffer useless System.out.println - io.hfile.TestReseekTo useless System.out.println - TestTableInputFormat does not shutdown the cluster - testGlobalMemStore does not shutdown the cluster - rest.client.TestRemoteAdmin: simplified, does not use local admin, single test instead of two. - HBaseTestingUtility#ensureSomeRegionServersAvailable starts only one server, should start the number of missing server instead. - TestMergeTool should starts/stops the dfs cluster with HBaseTestingUtility -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4703) Improvements in tests
[
https://issues.apache.org/jira/browse/HBASE-4703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139816#comment-13139816
]
Ted Yu commented on HBASE-4703:
---
I got the following test failures for the attached patch.
These two failed both in test suite and individually.
{code}
testFlushCommitsWithAbort(org.apache.hadoop.hbase.client.TestMultiParallel)
Time elapsed: 45.25 sec ERROR!
java.io.IOException: HRegionInfo was null or empty in .META.,
row=keyvalues={multi_test_table,ddd,1320006349447.09bf3a7042e4b84494f49b21d8e6f771./info:server/1320006349957/Put/vlen=19,
multi_test_table,ddd,1320006349447.09bf3a7042e4b84494f49b21d8e6f771./info:serverstartcode/1320006349957/Put/vlen=8}
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:908)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:769)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1429)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1314)
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:878)
at
org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:240)
at
org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:207)
{code}
{code}
testThreeRSAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting)
Time elapsed: 91.605 sec FAILURE!
java.lang.AssertionError:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at
org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testThreeRSAbort(TestDistributedLogSplitting.java:139)
{code}
The following failed in test suite but succeeded individually:
{code}
testMasterFailoverWithMockedRITOnDeadRS(org.apache.hadoop.hbase.master.TestMasterFailover)
Time elapsed: 180.011 sec ERROR!
java.lang.Exception: test timed out after 18 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.MiniHBaseCluster.waitForActiveAndReadyMaster(MiniHBaseCluster.java:385)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:923)
{code}
{code}
testRowRange(org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol)
Time elapsed: 0.037 sec FAILURE!
java.lang.AssertionError: Results should contain region
test,ccc,1320009243019.ffa363645a679c2c4caf69ede87bdfe3. for row 'ccc'
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.verifyRegionResults(TestServerCustomProtocol.java:335)
at
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.verifyRegionResults(TestServerCustomProtocol.java:327)
at
org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol.testRowRange(TestServerCustomProtocol.java:213)
{code}
Improvements in tests
-
Key: HBASE-4703
URL: https://issues.apache.org/jira/browse/HBASE-4703
Project: HBase
Issue Type: Improvement
Components: test
Affects Versions: 0.92.0
Environment: all
Reporter: nkeywal
Assignee: nkeywal
Priority: Minor
Attachments: 20111030_4703_all.patch, 20111030_4703_all.v2.patch
Global:
- when possible, make the test using the default cluster configuration for
the number of region (1 instead of 2 or 3). This allows a faster stop/start,
and is a step toward a shared cluster configuration.
- 'sleep': lower or remove the sleep based synchronisation in the tests (in
HBaseTestingUtility, TestGlobalMemStoreSize, TestAdmin,
TestCoprocessorEndpoint, TestHFileOutputFormat, TestLruBlockCache,
TestServerCustomProtocol, TestReplicationSink)
- Optimize 'put' by setting setWriteToWAL to false, when the 'put' is big
or in a loop. Not done for tests that rely on the WAL.
Local issues:
- TestTableInputFormatScan fully deletes the hadoop.tmp.dir directory on
tearDown, that makes it impossible to use in // with another test using this
directory
- TestIdLock logs too much (9000 lines per seconds). Test time lowered to 15
seconds to make it a part of the small subset
- TestMemoryBoundedLogMessageBuffer useless System.out.println
- io.hfile.TestReseekTo useless System.out.println
- TestTableInputFormat does not shutdown the cluster
- testGlobalMemStore does not shutdown the cluster
- rest.client.TestRemoteAdmin: simplified, does not use local admin, single
test instead of two.
-
[jira] [Commented] (HBASE-4603) Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers
[ https://issues.apache.org/jira/browse/HBASE-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139820#comment-13139820 ] Hudson commented on HBASE-4603: --- Integrated in HBase-TRUNK #2387 (See [https://builds.apache.org/job/HBase-TRUNK/2387/]) HBASE-4603 Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers stack : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/test/resources/hbase-site.xml Uneeded sleep time for tests in hbase.master.ServerManager#waitForRegionServers --- Key: HBASE-4603 URL: https://issues.apache.org/jira/browse/HBASE-4603 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.92.0 Environment: all. Reporter: nkeywal Assignee: nkeywal Priority: Minor Fix For: 0.92.0 Attachments: 20111017_4603_MiniHBaseCluster.patch, 4603-v2.txt This functions waits for at least 2 times hbase.master.wait.on.regionservers.interval, defaulted at 3 seconds, i.e. 6 seconds for every mini hbase cluster starts. In the context of a mini cluster, it's not useful, as the regions servers are created locally. Changing this to a lower value such as 100ms gives 5.8 second per HBase cluser start. It should lower the build time on the apache server by more than 8%. Beeing more aggressive (removing all the wait time) could be possible as well. To be studied later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139837#comment-13139837 ] Hadoop QA commented on HBASE-4583: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501538/4583-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.master.TestDistributedLogSplitting Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/106//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/106//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/106//console This message is automatically generated. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4690) Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure
[ https://issues.apache.org/jira/browse/HBASE-4690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139839#comment-13139839 ] Hudson commented on HBASE-4690: --- Integrated in HBase-TRUNK #2388 (See [https://builds.apache.org/job/HBase-TRUNK/2388/]) HBASE-4690 Intermittent TestRegionServerCoprocessorExceptionWithAbort failure tedyu : Files : * /hbase/trunk/CHANGES.txt * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java Intermittent TestRegionServerCoprocessorExceptionWithAbort#testExceptionFromCoprocessorDuringPut failure Key: HBASE-4690 URL: https://issues.apache.org/jira/browse/HBASE-4690 Project: HBase Issue Type: Test Affects Versions: 0.92.0 Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.92.0 Attachments: 4690-trunk.txt See https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92/83/testReport/junit/org.apache.hadoop.hbase.coprocessor/TestRegionServerCoprocessorExceptionWithAbort/testExceptionFromCoprocessorDuringPut/ Somehow getRSForFirstRegionInTable() wasn't able to retrieve the region server. One fix for this issue is to spin up MiniCluster with 1 region server so that we don't need to search for the region server where first region is hosted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139842#comment-13139842
]
jirapos...@reviews.apache.org commented on HBASE-4605:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2579/
---
(Updated 2011-10-31 00:48:49.738577)
Review request for hbase.
Changes
---
Updated based on comments on the last patch and in HBASE-4605 (corresponding
JIRA).
Added feautures:
- enable
- disable
- remove
- enable constraint
- disable constraint
- remove constraint
- actually enforcing ordering of constraints based on add
- update constraint
- removal of CPs in HTD
Also, updated tests to cover new cases (which all pass locally).
Still TODO: tie constraints into shell
Summary
---
Most of the implementation for adding constraints as a coprocessor.
Looking for general comments on style/structure, though nitpicks are ok too.
Currently missing implementation for disableConstraints() since that will
require adding removeCoprocessor() to HTD (also comments on if this is worth it
would be good).
This addresses bug HBASE-4605.
https://issues.apache.org/jira/browse/HBASE-4605
Diffs (updated)
-
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 99875b8
src/main/java/org/apache/hadoop/hbase/constraint/BaseConstraint.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/constraint/Constraint.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/constraint/Constraints.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/constraint/IntegerConstraint.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/constraint/IntegrationTestConstraint.java
PRE-CREATION
Diff: https://reviews.apache.org/r/2579/diff
Testing
---
Adding IntegrationTestConstraint and unit tests for Constraints and
IntegerConstraint. All of those pass.
Thanks,
Jesse
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139843#comment-13139843
]
jirapos...@reviews.apache.org commented on HBASE-4605:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2579/#review2934
---
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
https://reviews.apache.org/r/2579/#comment6568
Whoops - missed that one.
- Jesse
On 2011-10-31 00:48:49, Jesse Yates wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2579/
bq. ---
bq.
bq. (Updated 2011-10-31 00:48:49)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Most of the implementation for adding constraints as a coprocessor.
bq.
bq. Looking for general comments on style/structure, though nitpicks are ok
too.
bq.
bq. Currently missing implementation for disableConstraints() since that will
require adding removeCoprocessor() to HTD (also comments on if this is worth it
would be good).
bq.
bq.
bq. This addresses bug HBASE-4605.
bq. https://issues.apache.org/jira/browse/HBASE-4605
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 99875b8
bq.src/main/java/org/apache/hadoop/hbase/constraint/BaseConstraint.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/constraint/Constraint.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/constraint/ConstraintProcessor.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/constraint/Constraints.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/constraint/IntegerConstraint.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/constraint/IntegrationTestConstraint.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/2579/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Adding IntegrationTestConstraint and unit tests for Constraints and
IntegerConstraint. All of those pass.
bq.
bq.
bq. Thanks,
bq.
bq. Jesse
bq.
bq.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139854#comment-13139854
]
Ted Yu commented on HBASE-4605:
---
In the above patch, why do we need two loops in
Constraints.remove(HTableDescriptor) ?
Can we call desc.remove(e.getKey()) in the first loop ?
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139858#comment-13139858
]
Ted Yu commented on HBASE-4605:
---
I think the name of table should be added to the following exception:
{code}
throw new IllegalArgumentException(Constraint: + clazz.getName()
+ is not associated with this table.);
{code}
Constraints.getKeyValueForClass() may return null. I think we should check for
null in remove(HTableDescriptor, Class):
{code}
desc.remove(e.getKey().get());
{code}
and other methods.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4674) splitLog silently fails
[ https://issues.apache.org/jira/browse/HBASE-4674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139857#comment-13139857 ] Prakash Khemani commented on HBASE-4674: Stack, it is pretty obvious in the code. And yes, I have had seen lost edits a number of times. A simple way to reproduce this issue Create a table Kill namenode. That kills all region servers. Master doesn't die. Master tries to split logs and fails. But it ignores the failure and moves on to assign regions. Start namenode. Start regionservers The regions get assigned w/o their logs getting replayed. == BTW, the fix to this is being posted by Nicolas in HBASE-2312 splitLog silently fails --- Key: HBASE-4674 URL: https://issues.apache.org/jira/browse/HBASE-4674 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Environment: splitLog() can fail silently and region can open w/o its edits getting replayed. Reporter: Prakash Khemani Assignee: Prakash Khemani Priority: Blocker -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4064) Two concurrent unassigning of the same region caused the endless loop of Region has been PENDING_CLOSE for too long...
[
https://issues.apache.org/jira/browse/HBASE-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139860#comment-13139860
]
gaojinchao commented on HBASE-4064:
---
I found enable table issue was fixed in HBASE-4395.
I think this issue should use the same way to fix.
Two concurrent unassigning of the same region caused the endless loop of
Region has been PENDING_CLOSE for too long...
Key: HBASE-4064
URL: https://issues.apache.org/jira/browse/HBASE-4064
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.3
Reporter: Jieshan Bean
Fix For: 0.90.5
Attachments: HBASE-4064-v1.patch, HBASE-4064_branch90V2.patch,
disableflow.png
1. If there is a rubbish RegionState object with PENDING_CLOSE in
regionsInTransition(The RegionState was remained by some exception which
should be removed, that's why I called it as rubbish object), but the
region is not currently assigned anywhere, TimeoutMonitor will fall into an
endless loop:
2011-06-27 10:32:21,326 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
state=PENDING_CLOSE, ts=1309141555301
2011-06-27 10:32:21,326 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
2011-06-27 10:32:21,438 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
(offlining)
2011-06-27 10:32:21,441 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign
region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. but it is
not currently assigned anywhere
2011-06-27 10:32:31,207 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
state=PENDING_CLOSE, ts=1309141555301
2011-06-27 10:32:31,207 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
2011-06-27 10:32:31,215 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
(offlining)
2011-06-27 10:32:31,215 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign
region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. but it is
not currently assigned anywhere
2011-06-27 10:32:41,164 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed
out: test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
state=PENDING_CLOSE, ts=1309141555301
2011-06-27 10:32:41,164 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
2011-06-27 10:32:41,172 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of
region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f.
(offlining)
2011-06-27 10:32:41,172 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempted to unassign
region test2,070712,1308971310309.9a6e26d40293663a79523c58315b930f. but it is
not currently assigned anywhere
.
2 In the following scenario, two concurrent unassigning call of the same
region may lead to the above problem:
the first unassign call send rpc call success, the master watched the event
of RS_ZK_REGION_CLOSED, process this event, will create a
ClosedRegionHandler to remove the state of the region in master.eg.
while ClosedRegionHandler is running in
hbase.master.executor.closeregion.threads thread (A), another unassign call
of same region run in another thread(B).
while thread B run if (!regions.containsKey(region)), this.regions have
the region info, now cpu switch to thread A.
The thread A will remove the region from the sets of this.regions and
regionsInTransition, then switch to thread B. the thread B run continue,
will throw an exception with the msg of Server null returned
java.lang.NullPointerException: Passed server is null for
9a6e26d40293663a79523c58315b930f, but without removing the new-adding
RegionState from regionsInTransition,and it can not be removed for ever.
public void unassign(HRegionInfo region, boolean force) {
LOG.debug(Starting
[jira] [Commented] (HBASE-4677) Remove old single bulkLoadHFile method
[ https://issues.apache.org/jira/browse/HBASE-4677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139866#comment-13139866 ] Todd Lipcon commented on HBASE-4677: I don't think we need to deprecate this before removal, because IMO HRegionInterface is not a public API. Users should only be using the public interfaces like HTable, LoadIncrementalHFiles, etc. There is no real legitimate use to directly access the HRI API. Remove old single bulkLoadHFile method -- Key: HBASE-4677 URL: https://issues.apache.org/jira/browse/HBASE-4677 Project: HBase Issue Type: Sub-task Components: regionserver Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.92.0 Attachments: hbase-4677.patch In review for HBASE-4649, there is some debate as whether to remove, deprecate, or leave the HRegionServer.bulkLoadHFile method. https://reviews.apache.org/r/2545/ . This jira will take care of that for the 0.92 and trunk releases, and allow the same patch to remain for an optional 0.90.x patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Status: Patch Available (was: Open) Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Status: Open (was: Patch Available) Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Attachment: 4583-v3.txt Much cleaned up patch. Unifies the atomicUpsert (the called must acquire the locks, atomicUpsert releases them). Also includes rollback logic in case the sync of the WAL fails. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Status: Open (was: Patch Available) Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4704) A JRuby script for identifying active master
A JRuby script for identifying active master Key: HBASE-4704 URL: https://issues.apache.org/jira/browse/HBASE-4704 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Trivial Fix For: 0.94.0 This simple script reads the HBase master ZK node and outputs the hostname of the active master. This is needed so that operational scripts can decide where the primary master is running. I am also including a one-line hbase-jruby script so we can make our jruby scripts proper UNIX executables by including an #!/usr/bin/env hbase-jruby at the top. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4605) Constraints
[
https://issues.apache.org/jira/browse/HBASE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139887#comment-13139887
]
Ted Yu commented on HBASE-4605:
---
@Jesse:
I only see synchronized keyword on Constraints.add().
Have you tried using synchronization on other methods ?
Also, HTableDescriptor.values is protected field. We can change its actual
implementation to ConcurrentHashMap, etc to accommodate for the concurrency you
described.
If we store metadata about constraints in the Configuration object as I
described @ 29/Oct/11 04:20, we utilize the available serialization mechanism.
The current approach deals with serialization itself. This is not as flexible
as the above approach.
Constraints
---
Key: HBASE-4605
URL: https://issues.apache.org/jira/browse/HBASE-4605
Project: HBase
Issue Type: Improvement
Components: client, coprocessors
Affects Versions: 0.94.0
Reporter: Jesse Yates
Assignee: Jesse Yates
Attachments: constraint_as_cp.txt, java_Constraint_v2.patch
From Jesse's comment on dev:
{quote}
What I would like to propose is a simple interface that people can use to
implement a 'constraint' (matching the classic database definition). This
would help ease of adoption by helping HBase more easily check that box, help
minimize code duplication across organizations, and lead to easier adoption.
Essentially, people would implement a 'Constraint' interface for checking
keys before they are put into a table. Puts that are valid get written to the
table, but if not people can will throw an exception that gets propagated
back to the client explaining why the put was invalid.
Constraints would be set on a per-table basis and the user would be expected
to ensure the jars containing the constraint are present on the machines
serving that table.
Yes, people could roll their own mechanism for doing this via coprocessors
each time, but this would make it easier to do so, so you only have to
implement a very minimal interface and not worry about the specifics.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4704) A JRuby script for identifying active master
[ https://issues.apache.org/jira/browse/HBASE-4704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-4704: --- Attachment: D117.1.patch mbautin requested code review of HBASE-4704 [jira] A JRuby script for identifying active master. Reviewers: Kannan, Liyin, JIRA This simple script reads the HBase master ZK node and outputs the hostname of the active master. This is needed so that operational scripts can decide where the primary master is running. I am also adding a one-line hbase-jruby script so we can make our jruby scripts proper UNIX executables by including an #!/usr/bin/env hbase-jruby line at the top. TEST PLAN Run it. Kill the master. Wait for znode to expire. Run again. REVISION DETAIL https://reviews.facebook.net/D117 AFFECTED FILES bin/get-active-master.rb bin/hbase-jruby MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/249/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. A JRuby script for identifying active master Key: HBASE-4704 URL: https://issues.apache.org/jira/browse/HBASE-4704 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Trivial Fix For: 0.94.0 Attachments: D117.1.patch This simple script reads the HBase master ZK node and outputs the hostname of the active master. This is needed so that operational scripts can decide where the primary master is running. I am also including a one-line hbase-jruby script so we can make our jruby scripts proper UNIX executables by including an #!/usr/bin/env hbase-jruby at the top. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Attachment: 4583-v4.txt Latest patch (also on RB, but needs to be attached so that I can trigger a patch build). Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4583: - Status: Patch Available (was: Open) Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4705) HBase won't initialize if /hbase is not present
HBase won't initialize if /hbase is not present
---
Key: HBASE-4705
URL: https://issues.apache.org/jira/browse/HBASE-4705
Project: HBase
Issue Type: Bug
Reporter: Harsh J
{code}
2011-10-31 00:09:09,549 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled
exception. Starting shutdown.
java.io.FileNotFoundException: File does not exist: hdfs://C3S31:9000/hbase
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:731)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:163)
at org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:458)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:301)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
at
org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:426)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
at java.lang.Thread.run(Thread.java:662)
2011-10-31 00:09:09,551 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2011-10-31 00:09:09,551 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping
service threads
{code}
Trunk won't start HBase unless /hbase is already present, after HBASE-4680 (and
the silly error I made in HBASE-4510).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139907#comment-13139907 ] Hadoop QA commented on HBASE-4583: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501555/4583-v3.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.master.TestMasterFailover Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/107//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/107//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/107//console This message is automatically generated. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4680) FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have permissions
[
https://issues.apache.org/jira/browse/HBASE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139915#comment-13139915
]
Harsh J commented on HBASE-4680:
I apologize for missing the whole activity on this (didn't see a comment on
HBASE-4510, oops).
I think a better fix would have been to just catch other exceptions (which is
something I missed despite commenting).
This change is fine too, but has lead to HBASE-4705 scenarios. Could we revert
this and instead add a:
{code}
catch (Safemode) {
return true;
} catch (Others) {
ignore;
}
{code}
Kinda block, while still operating on the only dir guaranteed to exist (/ -
root)?
FSUtils.isInSafeMode() checks should operate on HBase root dir, where we have
permissions
-
Key: HBASE-4680
URL: https://issues.apache.org/jira/browse/HBASE-4680
Project: HBase
Issue Type: Bug
Components: util
Affects Versions: 0.92.0, 0.94.0
Reporter: Gary Helmling
Assignee: Gary Helmling
Priority: Critical
Fix For: 0.92.0
Attachments: HBASE-4680.patch
The HDFS safe mode check workaround introduced by HBASE-4510 performs a
{{FileSystem.setPermission()}} operation on the root directory (/) when
attempting to trigger a {{SafeModeException}}. As a result, it requires
superuser privileges when running with DFS permission checking enabled.
Changing the operations to act on the HBase root directory should be safe,
since the master process must have write access to it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4705) HBase won't initialize if /hbase is not present
[
https://issues.apache.org/jira/browse/HBASE-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139918#comment-13139918
]
Harsh J commented on HBASE-4705:
I apologize for missing the whole activity on HBASE-4680.
I think a better fix would have been, instead of the HBASE-4680, to just catch
other exceptions (which is something I missed despite commenting).
This change is fine too, but has lead to HBASE-4705 scenarios. Could we revert
HBASE-4680 and instead add a:
{code}
catch (Safemode) {
return true;
} catch (Others) {
ignore;
}
{code}
Kinda block, while still operating on the only dir guaranteed to exist (/ -
root)? This is so cause safemode is always checked first on the server side
before anything else is returned. The superuser check is not client side.
HBase won't initialize if /hbase is not present
---
Key: HBASE-4705
URL: https://issues.apache.org/jira/browse/HBASE-4705
Project: HBase
Issue Type: Bug
Reporter: Harsh J
{code}
2011-10-31 00:09:09,549 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.io.FileNotFoundException: File does not exist: hdfs://C3S31:9000/hbase
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:731)
at org.apache.hadoop.hbase.util.FSUtils.isInSafeMode(FSUtils.java:163)
at
org.apache.hadoop.hbase.util.FSUtils.waitOnSafeMode(FSUtils.java:458)
at
org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:301)
at
org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:127)
at
org.apache.hadoop.hbase.master.MasterFileSystem.init(MasterFileSystem.java:112)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:426)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
at java.lang.Thread.run(Thread.java:662)
2011-10-31 00:09:09,551 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2011-10-31 00:09:09,551 DEBUG org.apache.hadoop.hbase.master.HMaster:
Stopping service threads
{code}
Trunk won't start HBase unless /hbase is already present, after HBASE-4680
(and the silly error I made in HBASE-4510).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139929#comment-13139929 ] jirapos...@reviews.apache.org commented on HBASE-4120: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1421/ --- (Updated 2011-10-31 05:22:05.235409) Review request for hbase. Changes --- rewrite test cases. Summary --- Patch used for table priority alone,In this patch, not only tables can have different priorities but also the different actions like get,scan,put and delete can have priorities. This addresses bug HBase-4120. https://issues.apache.org/jira/browse/HBase-4120 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityFunction.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityHBaseServer.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/PriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1189169 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForActionPriority.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForPriorityJobQueue.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/allocation/test/TestForTablePriority.java PRE-CREATION Diff: https://reviews.apache.org/r/1421/diff Testing --- Tested with test cases in TestCase_For_TablePriority_trunk_v1.patch please apply the patch of HBASE-4181 first,in some circumstances this bug will affect the performance of client. Thanks, Jia isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-4120) isolation and allocation
[ https://issues.apache.org/jira/browse/HBASE-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Jia reassigned HBASE-4120: -- Assignee: Liu Jia isolation and allocation Key: HBASE-4120 URL: https://issues.apache.org/jira/browse/HBASE-4120 Project: HBase Issue Type: New Feature Components: master, regionserver Affects Versions: 0.90.2, 0.90.3, 0.90.4, 0.92.0 Reporter: Liu Jia Assignee: Liu Jia Fix For: 0.94.0 Attachments: Design_document_for_HBase_isolation_and_allocation.pdf, Design_document_for_HBase_isolation_and_allocation_Revised.pdf, HBase_isolation_and_allocation_user_guide.pdf, Performance_of_Table_priority.pdf, System Structure.jpg, TablePriority.patch The HBase isolation and allocation tool is designed to help users manage cluster resource among different application and tables. When we have a large scale of HBase cluster with many applications running on it, there will be lots of problems. In Taobao there is a cluster for many departments to test their applications performance, these applications are based on HBase. With one cluster which has 12 servers, there will be only one application running exclusively on this server, and many other applications must wait until the previous test finished. After we add allocation manage function to the cluster, applications can share the cluster and run concurrently. Also if the Test Engineer wants to make sure there is no interference, he/she can move out other tables from this group. In groups we use table priority to allocate resource, when system is busy; we can make sure high-priority tables are not affected lower-priority tables Different groups can have different region server configurations, some groups optimized for reading can have large block cache size, and others optimized for writing can have large memstore size. Tables and region servers can be moved easily between groups; after changing the configuration, a group can be restarted alone instead of restarting the whole cluster. git entry : https://github.com/ICT-Ope/HBase_allocation . We hope our work is helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-4706) HBASE-4120 Create shell or portal tool for user to manage priority and group
HBASE-4120 Create shell or portal tool for user to manage priority and group Key: HBASE-4706 URL: https://issues.apache.org/jira/browse/HBASE-4706 Project: HBase Issue Type: Sub-task Components: ipc, shell Affects Versions: 0.92.0 Reporter: Liu Jia Assignee: Liu Jia Fix For: 0.94.0 Add a tool for user to manage the functions provided by HBase-4120 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4277) HRS.closeRegion should be able to close regions with only the encoded name
[ https://issues.apache.org/jira/browse/HBASE-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139933#comment-13139933 ] ramkrishna.s.vasudevan commented on HBASE-4277: --- I will see if the patch still applies if not will prepare an updated on and then commit it. Thanks for your review Stack HRS.closeRegion should be able to close regions with only the encoded name -- Key: HBASE-4277 URL: https://issues.apache.org/jira/browse/HBASE-4277 Project: HBase Issue Type: Bug Affects Versions: 0.90.4 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Priority: Critical Fix For: 0.90.5 Attachments: HBASE-4277_0.90.patch As suggested by Stack in HBASE-4217 creating a new issue to provide a patch for 0.90.x version. We had some sort of an outage this morning due to a few racks losing power, and some regions were left in the following state: ERROR: Region UNKNOWN_REGION on sv4r17s9:60020, key=e32bbe1f48c9b3633c557dc0291b90a3, not on HDFS or in META but deployed on sv4r17s9:60020 That region was deleted by the master but the region server never got the memo. Right now there's no way to force close it because HRS.closeRegion requires an HRI and the only way to create one is to get it from .META. which in our case doesn't contain a row for that region. Basically we have to wait until that server is dead to get rid of the region and make hbck happy. The required change is to have closeRegion accept an encoded name in both HBA (when the RS address is provided) and HRS since it's able to find it anyways from it's list of live regions. bq.If a 0.90 version, we maybe should do that in another issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4706) HBASE-4120 Create shell or portal tool for user to manage priority and group
[ https://issues.apache.org/jira/browse/HBASE-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sinfox updated HBASE-4706: -- Attachment: TablePriorityJamon.patch portal tool for this issue. HBASE-4120 Create shell or portal tool for user to manage priority and group Key: HBASE-4706 URL: https://issues.apache.org/jira/browse/HBASE-4706 Project: HBase Issue Type: Sub-task Components: ipc, shell Affects Versions: 0.92.0 Reporter: Liu Jia Assignee: Liu Jia Fix For: 0.94.0 Attachments: TablePriorityJamon.patch Original Estimate: 504h Remaining Estimate: 504h Add a tool for user to manage the functions provided by HBase-4120 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4583) Integrate RWCC with Append and Increment operations
[ https://issues.apache.org/jira/browse/HBASE-4583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13139936#comment-13139936 ] Hadoop QA commented on HBASE-4583: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501559/4583-v4.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated -166 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestMasterFailover org.apache.hadoop.hbase.master.TestDistributedLogSplitting org.apache.hadoop.hbase.regionserver.TestSplitLogWorker Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/108//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/108//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/108//console This message is automatically generated. Integrate RWCC with Append and Increment operations --- Key: HBASE-4583 URL: https://issues.apache.org/jira/browse/HBASE-4583 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.0 Attachments: 4583-v2.txt, 4583-v3.txt, 4583-v4.txt, 4583.txt Currently Increment and Append operations do not work with RWCC and hence a client could see the results of multiple such operation mixed in the same Get/Scan. The semantics might be a bit more interesting here as upsert adds and removes to and from the memstore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
