[jira] [Commented] (HBASE-5670) Have Mutation implement the Row interface.
[ https://issues.apache.org/jira/browse/HBASE-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241017#comment-13241017 ] Hadoop QA commented on HBASE-5670: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520372/5670-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1337//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1337//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1337//console This message is automatically generated. Have Mutation implement the Row interface. -- Key: HBASE-5670 URL: https://issues.apache.org/jira/browse/HBASE-5670 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Trivial Fix For: 0.94.1 Attachments: 5670-0.94.txt, 5670-trunk.txt, 5670-trunk.txt In HBASE-4347 I factored some code from Put/Delete/Append in Mutation. In a discussion with a co-worker I noticed that Put/Delete/Append still implement the Row interface, but Mutation does not. In a trivial change I would like to move that interface up to Mutation, along with changing HTable.batch(ListRow) to HTable.batch(List? extends Row) (HConnection.processBatch takes List? extends Row already anyway), so that HTable.batch can be used with a list of Mutations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241018#comment-13241018 ] Jesse Yates commented on HBASE-5547: Implementation consideration: should we be backing up _all_ deletes or just those that are system enabled. For instance, we could delete a table via the api/shell. However, that is a pointed act and probably doesn't mean we want to back that data up. Proposing we keep it simple FTM and just backup system removed HFiles. Otherwise, we keep around tons of copies of files. Alternative I wouldn't mind (but increases the size of the patch and feels like scope creep) would be to backup even for table level deletes and on things like deleting a table, you need a bool to specify but not the backup. thoughts? Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records
[ https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241021#comment-13241021 ] Hadoop QA commented on HBASE-5564: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520371/5564v5.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1336//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1336//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1336//console This message is automatically generated. Bulkload is discarding duplicate records Key: HBASE-5564 URL: https://issues.apache.org/jira/browse/HBASE-5564 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0 Environment: HBase 0.92 Reporter: Laxman Assignee: Laxman Labels: bulkloader Fix For: 0.96.0 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch Duplicate records are getting discarded when duplicate records exists in same input file and more specifically if they exists in same split. Duplicate records are considered if the records are from diffrent different splits. Version under test: HBase 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5669) AggregationClient fails validation for open stoprow scan
[ https://issues.apache.org/jira/browse/HBASE-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mubarak Seyed updated HBASE-5669: - Attachment: HBASE-5669.trunk.v1.patch The attached file is a patch. Thanks. AggregationClient fails validation for open stoprow scan Key: HBASE-5669 URL: https://issues.apache.org/jira/browse/HBASE-5669 Project: HBase Issue Type: Bug Components: coprocessors Affects Versions: 0.92.1 Environment: n/a Reporter: Brian Rogers Priority: Minor Fix For: 0.92.2 Attachments: HBASE-5669.trunk.v1.patch Original Estimate: 2h Remaining Estimate: 2h AggregationClient.validateParameters throws an exception when the Scan has a valid startrow but an unset endrow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241043#comment-13241043 ] ramkrishna.s.vasudevan commented on HBASE-5617: --- @Stack I will be adding test cases for the rollback scenarios. I wanted to get the things clarified before giving a patch. Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tudor Scurtu updated HBASE-5625: Attachment: 5625v4.txt @Zhihong: I added a new method, 'TestResult.testPerformance()'. This must be of course removed if the patch is accepted. It prints the results, so it doesn't require an integrated profiler. It takes a few minutes to run... Mentioned that the buffer isn't cleared or flipped in the 'KeyValue.loadValue()' method comment as well. Is that what you meant? Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5673) The OOM problem of IPC client call cause all handle block
The OOM problem of IPC client call cause all handle block -- Key: HBASE-5673 URL: https://issues.apache.org/jira/browse/HBASE-5673 Project: HBase Issue Type: Bug Affects Versions: 0.90.6 Environment: 0.90.6 Reporter: xufeng Assignee: xufeng if HBaseClient meet unable to create new native thread exception, the call will never complete because it be lost in calls queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5673) The OOM problem of IPC client call cause all handle block
[ https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241065#comment-13241065 ] xufeng commented on HBASE-5673: --- I found this issue in my cluster. 1.I found any regionserver call not report to master because sockettimeout. {noformat} [2012-03-26 14:48:09,815] [INFO ] [regionserver20020] [org.apache.hadoop.hbase.regionserver.HRegionServer 1469] Attempting connect to Master server at DDB03:2 [2012-03-26 14:49:09,818] [INFO ] [regionserver20020] [org.apache.hadoop.ipc.HbaseRPC 360] Problem connecting to server: DDB03/192.168.28.53:2 [2012-03-26 14:49:09,819] [WARN ] [regionserver20020] [org.apache.hadoop.hbase.regionserver.HRegionServer 1483] Unable to connect to master. Retrying. Error was: java.net.SocketTimeoutException: Call to DDB03/192.168.28.53:2 failed on socket timeout exception: java.net.SocketTimeoutException: 6 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.28.53:59520 remote=DDB03/192.168.28.53:2] {noformat} 2.through the jstack log of master,I found that one handle is waitting and others is blocked(waitForMeta). {noformat} IPC Server handler 90 on 2 daemon prio=10 tid=0x7f219c54 nid=0x4c3f in Object.wait() [0x7f21963a7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:757) 。 IPC Server handler 87 on 2 daemon prio=10 tid=0x7f219c53a000 nid=0x4c37 waiting for monitor entry [0x7f21966aa000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMeta(CatalogTracker.java:397) - waiting to lock 0x000612486960 (a java.util.concurrent.atomic.AtomicBoolean) at org.apache.hadoop.hbase.catalog.CatalogTracker.waitForMetaServerConnectionDefault(CatalogTracker.java:437) 。。。 {noformat} 3.I also ensure that the waitting handle cause the others blocked,the waitting handle is waitting for the call to complete. 4.But the unable to create new native thread” happened, the IOException can not caught it. {noformat} protected synchronized void setupIOstreams() throws IOException { start(); } catch (IOException e) { markClosed(e); close(); throw e; } 。 {noformat} 5.thus the call will be lost in call queue and never to complete. {noformat} public Writable call(..) { .. synchronized (call) { while (!call.done) { try { call.wait(); // wait for the result } catch (InterruptedException ignored) { // save the fact that we were interrupted interrupted = true; } } .. } {noformat} The OOM problem of IPC client call cause all handle block -- Key: HBASE-5673 URL: https://issues.apache.org/jira/browse/HBASE-5673 Project: HBase Issue Type: Bug Affects Versions: 0.90.6 Environment: 0.90.6 Reporter: xufeng Assignee: xufeng if HBaseClient meet unable to create new native thread exception, the call will never complete because it be lost in calls queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5673) The OOM problem of IPC client call cause all handle block
[ https://issues.apache.org/jira/browse/HBASE-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241069#comment-13241069 ] xufeng commented on HBASE-5673: --- Step 4 miss some logs info: {noformat} java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:640) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:351) {noformat} The OOM problem of IPC client call cause all handle block -- Key: HBASE-5673 URL: https://issues.apache.org/jira/browse/HBASE-5673 Project: HBase Issue Type: Bug Affects Versions: 0.90.6 Environment: 0.90.6 Reporter: xufeng Assignee: xufeng if HBaseClient meet unable to create new native thread exception, the call will never complete because it be lost in calls queue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241077#comment-13241077 ] Jonathan Hsieh commented on HBASE-5638: --- Matteo, We should probably use the built-in boolean parser in configuration instead of doing a the string compare -- something like this: {code} if (conf.getBoolean(HConstants.CLUSTER_DISTRIBUTED, HConstants.DEFAULT_CLUSTER_DISTRIBUTED)) {code} see: http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/conf/Configuration.html#getBoolean(java.lang.String, boolean) We'd also end up changing HConstants.DEFAULT_CLUSTER_DISTRBUTED to be a boolean type instead of a string as well. Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records
[ https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241076#comment-13241076 ] ramkrishna.s.vasudevan commented on HBASE-5564: --- +1 on v5. Thanks for the patch Lakshman and Stack. @Stack So any new patches that we give should not have any findbugs even if in the old existing code? Ok i will take care of this and ensure people submitting patches over here also do that. Thanks. Bulkload is discarding duplicate records Key: HBASE-5564 URL: https://issues.apache.org/jira/browse/HBASE-5564 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0 Environment: HBase 0.92 Reporter: Laxman Assignee: Laxman Labels: bulkloader Fix For: 0.96.0 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch Duplicate records are getting discarded when duplicate records exists in same input file and more specifically if they exists in same split. Duplicate records are considered if the records are from diffrent different splits. Version under test: HBase 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records
[ https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241078#comment-13241078 ] Laxman commented on HBASE-5564: --- @stack, thanks for your review and clearing the findbugs. I was avoiding these changes as these are unrelated to this JIRA. @ram, thanks for reviewing the patch. Bulkload is discarding duplicate records Key: HBASE-5564 URL: https://issues.apache.org/jira/browse/HBASE-5564 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0 Environment: HBase 0.92 Reporter: Laxman Assignee: Laxman Labels: bulkloader Fix For: 0.96.0 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch Duplicate records are getting discarded when duplicate records exists in same input file and more specifically if they exists in same split. Duplicate records are considered if the records are from diffrent different splits. Version under test: HBase 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5635) If getTaskList() returns null splitlogWorker is down. It wont serve any requests.
[ https://issues.apache.org/jira/browse/HBASE-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241088#comment-13241088 ] ramkrishna.s.vasudevan commented on HBASE-5635: --- @Chinna {code} private ListString getTaskList() { for (int i = 0; i zkretries; i++) { try { return (ZKUtil.listChildrenAndWatchForNewChildren(this.watcher, this.watcher.splitLogZNode)); } {code} Currently there is no retry mechanism working in getTaskList. Though there is a loop there is no retry. One more thing is we can differentiate if really there was a problem in setting a watch by returning null and if the watch was success but there is no child for the split node. In that case i think we can return an empty list. Also as the thread is just going for a wait state without watch getting set successfully the thread will keep waiting. If getTaskList() returns null splitlogWorker is down. It wont serve any requests. -- Key: HBASE-5635 URL: https://issues.apache.org/jira/browse/HBASE-5635 Project: HBase Issue Type: Bug Components: wal Affects Versions: 0.92.1 Reporter: Kristam Subba Swathi Attachments: HBASE-5635.patch During the hlog split operation if all the zookeepers are down ,then the paths will be returned as null and the splitworker thread wil be exited Now this regionserver wil not be able to acquire any other tasks since the splitworker thread is exited Please find the attached code for more details {code} private ListString getTaskList() { for (int i = 0; i zkretries; i++) { try { return (ZKUtil.listChildrenAndWatchForNewChildren(this.watcher, this.watcher.splitLogZNode)); } catch (KeeperException e) { LOG.warn(Could not get children of znode + this.watcher.splitLogZNode, e); try { Thread.sleep(1000); } catch (InterruptedException e1) { LOG.warn(Interrupted while trying to get task list ..., e1); Thread.currentThread().interrupt(); return null; } } } {code} in the org.apache.hadoop.hbase.regionserver.SplitLogWorker -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241092#comment-13241092 ] Hadoop QA commented on HBASE-5625: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520387/5625v4.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1338//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1338//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1338//console This message is automatically generated. Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5651) [findbugs] Address wait/notify synchronization/inconsistency in sync
[ https://issues.apache.org/jira/browse/HBASE-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-5651: - Assignee: ramkrishna.s.vasudevan [findbugs] Address wait/notify synchronization/inconsistency in sync Key: HBASE-5651 URL: https://issues.apache.org/jira/browse/HBASE-5651 Project: HBase Issue Type: Sub-task Components: scripts Reporter: Jonathan Hsieh Assignee: ramkrishna.s.vasudevan See https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_MT_CORRECTNESS fix classes IS,LI,MWM, NN, SWL, UG, UW -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: HBASE-5638-trunk-v1.patch HBASE-5638-0.92-v1.patch HBASE-5638-0.90-v1.patch patch updated for 0.90, 0.92 and trunk to use conf.getBoolean() Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241150#comment-13241150 ] Cosmin Lehene commented on HBASE-5625: -- Microbenchmarking is a bit trickier than this. Perhaps we should mention some guideline it in the HBase developer documentation. There's a lot of stuff going on in the HotSpot (e.g. dynamic compilation) that needs to be taken into account. Here are some resources: http://www.slideshare.net/drorbr/so-you-want-to-write-your-own-benchmark-presentation https://wikis.oracle.com/display/HotSpotInternals/MicroBenchmarks We might be able to use something like junit-benchmarks. http://labs.carrotsearch.com/junit-benchmarks.html Cosmin Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241153#comment-13241153 ] Hadoop QA commented on HBASE-5638: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520408/HBASE-5638-trunk-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1339//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1339//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1339//console This message is automatically generated. Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: (was: HBASE-5638-trunk-v1.patch) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records
[ https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241167#comment-13241167 ] Uma Maheswara Rao G commented on HBASE-5564: Also don't forget to update the count in test-patch.properties according to the present count if we fix any existing findbugs. +Uma Bulkload is discarding duplicate records Key: HBASE-5564 URL: https://issues.apache.org/jira/browse/HBASE-5564 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0 Environment: HBase 0.92 Reporter: Laxman Assignee: Laxman Labels: bulkloader Fix For: 0.96.0 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch Duplicate records are getting discarded when duplicate records exists in same input file and more specifically if they exists in same split. Duplicate records are considered if the records are from diffrent different splits. Version under test: HBase 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: HBASE-5638-trunk-v1.patch Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: (was: HBASE-5638-0.90-v1.patch) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: (was: HBASE-5638-0.92-v1.patch) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: HBASE-5638-0.92-v1.patch HBASE-5638-0.90-v1.patch Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241169#comment-13241169 ] Hadoop QA commented on HBASE-5638: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520414/HBASE-5638-0.92-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1340//console This message is automatically generated. Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241170#comment-13241170 ] Hadoop QA commented on HBASE-5638: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520414/HBASE-5638-0.92-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1341//console This message is automatically generated. Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5667) RegexStringComparator supports java.util.regex.Pattern flags
[ https://issues.apache.org/jira/browse/HBASE-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241197#comment-13241197 ] David Arthur commented on HBASE-5667: - Anything needed from me to get this in? RegexStringComparator supports java.util.regex.Pattern flags Key: HBASE-5667 URL: https://issues.apache.org/jira/browse/HBASE-5667 Project: HBase Issue Type: Improvement Components: filters Reporter: David Arthur Priority: Minor Attachments: HBASE-5667.diff * Add constructor that takes in a Pattern * Add Pattern's flags to Writable fields, and actually use them when recomposing the Filter -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5669) AggregationClient fails validation for open stoprow scan
[ https://issues.apache.org/jira/browse/HBASE-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241206#comment-13241206 ] Brian Rogers commented on HBASE-5669: - Same changes I was going to make. Thanks for the quick turnaround, will save me some time! AggregationClient fails validation for open stoprow scan Key: HBASE-5669 URL: https://issues.apache.org/jira/browse/HBASE-5669 Project: HBase Issue Type: Bug Components: coprocessors Affects Versions: 0.92.1 Environment: n/a Reporter: Brian Rogers Priority: Minor Fix For: 0.92.2 Attachments: HBASE-5669.trunk.v1.patch Original Estimate: 2h Remaining Estimate: 2h AggregationClient.validateParameters throws an exception when the Scan has a valid startrow but an unset endrow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-1697) Discretionary access control
[ https://issues.apache.org/jira/browse/HBASE-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241232#comment-13241232 ] Laxman commented on HBASE-1697: --- No updates here from long time. From my understanding, to make HBase secure, we need huge contributions in this area. Also, this involves many challenges (architectural changes, maintain/break compatibility, ...). In spite of these challenges, it adds more value to HBase. Anyone interested to look into these security issues? Discretionary access control Key: HBASE-1697 URL: https://issues.apache.org/jira/browse/HBASE-1697 Project: HBase Issue Type: Improvement Components: security Reporter: Andrew Purtell Assignee: Andrew Purtell Consider implementing discretionary access control for HBase. Access control has three aspects: authentication, authorization and audit. - Authentication: Access is controlled by insisting on an authentication procedure to establish the identity of the user. The authentication procedure should minimally require a non-plaintext authentication factor (e.g. encrypted password with salt) and should ideally or at least optionally provide cryptographically strong confidence via public key certification. - Authorization: Access is controlled by specifying rights to resources via an access control list (ACL). An ACL is a list of permissions attached to an object. The list specifies who or what is allowed to access the object and what operations are allowed to be performed on the object, f.e. create, update, read, or delete. - Audit: Important actions taken by subjects should be logged for accountability, a chronological record which enables the full reconstruction and examination of a sequence of events, e.g. schema changes or data mutations. Logging activity should be protected from all subjects except for a restricted set with administrative privilege, perhaps to only a single super-user. Discretionary access control means the access policy for an object is determined by the owner of the object. Every object in the system must have a valid owner. Owners can assign access rights and permissions to other users. The initial owner of an object is the subject who created it. If subjects are deleted from a system, ownership of objects owned by them should revert to some super-user or otherwise valid default. HBase can enforce access policy at table, column family, or cell granularity. Cell granularity does not make much sense. An implementation which controls access at both the table and column family levels is recommended, though a first cut could consider control at the table level only. The initial set of permissions can be: Create (table schema or column family), update (table schema or column family), read (column family), delete (table or column family), execute (filters), and transfer ownership. The subject identities and access tokens could be stored in a new administrative table. ACLs on tables and column families can be stored in META. Access other than read access to catalog and administrative tables should be restricted to a set of administrative users or perhaps a single super-user. A data mutation on a user table by a subject without administrative or superuser privilege which results in a table split is an implicit temporary privilege elevation where the regionserver or master updates the catalog tables as necessary to support the split. Audit logging should be configurable on a per-table basis to avoid this overhead where it is not wanted. Consider supporting external authentication and subject identification mechanisms with Java library support: RADIUS/TACACS, Kerberos, LDAP. Consider logging audit trails to an HBase table (bigtable type schemas are natural for this) and optionally external logging options with Java library support -- syslog, etc., or maybe commons-logging is sufficient and punt to administrator to set up appropriate commons-logging/log4j configurations for their needs. If HBASE-1002 is considered, and the option to support filtering via upload of (perhaps complex) bytecode produced by some little language compiler is implemented, the execute privilege could be extended in a manner similar to how stored procedures in SQL land execute either with the privilege of the current user or the (table/procedure) creator. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4398) If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml.
[ https://issues.apache.org/jira/browse/HBASE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated HBASE-4398: - Attachment: HBASE-4398.patch I attatched a patch file. Tnis includes a patch for HRegionPartitioner and a test case. If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml. - Key: HBASE-4398 URL: https://issues.apache.org/jira/browse/HBASE-4398 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.90.4 Reporter: Takuya Ueshin Attachments: HBASE-4398.patch If HRegionPartitioner is used in MapReduce, client side configurations are overwritten by hbase-site.xml. We can reproduce the problem by the following instructions: {noformat} - Add HRegionPartitioner.class to the 4th argument of TableMapReduceUtil#initTableReducerJob() at line around 133 in src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java - Change or remove hbase.zookeeper.property.clientPort property in hbase-site.xml ( for example, changed to 12345 ). - run testMultiRegionTable() {noformat} Then I got error messages as following: {noformat} 2011-09-12 22:28:51,020 DEBUG [Thread-832] zookeeper.ZKUtil(93): hconnection opening connection to ZooKeeper with ensemble (localhost:12345) 2011-09-12 22:28:51,022 INFO [Thread-832] zookeeper.RecoverableZooKeeper(89): The identifier of this process is 43200@imac.local 2011-09-12 22:28:51,123 WARN [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2011-09-12 22:28:51,123 INFO [Thread-832] zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after sleeping 1000 ms = 2011-09-12 22:29:02,418 ERROR [Thread-832] mapreduce.HRegionPartitioner(125): java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2e54e48d closed 2011-09-12 22:29:02,422 WARN [Thread-832] mapred.LocalJobRunner$Job(256): job_local_0001 java.lang.NullPointerException at org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:128) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:527) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) {noformat} I think HTable should connect to ZooKeeper at port 21818 configured at client side instead of 12345 in hbase-site.xml and It might be caused by HBaseConfiguration.addHbaseResources(conf); in HRegionPartitioner#setConf(Configuration). And this might mean that all of client side configurations, also configured in hbase-site.xml, are overwritten caused by this problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5669) AggregationClient fails validation for open stoprow scan
[ https://issues.apache.org/jira/browse/HBASE-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241300#comment-13241300 ] Hadoop QA commented on HBASE-5669: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520373/HBASE-5669.trunk.v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1342//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1342//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1342//console This message is automatically generated. AggregationClient fails validation for open stoprow scan Key: HBASE-5669 URL: https://issues.apache.org/jira/browse/HBASE-5669 Project: HBase Issue Type: Bug Components: coprocessors Affects Versions: 0.92.1 Environment: n/a Reporter: Brian Rogers Assignee: Mubarak Seyed Priority: Minor Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: HBASE-5669.trunk.v1.patch Original Estimate: 2h Remaining Estimate: 2h AggregationClient.validateParameters throws an exception when the Scan has a valid startrow but an unset endrow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records
[ https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241307#comment-13241307 ] stack commented on HBASE-5564: -- Committed to trunk. Thanks for the patch Laxman. Thanks for the reminder on updating the count Uma. It seems that my minor addition only stopped the count rising so I didn't have to change the findbugs count (the test build was seeing two new findbug warnings when in fact there were none -- a variable name change was making it think the findbugs count had gone up). Bulkload is discarding duplicate records Key: HBASE-5564 URL: https://issues.apache.org/jira/browse/HBASE-5564 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0 Environment: HBase 0.92 Reporter: Laxman Assignee: Laxman Labels: bulkloader Fix For: 0.96.0 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch Duplicate records are getting discarded when duplicate records exists in same input file and more specifically if they exists in same split. Duplicate records are considered if the records are from diffrent different splits. Version under test: HBase 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5564) Bulkload is discarding duplicate records
[ https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5564: - Resolution: Fixed Status: Resolved (was: Patch Available) Bulkload is discarding duplicate records Key: HBASE-5564 URL: https://issues.apache.org/jira/browse/HBASE-5564 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0 Environment: HBase 0.92 Reporter: Laxman Assignee: Laxman Labels: bulkloader Fix For: 0.96.0 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch Duplicate records are getting discarded when duplicate records exists in same input file and more specifically if they exists in same split. Duplicate records are considered if the records are from diffrent different splits. Version under test: HBase 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5669) AggregationClient fails validation for open stoprow scan
[ https://issues.apache.org/jira/browse/HBASE-5669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5669: - Resolution: Fixed Status: Resolved (was: Patch Available) Applied to 0.92, 0.94 and trunk. Thanks for the patch Mubarak. Lars, this patch is marked 0.94.1 If we do another RC, we should change this to be 0.94.0. AggregationClient fails validation for open stoprow scan Key: HBASE-5669 URL: https://issues.apache.org/jira/browse/HBASE-5669 Project: HBase Issue Type: Bug Components: coprocessors Affects Versions: 0.92.1 Environment: n/a Reporter: Brian Rogers Assignee: Mubarak Seyed Priority: Minor Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: HBASE-5669.trunk.v1.patch Original Estimate: 2h Remaining Estimate: 2h AggregationClient.validateParameters throws an exception when the Scan has a valid startrow but an unset endrow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5671) hbase.metrics.showTableName should be true by default
[ https://issues.apache.org/jira/browse/HBASE-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-5671. -- Resolution: Fixed Hadoop Flags: Reviewed I committed to the 0.94 branch too... will be in 0.94.1 or in the next RC. hbase.metrics.showTableName should be true by default - Key: HBASE-5671 URL: https://issues.apache.org/jira/browse/HBASE-5671 Project: HBase Issue Type: Improvement Components: metrics Reporter: Enis Soztutar Assignee: Enis Soztutar Priority: Critical Fix For: 0.94.1 Attachments: HBASE-5671_v1.patch HBASE-4768 added per-cf metrics and a new configuration option hbase.metrics.showTableName. We should switch the conf option to true by default, since it is not intuitive (at least to me) to aggregate per-cf across tables by default, and it seems confusing to report on cf's without table names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241330#comment-13241330 ] jirapos...@reviews.apache.org commented on HBASE-4348: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4402/ --- (Updated 2012-03-29 15:57:19.858937) Review request for hbase. Changes --- changes done as per Jon's feedback. The access to MasterMetrics instance in the AssignmentManager class is guarded by a null check. Summary --- This patch is for adding Region in transition metrics to the HMaster metrics system. It also adds these metrics in the master ui, in the Region in transition section. I have attached the proposed new format in the jira 4348. This addresses bug HBase-4348. https://issues.apache.org/jira/browse/HBase-4348 Diffs (updated) - src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon 0dc0691 src/main/java/org/apache/hadoop/hbase/HConstants.java 19be4de src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java ae468ca src/main/java/org/apache/hadoop/hbase/master/HMaster.java c4b4d30 src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 83abc52 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java d68ce33 Diff: https://reviews.apache.org/r/4402/diff Testing --- Ran on a 5 node cluster and kill region servers randomly to observe the changes in the RIT metrics as emitted out by the Master's mxbean; mvn test passes without any failure. Thanks, Himanshu Add metrics for regions in transition - Key: HBASE-4348 URL: https://issues.apache.org/jira/browse/HBASE-4348 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Himanshu Vashishtha Priority: Minor Labels: noob Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, RITs.png, RegionInTransitions2.png, metrics-v2.patch The following metrics would be useful for monitoring the master: - the number of regions in transition - the number of regions in transition that have been in transition for more than a minute - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5655) Cap space usage of default log4j rolling policy
[ https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-5655: --- Attachment: 5655-v1.patch Proposed changes to have the RFA policy. Cap space usage of default log4j rolling policy --- Key: HBASE-5655 URL: https://issues.apache.org/jira/browse/HBASE-5655 Project: HBase Issue Type: Improvement Affects Versions: 0.92.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Attachments: 5655-v1.patch The current default log4j policy is to use Daily Rolling File Appender (DRFA). At times, its good to have a cap on the maximum size of the logs in order to limit its disk usage. Here is a proposal to set a new file appemder (RFA) as the default appender. It can be configured via env so that existing tools can use the current behavior of using DRFA instead. This is in parallel with jira Hadoop-8149. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records
[ https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241341#comment-13241341 ] Jonathan Hsieh commented on HBASE-5564: --- maybe not worry about find bugs for normal patches? (ideally it does go up though) the find bugs number isn't the focus of this patch. Sent from my iPhone Bulkload is discarding duplicate records Key: HBASE-5564 URL: https://issues.apache.org/jira/browse/HBASE-5564 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0 Environment: HBase 0.92 Reporter: Laxman Assignee: Laxman Labels: bulkloader Fix For: 0.96.0 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch Duplicate records are getting discarded when duplicate records exists in same input file and more specifically if they exists in same split. Duplicate records are considered if the records are from diffrent different splits. Version under test: HBase 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241346#comment-13241346 ] Lars Hofhansl commented on HBASE-5547: -- I'd just backup system removed HFiles if you can always tell the difference. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5670) Have Mutation implement the Row interface.
[ https://issues.apache.org/jira/browse/HBASE-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5670: - Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Have Mutation implement the Row interface. -- Key: HBASE-5670 URL: https://issues.apache.org/jira/browse/HBASE-5670 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Trivial Fix For: 0.96.0, 0.94.1 Attachments: 5670-0.94.txt, 5670-trunk.txt, 5670-trunk.txt In HBASE-4347 I factored some code from Put/Delete/Append in Mutation. In a discussion with a co-worker I noticed that Put/Delete/Append still implement the Row interface, but Mutation does not. In a trivial change I would like to move that interface up to Mutation, along with changing HTable.batch(ListRow) to HTable.batch(List? extends Row) (HConnection.processBatch takes List? extends Row already anyway), so that HTable.batch can be used with a list of Mutations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5671) hbase.metrics.showTableName should be true by default
[ https://issues.apache.org/jira/browse/HBASE-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5671: - Fix Version/s: 0.96.0 hbase.metrics.showTableName should be true by default - Key: HBASE-5671 URL: https://issues.apache.org/jira/browse/HBASE-5671 Project: HBase Issue Type: Improvement Components: metrics Reporter: Enis Soztutar Assignee: Enis Soztutar Priority: Critical Fix For: 0.96.0, 0.94.1 Attachments: HBASE-5671_v1.patch HBASE-4768 added per-cf metrics and a new configuration option hbase.metrics.showTableName. We should switch the conf option to true by default, since it is not intuitive (at least to me) to aggregate per-cf across tables by default, and it seems confusing to report on cf's without table names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5617: -- Status: Open (was: Patch Available) Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5617: -- Status: Patch Available (was: Open) Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch, HBASE-5617_2.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-5617: -- Attachment: HBASE-5617_2.patch @Andy This patch deals only with what is intended in the title of the JIRA. Can i raise another JIRA for adding other hooks? Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch, HBASE-5617_2.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241374#comment-13241374 ] Jimmy Xiang commented on HBASE-5619: @Stack, could you please commit this patch? I do have some changes to pb files. But I'd like to address them in HBASE-5620. Thanks. Create PB protocols for HRegionInterface Key: HBASE-5619 URL: https://issues.apache.org/jira/browse/HBASE-5619 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5619.patch, hbase-5619_v3.patch, hbase-5619_v4.patch Subtask of HBase-5443, separate HRegionInterface into admin protocol and client protocol, create the PB protocol buffer files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5671) hbase.metrics.showTableName should be true by default
[ https://issues.apache.org/jira/browse/HBASE-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241375#comment-13241375 ] Hudson commented on HBASE-5671: --- Integrated in HBase-0.94 #63 (See [https://builds.apache.org/job/HBase-0.94/63/]) HBASE-5671 hbase.metrics.showTableName should be true by default (Revision 1306924) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/main/resources/hbase-default.xml hbase.metrics.showTableName should be true by default - Key: HBASE-5671 URL: https://issues.apache.org/jira/browse/HBASE-5671 Project: HBase Issue Type: Improvement Components: metrics Reporter: Enis Soztutar Assignee: Enis Soztutar Priority: Critical Fix For: 0.96.0, 0.94.1 Attachments: HBASE-5671_v1.patch HBASE-4768 added per-cf metrics and a new configuration option hbase.metrics.showTableName. We should switch the conf option to true by default, since it is not intuitive (at least to me) to aggregate per-cf across tables by default, and it seems confusing to report on cf's without table names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241379#comment-13241379 ] Lars Hofhansl commented on HBASE-5617: -- In 0.94+ there are more places where we rollback the memstore. Check out HRegion.mutateRowsWithLocks in 0.94 and HRegion.processRowsWithLocks in 0.96. Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch, HBASE-5617_2.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table
[ https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene reassigned HBASE-5665: Assignee: Cosmin Lehene Repeated split causes HRegionServer failures and breaks table -- Key: HBASE-5665 URL: https://issues.apache.org/jira/browse/HBASE-5665 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0, 0.92.1 Reporter: Cosmin Lehene Assignee: Cosmin Lehene Priority: Blocker Repeated splits on large tables (2 consecutive would suffice) will essentially break the table (and the cluster), unrecoverable. The regionserver doing the split dies and the master will get into an infinite loop trying to assign regions that seem to have the files missing from HDFS. The table can be disabled once. upon trying to re-enable it, it will remain in an intermediary state forever. I was able to reproduce this on a smaller table consistently. {code} hbase(main):030:0 (0..1).each{|x| put 't1', #{x}, 'f1:t', 'dd'} hbase(main):030:0 (0..1000).each{|x| split 't1', #{x*10}} {code} Running overlapping splits in parallel (e.g. #{x*10+1}, #{x*10+2}... ) will reproduce the issue almost instantly and consistently. {code} 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in META 2012-03-28 10:57:16,321 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1.. compaction_queue=(0:1), split_queue=10 2012-03-28 10:57:16,343 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 java.io.IOException: Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363) at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237 at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1008) at org.apache.hadoop.hbase.io.HalfStoreFileReader.init(HalfStoreFileReader.java:65) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548) at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284) at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:221) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511) at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229) at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504) at org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484) ... 1 more 2012-03-28 10:57:16,345 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ld2,60020,1332957343833: Abort; we got an error after point-of-no-return {code} http://hastebin.com/diqinibajo.avrasm later edit: (I'm using the last 4 characters from each string) Region 94e3 has storefile 7237 Region 94e3 gets splited in daughters a: ffa1 and b: eee1 Daughter region ffa1 get's splitted in daughters a: 3124 and b: dc77 ffa1 has a reference: 7237.94e3 for it's store file when ffa1 gets splited it
[jira] [Updated] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table
[ https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-5665: - Attachment: HBASE-5665-0.92.patch Adding patch. From Reference.java documentation {code} * Note, a region is itself not splitable if it has instances of store file * references. References are cleaned up by compactions. {code} SplitTransaction.prepare() should check if the parent region has references. I added a unit test and patch. Funny enough the HRegion.hasReferences was implemented but only used from a unit test. I think recursive references wouldn't be so hard to have if there's a good reason to have them in the first place. Repeated split causes HRegionServer failures and breaks table -- Key: HBASE-5665 URL: https://issues.apache.org/jira/browse/HBASE-5665 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0, 0.92.1 Reporter: Cosmin Lehene Assignee: Cosmin Lehene Priority: Blocker Attachments: HBASE-5665-0.92.patch Repeated splits on large tables (2 consecutive would suffice) will essentially break the table (and the cluster), unrecoverable. The regionserver doing the split dies and the master will get into an infinite loop trying to assign regions that seem to have the files missing from HDFS. The table can be disabled once. upon trying to re-enable it, it will remain in an intermediary state forever. I was able to reproduce this on a smaller table consistently. {code} hbase(main):030:0 (0..1).each{|x| put 't1', #{x}, 'f1:t', 'dd'} hbase(main):030:0 (0..1000).each{|x| split 't1', #{x*10}} {code} Running overlapping splits in parallel (e.g. #{x*10+1}, #{x*10+2}... ) will reproduce the issue almost instantly and consistently. {code} 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in META 2012-03-28 10:57:16,321 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1.. compaction_queue=(0:1), split_queue=10 2012-03-28 10:57:16,343 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 java.io.IOException: Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363) at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237 at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1008) at org.apache.hadoop.hbase.io.HalfStoreFileReader.init(HalfStoreFileReader.java:65) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548) at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284) at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:221) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511) at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229) at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504) at
[jira] [Updated] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table
[ https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-5665: - Affects Version/s: 0.94.1 0.96.0 0.94.0 0.94 and trunk seem to suffer from this as well and not checking if parent has references. Repeated split causes HRegionServer failures and breaks table -- Key: HBASE-5665 URL: https://issues.apache.org/jira/browse/HBASE-5665 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0, 0.92.1, 0.94.0, 0.96.0, 0.94.1 Reporter: Cosmin Lehene Assignee: Cosmin Lehene Priority: Blocker Attachments: HBASE-5665-0.92.patch Repeated splits on large tables (2 consecutive would suffice) will essentially break the table (and the cluster), unrecoverable. The regionserver doing the split dies and the master will get into an infinite loop trying to assign regions that seem to have the files missing from HDFS. The table can be disabled once. upon trying to re-enable it, it will remain in an intermediary state forever. I was able to reproduce this on a smaller table consistently. {code} hbase(main):030:0 (0..1).each{|x| put 't1', #{x}, 'f1:t', 'dd'} hbase(main):030:0 (0..1000).each{|x| split 't1', #{x*10}} {code} Running overlapping splits in parallel (e.g. #{x*10+1}, #{x*10+2}... ) will reproduce the issue almost instantly and consistently. {code} 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in META 2012-03-28 10:57:16,321 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1.. compaction_queue=(0:1), split_queue=10 2012-03-28 10:57:16,343 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 java.io.IOException: Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363) at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237 at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1008) at org.apache.hadoop.hbase.io.HalfStoreFileReader.init(HalfStoreFileReader.java:65) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548) at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284) at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:221) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511) at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229) at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504) at org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484) ... 1 more 2012-03-28 10:57:16,345 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ld2,60020,1332957343833: Abort; we got an error after point-of-no-return {code} http://hastebin.com/diqinibajo.avrasm later edit: (I'm using the last 4 characters from each string) Region
[jira] [Commented] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241395#comment-13241395 ] Zhihong Yu commented on HBASE-5617: --- rollbackMemstore() doesn't throw exception. I want to understand why we need two hooks for the rollback scenario. Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch, HBASE-5617_2.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241397#comment-13241397 ] Zhihong Yu commented on HBASE-5625: --- Here is output from TestResult#testPerformance: {code} loadValue(): 122281 getValue(): 141426 {code} Improvement was about 13.5% Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241399#comment-13241399 ] Hadoop QA commented on HBASE-5617: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520451/HBASE-5617_2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestHRegion org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1343//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1343//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1343//console This message is automatically generated. Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch, HBASE-5617_2.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5638: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520414/HBASE-5638-0.92-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1341//console This message is automatically generated.) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5638: -- Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520414/HBASE-5638-0.92-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1340//console This message is automatically generated.) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5670) Have Mutation implement the Row interface.
[ https://issues.apache.org/jira/browse/HBASE-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241419#comment-13241419 ] Hudson commented on HBASE-5670: --- Integrated in HBase-0.94 #64 (See [https://builds.apache.org/job/HBase-0.94/64/]) HBASE-5670 Have Mutation implement the Row interface. (Revision 1306958) Result = SUCCESS larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Append.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Delete.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/HTablePool.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Mutation.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Put.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/client/RemoteHTable.java Have Mutation implement the Row interface. -- Key: HBASE-5670 URL: https://issues.apache.org/jira/browse/HBASE-5670 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Trivial Fix For: 0.96.0, 0.94.1 Attachments: 5670-0.94.txt, 5670-trunk.txt, 5670-trunk.txt In HBASE-4347 I factored some code from Put/Delete/Append in Mutation. In a discussion with a co-worker I noticed that Put/Delete/Append still implement the Row interface, but Mutation does not. In a trivial change I would like to move that interface up to Mutation, along with changing HTable.batch(ListRow) to HTable.batch(List? extends Row) (HConnection.processBatch takes List? extends Row already anyway), so that HTable.batch can be used with a list of Mutations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241435#comment-13241435 ] ramkrishna.s.vasudevan commented on HBASE-5617: --- @Lars I will update in other places also. @Ted What do you think? Only one hook is enough? For our use case one hook is enough. From your previous comment i felt you suggested like its better we have two. Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch, HBASE-5617_2.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241443#comment-13241443 ] Zhihong Yu commented on HBASE-5625: --- I have put some comments here: https://reviews.apache.org/r/4559/ @Tudor: Feel free to create your own review request for the new patch. Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5674) add support in HBase to overwrite hbase timestamp to a version number during major compaction
add support in HBase to overwrite hbase timestamp to a version number during major compaction - Key: HBASE-5674 URL: https://issues.apache.org/jira/browse/HBASE-5674 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Right now, a millisecond-level timestamp is attached to every record. In our case, we only need a version number (mostly it will be just zero etc). A millisecond timestamp is too heavy to carry. We should add support to overwrite it to zero during major compaction. KVs before major compaction will remain using system timestamp. And this should be configurable, so that we should not mess up if the hbase timestamp is specified by application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5617) Provide coprocessor hooks in put flow while rollbackMemstore.
[ https://issues.apache.org/jira/browse/HBASE-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241446#comment-13241446 ] Zhihong Yu commented on HBASE-5617: --- I think one hook should be enough. Provide coprocessor hooks in put flow while rollbackMemstore. - Key: HBASE-5617 URL: https://issues.apache.org/jira/browse/HBASE-5617 Project: HBase Issue Type: Improvement Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-5617_1.patch, HBASE-5617_2.patch With coprocessors hooks while put happens we have the provision to create new puts to other tables or regions. These puts can be done with writeToWal as false. In 0.94 and above the puts are first written to memstore and then to WAL. If any failure in the WAL append or sync the memstore is rollbacked. Now the problem is that if the put that happens in the main flow fails there is no way to rollback the puts that happened in the prePut. We can add coprocessor hooks to like pre/postRoolBackMemStore. Is any one hook enough here? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241452#comment-13241452 ] Jonathan Hsieh commented on HBASE-5599: --- Thanks for updating the patches. Instead of relying on shouldFixAssignments, could you add a new config flag for fixing the NO_VERSION_FILE? The fix isn't really a region assignment fix! Maybe call it -fixVersionFile, and have it on with the combo '-repair*' options? Can you add a test case for SHOULD_NOT_BE_DEPLOYED? You should be able to hack one of the test cases from TestHBaseFsck. I've had problems just using assign before so if tests come back failing, you may want to try HBaseFsckRepair.fixMultiAssignemnt or tryAssignemntRepair. The merge addition isn't really related -- was its inclusion intended? The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. Key: HBASE-5599 URL: https://issues.apache.org/jira/browse/HBASE-5599 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.90.6 Reporter: fulin wang Fix For: 0.90.6 Attachments: hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, hbase-5599-0.92_v5.patch, hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch The hbck tool can not fix the six scenarios. 1. Version file does not exist in root dir. Fix: I try to create a version file by 'FSUtils.setVersion' method. 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any region server. Fix: I get region info form the hdfs file, this region info write to '.META.' table. 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] Fix: I get region info form the hdfs file, this region info write to '.META.' table. 4. [REGIONNAME] should not be deployed according to META, but is deployed on [SERVERNAME] Fix: Close this region. 5. First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this error. 6. There is a hole in the region chain between [KEY] and [KEY]. You need to create a new regioninfo and region dir in hdfs to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this hole. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5655) Cap space usage of default log4j rolling policy
[ https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-5655: --- Status: Patch Available (was: Open) Cap space usage of default log4j rolling policy --- Key: HBASE-5655 URL: https://issues.apache.org/jira/browse/HBASE-5655 Project: HBase Issue Type: Improvement Affects Versions: 0.92.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Attachments: 5655-v1.patch The current default log4j policy is to use Daily Rolling File Appender (DRFA). At times, its good to have a cap on the maximum size of the logs in order to limit its disk usage. Here is a proposal to set a new file appemder (RFA) as the default appender. It can be configured via env so that existing tools can use the current behavior of using DRFA instead. This is in parallel with jira Hadoop-8149. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5655) Cap space usage of default log4j rolling policy
[ https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241460#comment-13241460 ] Himanshu Vashishtha commented on HBASE-5655: I have tested this by running HBase with this settings. It uses the RFA policy. Using an env property with HBASE_ROOT_LOGGER=DRFA takes the old DRFA policy. Cap space usage of default log4j rolling policy --- Key: HBASE-5655 URL: https://issues.apache.org/jira/browse/HBASE-5655 Project: HBase Issue Type: Improvement Affects Versions: 0.92.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Attachments: 5655-v1.patch The current default log4j policy is to use Daily Rolling File Appender (DRFA). At times, its good to have a cap on the maximum size of the logs in order to limit its disk usage. Here is a proposal to set a new file appemder (RFA) as the default appender. It can be configured via env so that existing tools can use the current behavior of using DRFA instead. This is in parallel with jira Hadoop-8149. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5638: -- Assignee: Matteo Bertozzi Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status
[ https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mubarak Seyed reassigned HBASE-5675: Assignee: Mubarak Seyed Create table fails if we keep refreshing master's UI for task monitor status Key: HBASE-5675 URL: https://issues.apache.org/jira/browse/HBASE-5675 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4 Reporter: Mubarak Seyed Assignee: Mubarak Seyed Labels: noob I tried to create a table with 2K pre-split regions, region assignment was in middle and i was keep refreshing master's web UI to find the status of the task using task monitor, table creation was failed and {{META}} was showing 2K regions with server location value is {{null}} and regions weren't deployed onto region-servers. {code} table_ACreating table table_A java.io.IOException: java.io.IOException: java.util.ConcurrentModificationException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294) at com.test.tools.hbase.schema.createIfNotExists(schema.java:520) at com.test.tools.hbase.schema.main(schema.java:627) Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.util.ConcurrentModificationException at java.util.SubList.checkForComodification(AbstractList.java:752) at java.util.SubList.add(AbstractList.java:632) at java.util.SubList.add(AbstractList.java:633) at java.util.SubList.add(AbstractList.java:633) .. .. at java.util.SubList.add(AbstractList.java:633) at java.util.AbstractList.add(AbstractList.java:91) at org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy5.createTable(Unknown Source) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241464#comment-13241464 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/#review6525 --- http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java https://reviews.apache.org/r/4096/#comment14193 Devaraj, the problem with up'ing the version number is that it makes the life of backwards-compatible clients like asynchbase even more difficult than it already is. I proposed another idea to Stack, I don't know if he shared it with you, so here it is: In the znode used to store the -ROOT- region, add the protocol version number (make it 5 there if you want). This way clients that are finding where -ROOT- is will be able to figure out the protocol version to use *before* connecting to -ROOT-. This better than parsing the string of the VersionMismatchException, which you said yourself is hacky (and also inefficient), so we don't wanna do that. - Benoit On 2012-03-01 03:40:14, Devaraj Das wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4096/ bq. --- bq. bq. (Updated 2012-03-01 03:40:14) bq. bq. bq. Review request for . bq. bq. bq. Summary bq. --- bq. bq. Switch RPC call envelope/headers to PBs bq. bq. bq. This addresses bug HBASE-5451. bq. https://issues.apache.org/jira/browse/HBASE-5451 bq. bq. bq. Diffs bq. - bq. bq.http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1294899 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1294899 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1294899 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1294899 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1294899 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/proto/RPCMessageProto.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4096/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Devaraj bq. bq. Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: (was: HBASE-5638-trunk-v1.patch) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: HBASE-5638-trunk-v1.patch Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: HBASE-5638-trunk-v1.patch Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: (was: HBASE-5638-trunk-v1.patch) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status
[ https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241471#comment-13241471 ] Mubarak Seyed commented on HBASE-5675: -- I could create a table using workarounds # Increase the RPC timeout value to 10 minutes # Create one table at a time # No refresh of master's web UI Create table fails if we keep refreshing master's UI for task monitor status Key: HBASE-5675 URL: https://issues.apache.org/jira/browse/HBASE-5675 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4 Reporter: Mubarak Seyed Assignee: Mubarak Seyed Labels: noob I tried to create a table with 2K pre-split regions, region assignment was in middle and i was keep refreshing master's web UI to find the status of the task using task monitor, table creation was failed and {{META}} was showing 2K regions with server location value is {{null}} and regions weren't deployed onto region-servers. {code} table_ACreating table table_A java.io.IOException: java.io.IOException: java.util.ConcurrentModificationException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294) at com.test.tools.hbase.schema.createIfNotExists(schema.java:520) at com.test.tools.hbase.schema.main(schema.java:627) Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.util.ConcurrentModificationException at java.util.SubList.checkForComodification(AbstractList.java:752) at java.util.SubList.add(AbstractList.java:632) at java.util.SubList.add(AbstractList.java:633) at java.util.SubList.add(AbstractList.java:633) .. .. at java.util.SubList.add(AbstractList.java:633) at java.util.AbstractList.add(AbstractList.java:91) at org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy5.createTable(Unknown Source) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status
[ https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241477#comment-13241477 ] Lars Hofhansl commented on HBASE-5675: -- TaskMonitor does not seem exist in 0.90.x. In 0.92.x it is properly synchronized - although the synchronization was changed in 0.94+ (unnecessarily IMHO). Create table fails if we keep refreshing master's UI for task monitor status Key: HBASE-5675 URL: https://issues.apache.org/jira/browse/HBASE-5675 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4 Reporter: Mubarak Seyed Assignee: Mubarak Seyed Labels: noob I tried to create a table with 2K pre-split regions, region assignment was in middle and i was keep refreshing master's web UI to find the status of the task using task monitor, table creation was failed and {{META}} was showing 2K regions with server location value is {{null}} and regions weren't deployed onto region-servers. {code} table_ACreating table table_A java.io.IOException: java.io.IOException: java.util.ConcurrentModificationException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294) at com.test.tools.hbase.schema.createIfNotExists(schema.java:520) at com.test.tools.hbase.schema.main(schema.java:627) Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.util.ConcurrentModificationException at java.util.SubList.checkForComodification(AbstractList.java:752) at java.util.SubList.add(AbstractList.java:632) at java.util.SubList.add(AbstractList.java:633) at java.util.SubList.add(AbstractList.java:633) .. .. at java.util.SubList.add(AbstractList.java:633) at java.util.AbstractList.add(AbstractList.java:91) at org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy5.createTable(Unknown Source) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5564) Bulkload is discarding duplicate records
[ https://issues.apache.org/jira/browse/HBASE-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241485#comment-13241485 ] Jonathan Hsieh commented on HBASE-5564: --- meant to say ideally it does not go up. I think stack's action (he didn't lower findbugs number on normal patch) captured the same idea. Bulkload is discarding duplicate records Key: HBASE-5564 URL: https://issues.apache.org/jira/browse/HBASE-5564 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0 Environment: HBase 0.92 Reporter: Laxman Assignee: Laxman Labels: bulkloader Fix For: 0.96.0 Attachments: 5564.lint, 5564v5.txt, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.1.patch, HBASE-5564_trunk.2.patch, HBASE-5564_trunk.3.patch, HBASE-5564_trunk.4_final.patch, HBASE-5564_trunk.patch Duplicate records are getting discarded when duplicate records exists in same input file and more specifically if they exists in same split. Duplicate records are considered if the records are from diffrent different splits. Version under test: HBase 0.92 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[ https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241493#comment-13241493 ] ramkrishna.s.vasudevan commented on HBASE-5097: --- Committed to 0.94, 0.92 and trunk. Sorry for the delay in committing the patches. Thanks to Stack, Andy, Lars and Ted for the review. RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE --- Key: HBASE-5097 URL: https://issues.apache.org/jira/browse/HBASE-5097 Project: HBase Issue Type: Bug Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: HBASE-5097.patch, HBASE-5097_1.patch, HBASE-5097_2.patch In HRegionServer.java openScanner() {code} r.prepareScanner(scan); RegionScanner s = null; if (r.getCoprocessorHost() != null) { s = r.getCoprocessorHost().preScannerOpen(scan); } if (s == null) { s = r.getScanner(scan); } if (r.getCoprocessorHost() != null) { s = r.getCoprocessorHost().postScannerOpen(scan, s); } {code} If we dont have implemention for postScannerOpen the RegionScanner is null and so throwing nullpointer {code} java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881) at org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) {code} Making this defect as blocker.. Pls feel free to change the priority if am wrong. Also correct me if my way of trying out coprocessors without implementing postScannerOpen is wrong. Am just a learner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5655) Cap space usage of default log4j rolling policy
[ https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241512#comment-13241512 ] Hadoop QA commented on HBASE-5655: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520446/5655-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster org.apache.hadoop.hbase.mapreduce.TestImportTsv org.apache.hadoop.hbase.mapred.TestTableMapReduce org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1344//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1344//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1344//console This message is automatically generated. Cap space usage of default log4j rolling policy --- Key: HBASE-5655 URL: https://issues.apache.org/jira/browse/HBASE-5655 Project: HBase Issue Type: Improvement Affects Versions: 0.92.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Attachments: 5655-v1.patch The current default log4j policy is to use Daily Rolling File Appender (DRFA). At times, its good to have a cap on the maximum size of the logs in order to limit its disk usage. Here is a proposal to set a new file appemder (RFA) as the default appender. It can be configured via env so that existing tools can use the current behavior of using DRFA instead. This is in parallel with jira Hadoop-8149. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-50) Snapshot of table
[ https://issues.apache.org/jira/browse/HBASE-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241515#comment-13241515 ] Zhihong Yu commented on HBASE-50: - @Jesse: When you create new review request on review board, leave Bugs field empty. Thanks Snapshot of table - Key: HBASE-50 URL: https://issues.apache.org/jira/browse/HBASE-50 Project: HBase Issue Type: New Feature Affects Versions: 0.96.0 Reporter: Billy Pearson Assignee: Li Chongxin Priority: Minor Labels: gsoc Fix For: 0.96.0 Attachments: HBase Snapshot Design Report V2.pdf, HBase Snapshot Design Report V3.pdf, HBase Snapshot Implementation Plan.pdf, Snapshot Class Diagram.png Havening an option to take a snapshot of a table would be vary useful in production. What I would like to see this option do is do a merge of all the data into one or more files stored in the same folder on the dfs. This way we could save data in case of a software bug in hadoop or user code. The other advantage would be to be able to export a table to multi locations. Say I had a read_only table that must be online. I could take a snapshot of it when needed and export it to a separate data center and have it loaded there and then i would have it online at multi data centers for load balancing and failover. I understand that hadoop takes the need out of havening backup to protect from failed servers, but this does not protect use from software bugs that might delete or alter data in ways we did not plan. We should have a way we can roll back a dataset. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status
[ https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mubarak Seyed updated HBASE-5675: - Affects Version/s: 0.94.0 0.92.0 Create table fails if we keep refreshing master's UI for task monitor status Key: HBASE-5675 URL: https://issues.apache.org/jira/browse/HBASE-5675 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Mubarak Seyed Assignee: Mubarak Seyed Labels: noob I tried to create a table with 2K pre-split regions, region assignment was in middle and i was keep refreshing master's web UI to find the status of the task using task monitor, table creation was failed and {{META}} was showing 2K regions with server location value is {{null}} and regions weren't deployed onto region-servers. {code} table_ACreating table table_A java.io.IOException: java.io.IOException: java.util.ConcurrentModificationException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294) at com.test.tools.hbase.schema.createIfNotExists(schema.java:520) at com.test.tools.hbase.schema.main(schema.java:627) Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.util.ConcurrentModificationException at java.util.SubList.checkForComodification(AbstractList.java:752) at java.util.SubList.add(AbstractList.java:632) at java.util.SubList.add(AbstractList.java:633) at java.util.SubList.add(AbstractList.java:633) .. .. at java.util.SubList.add(AbstractList.java:633) at java.util.AbstractList.add(AbstractList.java:91) at org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy5.createTable(Unknown Source) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status
[ https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241513#comment-13241513 ] Mubarak Seyed commented on HBASE-5675: -- @Lars CDH3u2 backports distributed log splitting to 0.90.x branch and task monitor stuff as well. I have verified the code {code} public ListStoreFile close(final boolean abort) throws IOException { // Only allow one thread to close at a time. Serialize them so dual // threads attempting to close will run up against each other. MonitoredTask status = TaskMonitor.get().createStatus( Closing region + this + (abort ? due to abort : )); status.setStatus(Waiting for close lock); try { synchronized (closeLock) { return doClose(abort, status); } } finally { status.cleanup(); } } {code} It is affected in 0.92 and above i believe. Thanks. Create table fails if we keep refreshing master's UI for task monitor status Key: HBASE-5675 URL: https://issues.apache.org/jira/browse/HBASE-5675 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Mubarak Seyed Assignee: Mubarak Seyed Labels: noob I tried to create a table with 2K pre-split regions, region assignment was in middle and i was keep refreshing master's web UI to find the status of the task using task monitor, table creation was failed and {{META}} was showing 2K regions with server location value is {{null}} and regions weren't deployed onto region-servers. {code} table_ACreating table table_A java.io.IOException: java.io.IOException: java.util.ConcurrentModificationException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294) at com.test.tools.hbase.schema.createIfNotExists(schema.java:520) at com.test.tools.hbase.schema.main(schema.java:627) Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.util.ConcurrentModificationException at java.util.SubList.checkForComodification(AbstractList.java:752) at java.util.SubList.add(AbstractList.java:632) at java.util.SubList.add(AbstractList.java:633) at java.util.SubList.add(AbstractList.java:633) .. .. at java.util.SubList.add(AbstractList.java:633) at java.util.AbstractList.add(AbstractList.java:91) at org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy5.createTable(Unknown Source) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HBASE-50) Snapshot of table
[ https://issues.apache.org/jira/browse/HBASE-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12906486#comment-12906486 ] Zhihong Yu edited comment on HBASE-50 at 3/29/12 7:02 PM: -- Message from: Chongxin Li lichong...@zju.edu.cn --- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/467/ --- (Updated 2010-09-06 04:34:53.459404) Review request for hbase. Changes --- Add Mapreduce based export (ExportSnapshot) and import (ImportSnapshot) for snapshot, so that snapshot of an hbase table could be exported and imported to other data centers. Unit test (TestSnapshotExport) has passed. Summary --- This patch includes the first three sub-tasks of HBASE-50: 1. Start and monitor the creation of snapshot via ZooKeeper 2. Create snapshot of an HBase table 3. Some existing functions of HBase are modified to support snapshot Currently snapshots can be created as expected, but can not be restored or deleted yet This addresses bug HBASE-50. http://issues.apache.org/jira/browse/HBASE-50 Diff: http://review.cloudera.org/r/467/diff Testing --- Unit tests and integration tests with mini cluster passed. Thanks, Chongxin was (Author: hbasereviewboard): Message from: Chongxin Li lichong...@zju.edu.cn --- This is an automatically generated e-mail. To reply, visit: http://review.cloudera.org/r/467/ --- (Updated 2010-09-06 04:34:53.459404) Review request for hbase. Changes --- Add Mapreduce based export (ExportSnapshot) and import (ImportSnapshot) for snapshot, so that snapshot of an hbase table could be exported and imported to other data centers. Unit test (TestSnapshotExport) has passed. Summary --- This patch includes the first three sub-tasks of HBASE-50: 1. Start and monitor the creation of snapshot via ZooKeeper 2. Create snapshot of an HBase table 3. Some existing functions of HBase are modified to support snapshot Currently snapshots can be created as expected, but can not be restored or deleted yet This addresses bug HBASE-50. http://issues.apache.org/jira/browse/HBASE-50 Diffs (updated) - bin/add_snapshot_family.rb PRE-CREATION src/main/java/org/apache/hadoop/hbase/HConstants.java bfaa4a1 src/main/java/org/apache/hadoop/hbase/HRegionInfo.java ee94690 src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java 0d57270 src/main/java/org/apache/hadoop/hbase/SnapshotDescriptor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/SnapshotExistsException.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/TablePartiallyOpenException.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 8b01aa0 src/main/java/org/apache/hadoop/hbase/io/HalfStoreFileReader.java d35a28a src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java 20860d6 src/main/java/org/apache/hadoop/hbase/io/Reference.java 219203c src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPCProtocolVersion.java d4bcbed src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java bd48a4b src/main/java/org/apache/hadoop/hbase/mapreduce/ExportSnapshot.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/mapreduce/ImportSnapshot.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 1183584 src/main/java/org/apache/hadoop/hbase/master/BaseScanner.java 2deea4a src/main/java/org/apache/hadoop/hbase/master/DeleteSnapshot.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/HMaster.java 4735304 src/main/java/org/apache/hadoop/hbase/master/LogsCleaner.java 9d1a8b8 src/main/java/org/apache/hadoop/hbase/master/RestoreSnapshot.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/SnapshotLogCleaner.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/SnapshotMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/SnapshotOperation.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/SnapshotSentinel.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/TableDelete.java 1153e62 src/main/java/org/apache/hadoop/hbase/master/TableSnapshot.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9fdd86d src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8356d64 src/main/java/org/apache/hadoop/hbase/regionserver/Snapshotter.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java f1d52b7 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java ae9e190
[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status
[ https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241518#comment-13241518 ] stack commented on HBASE-5675: -- This is still an issue then? Create table fails if we keep refreshing master's UI for task monitor status Key: HBASE-5675 URL: https://issues.apache.org/jira/browse/HBASE-5675 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Mubarak Seyed Assignee: Mubarak Seyed Labels: noob I tried to create a table with 2K pre-split regions, region assignment was in middle and i was keep refreshing master's web UI to find the status of the task using task monitor, table creation was failed and {{META}} was showing 2K regions with server location value is {{null}} and regions weren't deployed onto region-servers. {code} table_ACreating table table_A java.io.IOException: java.io.IOException: java.util.ConcurrentModificationException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294) at com.test.tools.hbase.schema.createIfNotExists(schema.java:520) at com.test.tools.hbase.schema.main(schema.java:627) Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.util.ConcurrentModificationException at java.util.SubList.checkForComodification(AbstractList.java:752) at java.util.SubList.add(AbstractList.java:632) at java.util.SubList.add(AbstractList.java:633) at java.util.SubList.add(AbstractList.java:633) .. .. at java.util.SubList.add(AbstractList.java:633) at java.util.AbstractList.add(AbstractList.java:91) at org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy5.createTable(Unknown Source) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241524#comment-13241524 ] Hadoop QA commented on HBASE-5638: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520466/HBASE-5638-trunk-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.mapreduce.TestImportTsv Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1345//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1345//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1345//console This message is automatically generated. Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5675) Create table fails if we keep refreshing master's UI for task monitor status
[ https://issues.apache.org/jira/browse/HBASE-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241531#comment-13241531 ] Andrew Purtell commented on HBASE-5675: --- We saw this CME in our internal distro but solved it by bringing forward TaskMonitor to the latest in 0.92. Create table fails if we keep refreshing master's UI for task monitor status Key: HBASE-5675 URL: https://issues.apache.org/jira/browse/HBASE-5675 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4, 0.92.0, 0.94.0 Reporter: Mubarak Seyed Assignee: Mubarak Seyed Labels: noob I tried to create a table with 2K pre-split regions, region assignment was in middle and i was keep refreshing master's web UI to find the status of the task using task monitor, table creation was failed and {{META}} was showing 2K regions with server location value is {{null}} and regions weren't deployed onto region-servers. {code} table_ACreating table table_A java.io.IOException: java.io.IOException: java.util.ConcurrentModificationException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:384) at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:294) at com.test.tools.hbase.schema.createIfNotExists(schema.java:520) at com.test.tools.hbase.schema.main(schema.java:627) Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.util.ConcurrentModificationException at java.util.SubList.checkForComodification(AbstractList.java:752) at java.util.SubList.add(AbstractList.java:632) at java.util.SubList.add(AbstractList.java:633) at java.util.SubList.add(AbstractList.java:633) .. .. at java.util.SubList.add(AbstractList.java:633) at java.util.AbstractList.add(AbstractList.java:91) at org.apache.hadoop.hbase.monitoring.TaskMonitor.createStatus(TaskMonitor.java:76) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:510) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:490) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:853) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:813) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:780) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:771) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) at $Proxy5.createTable(Unknown Source) at org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsync(HBaseAdmin.java:382) {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241530#comment-13241530 ] stack commented on HBASE-5638: -- lgtm Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-trunk-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5655) Cap space usage of default log4j rolling policy
[ https://issues.apache.org/jira/browse/HBASE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241536#comment-13241536 ] stack commented on HBASE-5655: -- Did you intend to change this H? {code} -hbase.root.logger=INFO,console +hbase.root.logger=DEBUG,console {code} I think you should add more comment in hbase-env.sh suggesting DRFA if folks want the old behavior (though it has issues -- point to this issue?). Good stuff Cap space usage of default log4j rolling policy --- Key: HBASE-5655 URL: https://issues.apache.org/jira/browse/HBASE-5655 Project: HBase Issue Type: Improvement Affects Versions: 0.92.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Attachments: 5655-v1.patch The current default log4j policy is to use Daily Rolling File Appender (DRFA). At times, its good to have a cap on the maximum size of the logs in order to limit its disk usage. Here is a proposal to set a new file appemder (RFA) as the default appender. It can be configured via env so that existing tools can use the current behavior of using DRFA instead. This is in parallel with jira Hadoop-8149. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5674) add support in HBase to overwrite hbase timestamp to a version number during major compaction
[ https://issues.apache.org/jira/browse/HBASE-5674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241544#comment-13241544 ] stack commented on HBASE-5674: -- bq. A millisecond timestamp is too heavy to carry. For whom? Can you not just have your client specify timestamp of 0? add support in HBase to overwrite hbase timestamp to a version number during major compaction - Key: HBASE-5674 URL: https://issues.apache.org/jira/browse/HBASE-5674 Project: HBase Issue Type: Improvement Reporter: He Yongqiang Assignee: He Yongqiang Right now, a millisecond-level timestamp is attached to every record. In our case, we only need a version number (mostly it will be just zero etc). A millisecond timestamp is too heavy to carry. We should add support to overwrite it to zero during major compaction. KVs before major compaction will remain using system timestamp. And this should be configurable, so that we should not mess up if the hbase timestamp is specified by application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: (was: HBASE-5638-trunk-v2.patch) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: HBASE-5638-trunk-v2.patch HBASE-5638-0.92-v2.patch HBASE-5638-0.90-v2.patch Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: HBASE-5638-trunk-v2.patch HBASE-5638-0.90-v2.patch HBASE-5638-0.92-v2.patch Improve the readability by explicit check against LOCAL or DISTRIBUTED constant Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5638: --- Attachment: (was: HBASE-5638-0.90-v2.patch) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table
[ https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241556#comment-13241556 ] stack commented on HBASE-5665: -- Or, is this a problem only with forced splits? It doesn't happen when we split 'naturally' because we'll check for references? Repeated split causes HRegionServer failures and breaks table -- Key: HBASE-5665 URL: https://issues.apache.org/jira/browse/HBASE-5665 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0, 0.92.1, 0.94.0, 0.96.0, 0.94.1 Reporter: Cosmin Lehene Assignee: Cosmin Lehene Priority: Blocker Attachments: HBASE-5665-0.92.patch Repeated splits on large tables (2 consecutive would suffice) will essentially break the table (and the cluster), unrecoverable. The regionserver doing the split dies and the master will get into an infinite loop trying to assign regions that seem to have the files missing from HDFS. The table can be disabled once. upon trying to re-enable it, it will remain in an intermediary state forever. I was able to reproduce this on a smaller table consistently. {code} hbase(main):030:0 (0..1).each{|x| put 't1', #{x}, 'f1:t', 'dd'} hbase(main):030:0 (0..1000).each{|x| split 't1', #{x*10}} {code} Running overlapping splits in parallel (e.g. #{x*10+1}, #{x*10+2}... ) will reproduce the issue almost instantly and consistently. {code} 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in META 2012-03-28 10:57:16,321 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1.. compaction_queue=(0:1), split_queue=10 2012-03-28 10:57:16,343 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 java.io.IOException: Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363) at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237 at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1008) at org.apache.hadoop.hbase.io.HalfStoreFileReader.init(HalfStoreFileReader.java:65) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548) at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284) at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:221) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511) at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229) at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504) at org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484) ... 1 more 2012-03-28 10:57:16,345 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ld2,60020,1332957343833: Abort; we got an error after point-of-no-return {code} http://hastebin.com/diqinibajo.avrasm later edit: (I'm using the last 4 characters from each string) Region 94e3 has
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241563#comment-13241563 ] stack commented on HBASE-5547: -- What Lars said. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241560#comment-13241560 ] stack commented on HBASE-5619: -- Can you fix this Jimmy? Happens when I try to build after applying the patch: {code} main: [exec] target/compile-proto.sh: line 12: protoc: command not found [exec] target/compile-proto.sh: line 12: protoc: command not found [exec] target/compile-proto.sh: line 12: protoc: command not found [INFO] [ERROR] BUILD ERROR [INFO] [INFO] An Ant BuildException has occured: exec returned: 127 {code} Create PB protocols for HRegionInterface Key: HBASE-5619 URL: https://issues.apache.org/jira/browse/HBASE-5619 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5619.patch, hbase-5619_v3.patch, hbase-5619_v4.patch Subtask of HBase-5443, separate HRegionInterface into admin protocol and client protocol, create the PB protocol buffer files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241565#comment-13241565 ] Jonathan Hsieh commented on HBASE-5638: --- Lgtm. I'll do a quick test and commit to 0.90/0.92/0.94/trunk branches. Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5097) RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE
[ https://issues.apache.org/jira/browse/HBASE-5097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241567#comment-13241567 ] Hudson commented on HBASE-5097: --- Integrated in HBase-0.94 #65 (See [https://builds.apache.org/job/HBase-0.94/65/]) HBASE-5097 RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE (Ram) (Revision 1307035) Result = SUCCESS ramkrishna : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java RegionObserver implementation whose preScannerOpen and postScannerOpen Impl return null can stall the system initialization through NPE --- Key: HBASE-5097 URL: https://issues.apache.org/jira/browse/HBASE-5097 Project: HBase Issue Type: Bug Components: coprocessors Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.92.2, 0.96.0, 0.94.1 Attachments: HBASE-5097.patch, HBASE-5097_1.patch, HBASE-5097_2.patch In HRegionServer.java openScanner() {code} r.prepareScanner(scan); RegionScanner s = null; if (r.getCoprocessorHost() != null) { s = r.getCoprocessorHost().preScannerOpen(scan); } if (s == null) { s = r.getScanner(scan); } if (r.getCoprocessorHost() != null) { s = r.getCoprocessorHost().postScannerOpen(scan, s); } {code} If we dont have implemention for postScannerOpen the RegionScanner is null and so throwing nullpointer {code} java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881) at org.apache.hadoop.hbase.regionserver.HRegionServer.addScanner(HRegionServer.java:2282) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2272) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) {code} Making this defect as blocker.. Pls feel free to change the priority if am wrong. Also correct me if my way of trying out coprocessors without implementing postScannerOpen is wrong. Am just a learner. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table
[ https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241586#comment-13241586 ] Cosmin Lehene commented on HBASE-5665: -- Indeed it seems to be a problem with forced splits. I'm not sure though if the natural splits are safe - they seem to be, but I need to test that too. RegionSplitPolicy.getSplitPoint() calls Store.getSplitPoint() Store.getSplitPoint seems to do the check. {code} for (StoreFile sf : storefiles) { if (sf.isReference()) { // Should already be enforced since we return false in this case assert false : getSplitPoint() called on a region that can't split!; return null; } {code} BTW, we also have Store.hasReferences() {code} private boolean hasReferences(CollectionStoreFile files) { if (files != null files.size() 0) { for (StoreFile hsf: files) { if (hsf.isReference()) { return true; } } } return false; } {code} However here's the code in HRegion.checkSplit() If there's an explicit split point it won't get to do the reference check. {code} public byte[] checkSplit() { // Can't split META if (getRegionInfo().isMetaRegion()) { if (shouldForceSplit()) { LOG.warn(Cannot split meta regions in HBase 0.20 and above); } return null; } if (this.explicitSplitPoint != null) { return this.explicitSplitPoint; } if (!splitPolicy.shouldSplit()) { return null; } byte[] ret = splitPolicy.getSplitPoint(); if (ret != null) { try { checkRow(ret, calculated split); } catch (IOException e) { LOG.error(Ignoring invalid split, e); return null; } } return ret; } {code} Multiple return points + a ret variable - this could use some polishing too :) I'm a bit puzzled about the natural split, because, I've seen the problem with a forced split from UI where I don't think we provide an explicit split point. Cosmin Repeated split causes HRegionServer failures and breaks table -- Key: HBASE-5665 URL: https://issues.apache.org/jira/browse/HBASE-5665 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0, 0.92.1, 0.94.0, 0.96.0, 0.94.1 Reporter: Cosmin Lehene Assignee: Cosmin Lehene Priority: Blocker Attachments: HBASE-5665-0.92.patch Repeated splits on large tables (2 consecutive would suffice) will essentially break the table (and the cluster), unrecoverable. The regionserver doing the split dies and the master will get into an infinite loop trying to assign regions that seem to have the files missing from HDFS. The table can be disabled once. upon trying to re-enable it, it will remain in an intermediary state forever. I was able to reproduce this on a smaller table consistently. {code} hbase(main):030:0 (0..1).each{|x| put 't1', #{x}, 'f1:t', 'dd'} hbase(main):030:0 (0..1000).each{|x| split 't1', #{x*10}} {code} Running overlapping splits in parallel (e.g. #{x*10+1}, #{x*10+2}... ) will reproduce the issue almost instantly and consistently. {code} 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in META 2012-03-28 10:57:16,321 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1.. compaction_queue=(0:1), split_queue=10 2012-03-28 10:57:16,343 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 java.io.IOException: Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363) at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237 at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822) at
[jira] [Updated] (HBASE-5638) Backport to 0.90 and 0.92 - NPE reading ZK config in HBase
[ https://issues.apache.org/jira/browse/HBASE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5638: -- Resolution: Fixed Fix Version/s: 0.94.1 0.94.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Matteo, and thanks for review stack. Backport to 0.90 and 0.92 - NPE reading ZK config in HBase -- Key: HBASE-5638 URL: https://issues.apache.org/jira/browse/HBASE-5638 Project: HBase Issue Type: Sub-task Components: zookeeper Affects Versions: 0.90.6, 0.92.1 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.90.7, 0.92.2, 0.94.0, 0.94.1 Attachments: HBASE-5633-0.90.patch, HBASE-5633-0.92.patch, HBASE-5638-0.90-v1.patch, HBASE-5638-0.90-v2.patch, HBASE-5638-0.92-v1.patch, HBASE-5638-0.92-v2.patch, HBASE-5638-trunk-v1.patch, HBASE-5638-trunk-v2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table
[ https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cosmin Lehene updated HBASE-5665: - Affects Version/s: (was: 0.94.1) (was: 0.96.0) (was: 0.94.0) Status: Patch Available (was: Open) Repeated split causes HRegionServer failures and breaks table -- Key: HBASE-5665 URL: https://issues.apache.org/jira/browse/HBASE-5665 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.1, 0.92.0 Reporter: Cosmin Lehene Assignee: Cosmin Lehene Priority: Blocker Attachments: HBASE-5665-0.92.patch Repeated splits on large tables (2 consecutive would suffice) will essentially break the table (and the cluster), unrecoverable. The regionserver doing the split dies and the master will get into an infinite loop trying to assign regions that seem to have the files missing from HDFS. The table can be disabled once. upon trying to re-enable it, it will remain in an intermediary state forever. I was able to reproduce this on a smaller table consistently. {code} hbase(main):030:0 (0..1).each{|x| put 't1', #{x}, 'f1:t', 'dd'} hbase(main):030:0 (0..1000).each{|x| split 't1', #{x*10}} {code} Running overlapping splits in parallel (e.g. #{x*10+1}, #{x*10+2}... ) will reproduce the issue almost instantly and consistently. {code} 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in META 2012-03-28 10:57:16,321 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1.. compaction_queue=(0:1), split_queue=10 2012-03-28 10:57:16,343 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 java.io.IOException: Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363) at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237 at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1008) at org.apache.hadoop.hbase.io.HalfStoreFileReader.init(HalfStoreFileReader.java:65) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548) at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284) at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:221) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511) at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229) at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504) at org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484) ... 1 more 2012-03-28 10:57:16,345 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ld2,60020,1332957343833: Abort; we got an error after point-of-no-return {code} http://hastebin.com/diqinibajo.avrasm later edit: (I'm using the last 4 characters from each string) Region 94e3 has storefile 7237 Region 94e3 gets
[jira] [Commented] (HBASE-5665) Repeated split causes HRegionServer failures and breaks table
[ https://issues.apache.org/jira/browse/HBASE-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241593#comment-13241593 ] Cosmin Lehene commented on HBASE-5665: -- BTW - I don't think getSplitPoint should do that check, and we also shouldn't have to places where we check for references - perhaps we should have another JIRA to fix this in trunk? Repeated split causes HRegionServer failures and breaks table -- Key: HBASE-5665 URL: https://issues.apache.org/jira/browse/HBASE-5665 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.92.0, 0.92.1 Reporter: Cosmin Lehene Assignee: Cosmin Lehene Priority: Blocker Attachments: HBASE-5665-0.92.patch Repeated splits on large tables (2 consecutive would suffice) will essentially break the table (and the cluster), unrecoverable. The regionserver doing the split dies and the master will get into an infinite loop trying to assign regions that seem to have the files missing from HDFS. The table can be disabled once. upon trying to re-enable it, it will remain in an intermediary state forever. I was able to reproduce this on a smaller table consistently. {code} hbase(main):030:0 (0..1).each{|x| put 't1', #{x}, 'f1:t', 'dd'} hbase(main):030:0 (0..1000).each{|x| split 't1', #{x*10}} {code} Running overlapping splits in parallel (e.g. #{x*10+1}, #{x*10+2}... ) will reproduce the issue almost instantly and consistently. {code} 2012-03-28 10:57:16,320 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Offlined parent region t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1. in META 2012-03-28 10:57:16,321 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for t1,5,1332957435767.648d30de55a5cec6fc2f56dcb3c7eee1.. compaction_queue=(0:1), split_queue=10 2012-03-28 10:57:16,343 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup of failed split of t1,,1332957435767.2fb0473f4e71339e88dab0ee0d4dffa1.; Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 java.io.IOException: Failed ld2,60020,1332957343833-daughterOpener=2469c5650ea2aeed631eb85d3cdc3124 at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:363) at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:451) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.FileNotFoundException: File does not exist: /hbase/t1/589c44cabba419c6ad8c9b427e5894e3.2fb0473f4e71339e88dab0ee0d4dffa1/f1/d62a852c25ad44e09518e102ca557237 at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1822) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.init(DFSClient.java:1813) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:544) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:187) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:456) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:341) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.init(StoreFile.java:1008) at org.apache.hadoop.hbase.io.HalfStoreFileReader.init(HalfStoreFileReader.java:65) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:467) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:548) at org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:284) at org.apache.hadoop.hbase.regionserver.Store.init(Store.java:221) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:2511) at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:450) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3229) at org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:504) at org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:484) ... 1 more 2012-03-28 10:57:16,345 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server ld2,60020,1332957343833: Abort; we got an error after point-of-no-return {code} http://hastebin.com/diqinibajo.avrasm later edit: (I'm using the last 4
[jira] [Commented] (HBASE-5619) Create PB protocols for HRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241594#comment-13241594 ] Zhihong Yu commented on HBASE-5619: --- @Stack: Was protoc installed on the box you performed the build. I did a clean build after applying patch v4 and didn't see the above error. Create PB protocols for HRegionInterface Key: HBASE-5619 URL: https://issues.apache.org/jira/browse/HBASE-5619 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5619.patch, hbase-5619_v3.patch, hbase-5619_v4.patch Subtask of HBase-5443, separate HRegionInterface into admin protocol and client protocol, create the PB protocol buffer files -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13241600#comment-13241600 ] jirapos...@reviews.apache.org commented on HBASE-4348: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4402/#review6537 --- Ship it! lgtm. I'll fix the spacing nits as I commit. src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java https://reviews.apache.org/r/4402/#comment14199 nit: spacing src/main/java/org/apache/hadoop/hbase/master/HMaster.java https://reviews.apache.org/r/4402/#comment14200 spacing src/main/java/org/apache/hadoop/hbase/master/HMaster.java https://reviews.apache.org/r/4402/#comment14201 spacing - jmhsieh On 2012-03-29 15:57:19, Himanshu Vashishtha wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4402/ bq. --- bq. bq. (Updated 2012-03-29 15:57:19) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This patch is for adding Region in transition metrics to the HMaster metrics system. It also adds these metrics in the master ui, in the Region in transition section. I have attached the proposed new format in the jira 4348. bq. bq. bq. This addresses bug HBase-4348. bq. https://issues.apache.org/jira/browse/HBase-4348 bq. bq. bq. Diffs bq. - bq. bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon 0dc0691 bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 19be4de bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java ae468ca bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java c4b4d30 bq.src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 83abc52 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java d68ce33 bq. bq. Diff: https://reviews.apache.org/r/4402/diff bq. bq. bq. Testing bq. --- bq. bq. Ran on a 5 node cluster and kill region servers randomly to observe the changes in the RIT metrics as emitted out by the Master's mxbean; bq. bq. mvn test passes without any failure. bq. bq. bq. Thanks, bq. bq. Himanshu bq. bq. Add metrics for regions in transition - Key: HBASE-4348 URL: https://issues.apache.org/jira/browse/HBASE-4348 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Himanshu Vashishtha Priority: Minor Labels: noob Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, RITs.png, RegionInTransitions2.png, metrics-v2.patch The following metrics would be useful for monitoring the master: - the number of regions in transition - the number of regions in transition that have been in transition for more than a minute - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira