[jira] [Commented] (HBASE-5615) the master never does balance because of balancing the parent region
[ https://issues.apache.org/jira/browse/HBASE-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249696#comment-13249696 ] ramkrishna.s.vasudevan commented on HBASE-5615: --- @Lars Sorry for not able to comment on this issue over the weekend. The issue is present in 0.92+ but the fix is not this. We may have to work on the correct fix for this. the master never does balance because of balancing the parent region Key: HBASE-5615 URL: https://issues.apache.org/jira/browse/HBASE-5615 Project: HBase Issue Type: Bug Affects Versions: 0.90.7 Reporter: xufeng Assignee: xufeng Priority: Critical Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 Attachments: 5615-trunk.txt, HBASE-5615-90.patch, HBASE-5615.patch, NoPatched-surefire-report-5615-90.html, Patched_surefire-report-5615-90.html the master never do balance becauseof when master do rebuildUserRegions()，it will add the parent region into AssignmentManager#servers, if balancer let the parent region to move,the parent will in RIT forever.thus balance will never be executed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5744: --- Attachment: D2685.2.patch mbautin updated the revision [jira] [HBASE-5744] Thrift server metrics should be long instead of int. Reviewers: stack, sc, Kannan, JIRA Removing the old unused overload of verifyMetrics. REVISION DETAIL https://reviews.facebook.net/D2685 AFFECTED FILES src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch, D2685.1.patch, D2685.2.patch, jira-HBASE-5744-89-fb-Thrift-server-metrics-should-b-2012-04-07_21_39_35.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HBASE-5744: --- Attachment: D2685.3.patch mbautin updated the revision [jira] [HBASE-5744] Thrift server metrics should be long instead of int. Reviewers: stack, sc, Kannan, JIRA Submitted incomplete diff last time, adding other changes. REVISION DETAIL https://reviews.facebook.net/D2685 AFFECTED FILES src/main/java/org/apache/hadoop/hbase/thrift/HbaseHandlerMetricsProxy.java src/main/java/org/apache/hadoop/hbase/thrift/ThriftMetrics.java src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch, D2685.1.patch, D2685.2.patch, D2685.3.patch, jira-HBASE-5744-89-fb-Thrift-server-metrics-should-b-2012-04-07_21_39_35.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249744#comment-13249744 ] Hadoop QA commented on HBASE-5744: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521924/D2685.3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1452//console This message is automatically generated. Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch, D2685.1.patch, D2685.2.patch, D2685.3.patch, jira-HBASE-5744-89-fb-Thrift-server-metrics-should-b-2012-04-07_21_39_35.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fulin wang updated HBASE-5599: -- Attachment: 0.90-surefire-report-hbck.html hbase-5599-trunk_v7.patch hbase-5599-0.90_v7.patch 1. I modify the hbck code before the six error and nine skippe, the 0.90-surefire-report-hbck.html file is my test result for my changed. 2. This is a trunk version test result. --- Test set: org.apache.hadoop.hbase.util.TestHBaseFsck --- Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 197.475 sec The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. Key: HBASE-5599 URL: https://issues.apache.org/jira/browse/HBASE-5599 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.90.6 Reporter: fulin wang Assignee: fulin wang Fix For: 0.90.6 Attachments: 0.90-surefire-report-hbck.html, hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, hbase-5599-0.90_v6.patch, hbase-5599-0.90_v7.patch, hbase-5599-0.92_v5.patch, hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch, hbase-5599-trunk_v7.patch The hbck tool can not fix the six scenarios. 1. Version file does not exist in root dir. Fix: I try to create a version file by 'FSUtils.setVersion' method. 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any region server. Fix: I get region info form the hdfs file, this region info write to '.META.' table. 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] Fix: I get region info form the hdfs file, this region info write to '.META.' table. 4. [REGIONNAME] should not be deployed according to META, but is deployed on [SERVERNAME] Fix: Close this region. 5. First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this error. 6. There is a hole in the region chain between [KEY] and [KEY]. You need to create a new regioninfo and region dir in hdfs to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this hole. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5653) [findbugs] fix perf warnings
[ https://issues.apache.org/jira/browse/HBASE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249749#comment-13249749 ] Uma Maheswara Rao G commented on HBASE-5653: @Jon, please take a look, when you get some time. [findbugs] fix perf warnings Key: HBASE-5653 URL: https://issues.apache.org/jira/browse/HBASE-5653 Project: HBase Issue Type: Sub-task Components: scripts Reporter: Jonathan Hsieh Assignee: Uma Maheswara Rao G Attachments: HBASE-5653.patch See https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html#Warnings_PERFORMANCE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5677) The master never does balance because duplicate openhandled the one region
[ https://issues.apache.org/jira/browse/HBASE-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249753#comment-13249753 ] xufeng commented on HBASE-5677: --- @Ted I test the 0.92 in my cluster by reproduce steps. then I run the hbck tool to check the health of cluster and found many multiply error. I think it also has problem in 0.92. The master never does balance because duplicate openhandled the one region -- Key: HBASE-5677 URL: https://issues.apache.org/jira/browse/HBASE-5677 Project: HBase Issue Type: Bug Affects Versions: 0.90.6 Environment: 0.90 Reporter: xufeng Assignee: xufeng If region be assigned When the master is doing initialization(before do processFailover),the region will be duplicate openhandled. because the unassigned node in zookeeper will be handled again in AssignmentManager#processFailover() it cause the region in RIT,thus the master never does balance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5677) The master never does balance because duplicate openhandled the one region
[ https://issues.apache.org/jira/browse/HBASE-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249754#comment-13249754 ] xufeng commented on HBASE-5677: --- I got lasted 0.92 version(revision 1311105) from https://svn.apache.org/repos/asf/hbase/branches/0.92 then compiled it. The master never does balance because duplicate openhandled the one region -- Key: HBASE-5677 URL: https://issues.apache.org/jira/browse/HBASE-5677 Project: HBase Issue Type: Bug Affects Versions: 0.90.6 Environment: 0.90 Reporter: xufeng Assignee: xufeng If region be assigned When the master is doing initialization(before do processFailover),the region will be duplicate openhandled. because the unassigned node in zookeeper will be handled again in AssignmentManager#processFailover() it cause the region in RIT,thus the master never does balance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249756#comment-13249756 ] Hudson commented on HBASE-5744: --- Integrated in HBase-TRUNK #2729 (See [https://builds.apache.org/job/HBase-TRUNK/2729/]) [jira] [HBASE-5744] [89-fb] Thrift server metrics should be long instead of int Summary: As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. There is a bug where we truncate a nanosecond latency to int, which is a problem with RPCs that take more than 2.147483647 seconds to process. Test Plan: TestThriftServer is updated to test for the failure case (an RPC is artificially made to take 3 seconds). The new test case fails without the fix. Re-run all unit tests. Reviewers: sc, dhruba, Kannan, Liyin, JIRA Reviewed By: sc CC: stack Differential Revision: https://reviews.facebook.net/D2679 (Revision 1311167) Result = SUCCESS mbautin : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/HbaseHandlerMetricsProxy.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/ThriftMetrics.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch, D2685.1.patch, D2685.2.patch, D2685.3.patch, jira-HBASE-5744-89-fb-Thrift-server-metrics-should-b-2012-04-07_21_39_35.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-1936) ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services
[ https://issues.apache.org/jira/browse/HBASE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Ploeg reassigned HBASE-1936: --- Assignee: (was: Daniel Ploeg) ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services --- Key: HBASE-1936 URL: https://issues.apache.org/jira/browse/HBASE-1936 Project: HBase Issue Type: New Feature Reporter: stack Labels: noob Attachments: cp_from_hdfs.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5599) The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_
[ https://issues.apache.org/jira/browse/HBASE-5599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249767#comment-13249767 ] Hadoop QA commented on HBASE-5599: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521930/hbase-5599-trunk_v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestLogRolling Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1453//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1453//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1453//console This message is automatically generated. The hbck tool can not fix the six scenarios, it is NO_VERSION_FILE, NOT_IN_META_OR_DEPLOYED, NOT_IN_META, SHOULD_NOT_BE_DEPLOYED, FIRST_REGION_STARTKEY_NOT_EMPTY, HOLE_IN_REGION_CHAIN. Key: HBASE-5599 URL: https://issues.apache.org/jira/browse/HBASE-5599 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.90.6 Reporter: fulin wang Assignee: fulin wang Fix For: 0.90.6 Attachments: 0.90-surefire-report-hbck.html, hbase-5599-0.90.patch, hbase-5599-0.90_v2.patch, hbase-5599-0.90_v3.patch, hbase-5599-0.90_v5.patch, hbase-5599-0.90_v6.patch, hbase-5599-0.90_v7.patch, hbase-5599-0.92_v5.patch, hbase-5599-0.94_v5.patch, hbase-5599-trunk_v5.patch, hbase-5599-trunk_v7.patch The hbck tool can not fix the six scenarios. 1. Version file does not exist in root dir. Fix: I try to create a version file by 'FSUtils.setVersion' method. 2. [REGIONNAME][KEY] on HDFS, but not listed in META or deployed on any region server. Fix: I get region info form the hdfs file, this region info write to '.META.' table. 3. [REGIONNAME][KEY] not in META, but deployed on [SERVERNAME] Fix: I get region info form the hdfs file, this region info write to '.META.' table. 4. [REGIONNAME] should not be deployed according to META, but is deployed on [SERVERNAME] Fix: Close this region. 5. First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this error. 6. There is a hole in the region chain between [KEY] and [KEY]. You need to create a new regioninfo and region dir in hdfs to plug the hole. Fix: The region info is not in hdfs and .META., so it create a empty region for this hole. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5677) The master never does balance because duplicate openhandled the one region
[
https://issues.apache.org/jira/browse/HBASE-5677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249821#comment-13249821
]
Zhihong Yu commented on HBASE-5677:
---
There is this method in HMasterInterface:
{code}
/** @return true if master is available */
public boolean isMasterRunning();
{code}
If we introduce isMasterAvailable(), that would create confusion, right ?
The master never does balance because duplicate openhandled the one region
--
Key: HBASE-5677
URL: https://issues.apache.org/jira/browse/HBASE-5677
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.6
Environment: 0.90
Reporter: xufeng
Assignee: xufeng
If region be assigned When the master is doing initialization(before do
processFailover),the region will be duplicate openhandled.
because the unassigned node in zookeeper will be handled again in
AssignmentManager#processFailover()
it cause the region in RIT,thus the master never does balance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5726) TestSplitTransactionOnCluster occasionally failing
[
https://issues.apache.org/jira/browse/HBASE-5726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HBASE-5726:
---
Attachment:
Hbase.log_testRSSplitEphemeralsDisappearButDaughtersAreOnlinedAfterShutdownHandling
Hbase.log_testExistingZnodeBlocksSplitAndWeRollback
testShutdownFixupWhenDaughterHasSplit
TestSplitTransactionOnCluster occasionally failing
--
Key: HBASE-5726
URL: https://issues.apache.org/jira/browse/HBASE-5726
Project: HBase
Issue Type: Bug
Reporter: Uma Maheswara Rao G
Attachments: Hbase.log_testExistingZnodeBlocksSplitAndWeRollback
testShutdownFixupWhenDaughterHasSplit,
Hbase.log_testRSSplitEphemeralsDisappearButDaughtersAreOnlinedAfterShutdownHandling
When I ran TestSplitTransactionOnCluster, some times tests are failing.
{quote}
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.getAndCheckSingleTableRegion(TestSplitTransactionOnCluster.java:89)
at
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testShutdownFixupWhenDaughterHasSplit(TestSplitTransactionOnCluster.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:62)
{quote}
Seems like test is flaky, random other cases also fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249865#comment-13249865 ] stack commented on HBASE-5727: -- +1 on patch. On commit should add curlies around the log statement. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Attachments: 5727.1.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5748) Enable lib directory in jar file for coprocessor
Enable lib directory in jar file for coprocessor Key: HBASE-5748 URL: https://issues.apache.org/jira/browse/HBASE-5748 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.1, 0.94.0 Reporter: Takuya Ueshin Assignee: Takuya Ueshin Hadoop MapReduce job can use external libraries in 'lib' directory in the job.jar file. It is useful that jar files for coprocessor can use external libraries in the same way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-5727: --- Attachment: 5727.2.patch Ok here is the patch with the curlies around the LOG. I don't think we need to bother hudson since the only delta in this patch is the curly-brace. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249880#comment-13249880 ] Hudson commented on HBASE-5335: --- Integrated in HBase-TRUNK #2730 (See [https://builds.apache.org/job/HBase-TRUNK/2730/]) [jira] [HBASE-5335] Dynamic Schema Config Summary: Ability to add config options on a per-table per-cf basis Test Plan: - mvn test Reviewers: JIRA, Kannan, stack, mbautin, Liyin Reviewed By: mbautin CC: tedyu Differential Revision: https://reviews.facebook.net/D2247 (Revision 1311269) Result = SUCCESS nspiegelberg : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompoundConfiguration.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/trunk/src/main/ruby/hbase.rb * /hbase/trunk/src/main/ruby/hbase/admin.rb * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Fix For: 0.96.0 Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch, D2247.4.patch, D2247.5.patch, D2247.6.patch, D2247.7.patch, D2247.8.patch, HBASE-5335-trunk-2.patch, HBASE-5335-trunk-3.patch, HBASE-5335-trunk-3.patch, HBASE-5335-trunk-4.patch, HBASE-5335-trunk.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-1936) ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services
[ https://issues.apache.org/jira/browse/HBASE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jieshan Bean reassigned HBASE-1936: --- Assignee: Jieshan Bean ClassLoader that loads from hdfs; useful adding filters to classpath without having to restart services --- Key: HBASE-1936 URL: https://issues.apache.org/jira/browse/HBASE-1936 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Jieshan Bean Labels: noob Attachments: cp_from_hdfs.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5727: - Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks DD. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249890#comment-13249890 ] Tudor Scurtu commented on HBASE-5625: - @Zhihing, Stack: What do you think about the new changes? Tudor Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Fix For: 0.96.0 Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt, 5625v5.txt, 5625v6.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5748) Enable lib directory in jar file for coprocessor
[ https://issues.apache.org/jira/browse/HBASE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated HBASE-5748: - Attachment: HBASE-5748.patch I attatched a patch file. Could someone please review it or comment on this issue? Enable lib directory in jar file for coprocessor Key: HBASE-5748 URL: https://issues.apache.org/jira/browse/HBASE-5748 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.1, 0.94.0 Reporter: Takuya Ueshin Assignee: Takuya Ueshin Attachments: HBASE-5748.patch Hadoop MapReduce job can use external libraries in 'lib' directory in the job.jar file. It is useful that jar files for coprocessor can use external libraries in the same way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5748) Enable lib directory in jar file for coprocessor
[ https://issues.apache.org/jira/browse/HBASE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5748: - Hadoop Flags: Reviewed Status: Patch Available (was: Open) +1 on patch. Passing by hadoopqa. Enable lib directory in jar file for coprocessor Key: HBASE-5748 URL: https://issues.apache.org/jira/browse/HBASE-5748 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.1, 0.94.0 Reporter: Takuya Ueshin Assignee: Takuya Ueshin Attachments: HBASE-5748.patch Hadoop MapReduce job can use external libraries in 'lib' directory in the job.jar file. It is useful that jar files for coprocessor can use external libraries in the same way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5699) Run with 1 WAL in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249902#comment-13249902 ] stack commented on HBASE-5699: -- @binlijin What Chunhui says. I'd think that if it were a bigger cluster you'd see a more marked improvement. What about recovery? How does log splitting work with all the extra WALs? Run with 1 WAL in HRegionServer - Key: HBASE-5699 URL: https://issues.apache.org/jira/browse/HBASE-5699 Project: HBase Issue Type: Improvement Reporter: binlijin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249907#comment-13249907 ] Zhihong Yu commented on HBASE-5723: --- @Todd: Do you think the attached proposal belongs to HBASE-3340 ? Simple Design of Secondary Index Key: HBASE-5723 URL: https://issues.apache.org/jira/browse/HBASE-5723 Project: HBase Issue Type: New Feature Components: coprocessors Reporter: ShiXing Priority: Minor Attachments: Simple Design of HBase SecondaryIndex.pdf Use coprocessor to create index. And primary tables' compaction to purge the stale data. Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249908#comment-13249908 ] stack commented on HBASE-4336: -- Is there will to do this for 0.96? Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5666) RegionServer doesn't retry to check if base node is available
[
https://issues.apache.org/jira/browse/HBASE-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-5666:
---
Affects Version/s: 0.96.0
0.94.0
0.92.1
RegionServer doesn't retry to check if base node is available
-
Key: HBASE-5666
URL: https://issues.apache.org/jira/browse/HBASE-5666
Project: HBase
Issue Type: Bug
Components: regionserver, zookeeper
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Attachments: HBASE-5666-v1.patch, HBASE-5666-v2.patch,
HBASE-5666-v3.patch, HBASE-5666-v4.patch, HBASE-5666-v5.patch,
HBASE-5666-v6.patch, hbase-1-regionserver.log, hbase-2-regionserver.log,
hbase-3-regionserver.log, hbase-master.log, hbase-regionserver.log,
hbase-zookeeper.log
I've a script that starts hbase and a couple of region servers in distributed
mode (hbase.cluster.distributed = true)
{code}
$HBASE_HOME/bin/start-hbase.sh
$HBASE_HOME/bin/local-regionservers.sh start 1 2 3
{code}
but the region servers are not able to start...
It seems that during the RS start the the znode is still not available, and
HRegionServer.initializeZooKeeper() check just once if the base not is
available.
{code}
2012-03-28 21:54:05,013 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Check the value
configured in 'zookeeper.znode.parent'. There could be a mismatch with the
one configured in the master.
2012-03-28 21:54:08,598 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
localhost,60202,133296824: Initialization of RS failed. Hence aborting
RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:626)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:596)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:558)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5604) HLog replay tool that generates HFiles for use by LoadIncrementalHFiles.
[ https://issues.apache.org/jira/browse/HBASE-5604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249910#comment-13249910 ] stack commented on HBASE-5604: -- All sounds good Lars. HLog replay tool that generates HFiles for use by LoadIncrementalHFiles. Key: HBASE-5604 URL: https://issues.apache.org/jira/browse/HBASE-5604 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Attachments: 5604-v4.txt, 5604-v6.txt, HLog-5604-v3.txt Just an idea I had. Might be useful for restore of a backup using the HLogs. This could an M/R (with a mapper per HLog file). The tool would get a timerange and a (set of) table(s). We'd pick the right HLogs based on time before the M/R job is started and then have a mapper per HLog file. The mapper would then go through the HLog, filter all WALEdits that didn't fit into the time range or are not any of the tables and then uses HFileOutputFormat to generate HFiles. Would need to indicate the splits we want, probably from a live table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249915#comment-13249915 ] Jesse Yates commented on HBASE-4336: I'm giving up the torch on this one, if someone wants to pick it up. The combination of having other, more pressing stuff and this being a _total nightmare_ given the dependency interweavings makes it really hard (read: lots of time, but theoretically doable). I could spend the time if we just wanted to make two modules - main and security since those are fairly well separated. Otherwise, I've gotta pass :-/ Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249917#comment-13249917 ] Hadoop QA commented on HBASE-5727: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521965/5727.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1454//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1454//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1454//console This message is automatically generated. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5748) Enable lib directory in jar file for coprocessor
[ https://issues.apache.org/jira/browse/HBASE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5748: -- Fix Version/s: 0.96.0 0.94.0 Enable lib directory in jar file for coprocessor Key: HBASE-5748 URL: https://issues.apache.org/jira/browse/HBASE-5748 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.1, 0.94.0 Reporter: Takuya Ueshin Assignee: Takuya Ueshin Fix For: 0.94.0, 0.96.0 Attachments: HBASE-5748.patch Hadoop MapReduce job can use external libraries in 'lib' directory in the job.jar file. It is useful that jar files for coprocessor can use external libraries in the same way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5723) Simple Design of Secondary Index
[ https://issues.apache.org/jira/browse/HBASE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249927#comment-13249927 ] Todd Lipcon commented on HBASE-5723: This seems to give even fewer guarantees than HBASE-3340 (which as I understood it used the WAL to ensure reasonably timely consistency of the index table). I'm still of the opinion that these things should start as separate projects, rather than inside HBase core, and only if we see a high demand for them and good maintenance, we should pull it in. I don't think there is much demand for non-consistent indexes. Simple Design of Secondary Index Key: HBASE-5723 URL: https://issues.apache.org/jira/browse/HBASE-5723 Project: HBase Issue Type: New Feature Components: coprocessors Reporter: ShiXing Priority: Minor Attachments: Simple Design of HBase SecondaryIndex.pdf Use coprocessor to create index. And primary tables' compaction to purge the stale data. Attach file is the Design of the Seconday Index. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249928#comment-13249928 ] stack commented on HBASE-4336: -- @Jesse No problem. Yeah, it could be the horrors but it might not be too bad if we just put the modules in place and gradually migrate code. Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249931#comment-13249931 ] Hudson commented on HBASE-5727: --- Integrated in HBase-TRUNK #2731 (See [https://builds.apache.org/job/HBase-TRUNK/2731/]) HBASE-5727 secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' (Revision 1311287) Result = SUCCESS stack : Files : * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249932#comment-13249932 ] Hudson commented on HBASE-5451: --- Integrated in HBase-TRUNK #2731 (See [https://builds.apache.org/job/HBase-TRUNK/2731/]) HBASE-5727 secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' (Revision 1311287) Result = SUCCESS stack : Files : * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 5305v7.txt, 5305v7.txt, rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2, rpc-proto.r5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249935#comment-13249935 ] Hudson commented on HBASE-5451: --- Integrated in HBase-TRUNK-security #164 (See [https://builds.apache.org/job/HBase-TRUNK-security/164/]) HBASE-5727 secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' (Revision 1311287) Result = FAILURE stack : Files : * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 5305v7.txt, 5305v7.txt, rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2, rpc-proto.r5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249936#comment-13249936 ] Hudson commented on HBASE-5335: --- Integrated in HBase-TRUNK-security #164 (See [https://builds.apache.org/job/HBase-TRUNK-security/164/]) [jira] [HBASE-5335] Dynamic Schema Config Summary: Ability to add config options on a per-table per-cf basis Test Plan: - mvn test Reviewers: JIRA, Kannan, stack, mbautin, Liyin Reviewed By: mbautin CC: tedyu Differential Revision: https://reviews.facebook.net/D2247 (Revision 1311269) Result = FAILURE nspiegelberg : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/CompoundConfiguration.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/trunk/src/main/ruby/hbase.rb * /hbase/trunk/src/main/ruby/hbase/admin.rb * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/HBaseTestCase.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Fix For: 0.96.0 Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch, D2247.4.patch, D2247.5.patch, D2247.6.patch, D2247.7.patch, D2247.8.patch, HBASE-5335-trunk-2.patch, HBASE-5335-trunk-3.patch, HBASE-5335-trunk-3.patch, HBASE-5335-trunk-4.patch, HBASE-5335-trunk.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249934#comment-13249934 ] Hudson commented on HBASE-5727: --- Integrated in HBase-TRUNK-security #164 (See [https://builds.apache.org/job/HBase-TRUNK-security/164/]) HBASE-5727 secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' (Revision 1311287) Result = FAILURE stack : Files : * /hbase/trunk/security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249937#comment-13249937 ] Hudson commented on HBASE-5744: --- Integrated in HBase-TRUNK-security #164 (See [https://builds.apache.org/job/HBase-TRUNK-security/164/]) [jira] [HBASE-5744] [89-fb] Thrift server metrics should be long instead of int Summary: As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. There is a bug where we truncate a nanosecond latency to int, which is a problem with RPCs that take more than 2.147483647 seconds to process. Test Plan: TestThriftServer is updated to test for the failure case (an RPC is artificially made to take 3 seconds). The new test case fails without the fix. Re-run all unit tests. Reviewers: sc, dhruba, Kannan, Liyin, JIRA Reviewed By: sc CC: stack Differential Revision: https://reviews.facebook.net/D2679 (Revision 1311167) Result = FAILURE mbautin : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/HbaseHandlerMetricsProxy.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/thrift/ThriftMetrics.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch, D2685.1.patch, D2685.2.patch, D2685.3.patch, jira-HBASE-5744-89-fb-Thrift-server-metrics-should-b-2012-04-07_21_39_35.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[
https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu reopened HBASE-5727:
---
Compilation failed because unresolved conflict in
security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java:
{code}
.mine
final DataOutputBuffer d = new DataOutputBuffer();
{code}
secure hbase build broke because of 'HBASE-5451 Switch RPC call
envelope/headers to PBs'
Key: HBASE-5727
URL: https://issues.apache.org/jira/browse/HBASE-5727
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: Devaraj Das
Priority: Blocker
Fix For: 0.96.0
Attachments: 5727.1.patch, 5727.2.patch, 5727.patch
If you build with the security profile -- i.e. add '-P security' on the
command line -- you'll see that the secure build is broke since we messed in
rpc.
Assigning Deveraj to take a look. If you can't work on this now DD, just
give it back to me and I'll have a go at it. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[
https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Yu updated HBASE-5727:
--
Comment: was deleted
(was: Compilation failed because unresolved conflict in
security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java:
{code}
.mine
final DataOutputBuffer d = new DataOutputBuffer();
{code})
secure hbase build broke because of 'HBASE-5451 Switch RPC call
envelope/headers to PBs'
Key: HBASE-5727
URL: https://issues.apache.org/jira/browse/HBASE-5727
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: Devaraj Das
Priority: Blocker
Fix For: 0.96.0
Attachments: 5727.1.patch, 5727.2.patch, 5727.patch
If you build with the security profile -- i.e. add '-P security' on the
command line -- you'll see that the secure build is broke since we messed in
rpc.
Assigning Deveraj to take a look. If you can't work on this now DD, just
give it back to me and I'll have a go at it. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5740) Compaction interruption may be due to balacing
[ https://issues.apache.org/jira/browse/HBASE-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249951#comment-13249951 ] Jimmy Xiang commented on HBASE-5740: @JD, any comments on the second patch? Compaction interruption may be due to balacing -- Key: HBASE-5740 URL: https://issues.apache.org/jira/browse/HBASE-5740 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Trivial Fix For: 0.96.0 Attachments: hbase-5740.patch, hbase-5740_v2.patch Currently, the log shows Aborting compaction of store LOG in region because user requested stop. But it is actually because of balancing. Currently, there is no way to figure out who closed the region. So it is better to change the message to say it is because of user, or balancing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das resolved HBASE-5727. Resolution: Fixed I guess Zhihong forgot to 'resolve' the issue. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[
https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249954#comment-13249954
]
Zhihong Yu commented on HBASE-5727:
---
Nicolas checked in HBASE-5335 a little earlier than Stack checked in this patch.
So we now have the following compilation failure:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile (default-compile)
on project hbase: Compilation failure: Compilation failure:
[ERROR]
/Users/zhihyu/trunk-hbase/security/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java:[194,23]
cannot find symbol
[ERROR] symbol : method getConf()
[ERROR] location: class org.apache.hadoop.hbase.regionserver.HRegion
{code}
secure hbase build broke because of 'HBASE-5451 Switch RPC call
envelope/headers to PBs'
Key: HBASE-5727
URL: https://issues.apache.org/jira/browse/HBASE-5727
Project: HBase
Issue Type: Bug
Reporter: stack
Assignee: Devaraj Das
Priority: Blocker
Fix For: 0.96.0
Attachments: 5727.1.patch, 5727.2.patch, 5727.patch
If you build with the security profile -- i.e. add '-P security' on the
command line -- you'll see that the secure build is broke since we messed in
rpc.
Assigning Deveraj to take a look. If you can't work on this now DD, just
give it back to me and I'll have a go at it. Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5748) Enable lib directory in jar file for coprocessor
[ https://issues.apache.org/jira/browse/HBASE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249956#comment-13249956 ] Hadoop QA commented on HBASE-5748: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521967/HBASE-5748.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1455//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1455//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1455//console This message is automatically generated. Enable lib directory in jar file for coprocessor Key: HBASE-5748 URL: https://issues.apache.org/jira/browse/HBASE-5748 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.1, 0.94.0 Reporter: Takuya Ueshin Assignee: Takuya Ueshin Fix For: 0.94.0, 0.96.0 Attachments: HBASE-5748.patch Hadoop MapReduce job can use external libraries in 'lib' directory in the job.jar file. It is useful that jar files for coprocessor can use external libraries in the same way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu reopened HBASE-5727: --- The compilation issue is real. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5741) ImportTsv does not check for table existence
[ https://issues.apache.org/jira/browse/HBASE-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249966#comment-13249966 ] Clint Heath commented on HBASE-5741: Thanks for the feedback. We I have customers who are using the workflow you mentioned. This is why I discovered the confusing javadoc reference. ImportTsv does not check for table existence - Key: HBASE-5741 URL: https://issues.apache.org/jira/browse/HBASE-5741 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.90.4 Reporter: Clint Heath Assignee: Himanshu Vashishtha The usage statement for the importtsv command to hbase claims this: Note: if you do not use this option, then the target table must already exist in HBase (in reference to the importtsv.bulk.output command-line option) The truth is, the table must exist no matter what, importtsv cannot and will not create it for you. This is the case because the createSubmittableJob method of ImportTsv does not even attempt to check if the table exists already, much less create it: (From org.apache.hadoop.hbase.mapreduce.ImportTsv.java) 305 HTable table = new HTable(conf, tableName); The HTable method signature in use there assumes the table exists and runs a meta scan on it: (From org.apache.hadoop.hbase.client.HTable.java) 142 * Creates an object to access a HBase table. ... 151 public HTable(Configuration conf, final String tableName) What we should do inside of createSubmittableJob is something similar to what the completebulkloads command would do: (Taken from org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.java) 690 boolean tableExists = this.doesTableExist(tableName); 691 if (!tableExists) this.createTable(tableName,dirPath); Currently the docs are misleading, the table in fact must exist prior to running importtsv. We should check if it exists rather than assume it's already there and throw the below exception: 12/03/14 17:15:42 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch META table: org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for table: myTable2, row=myTable2,,99 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249967#comment-13249967 ] Devaraj Das commented on HBASE-5727: But this isn't related to the patch committed, right? The committed patch doesn't touch any of the affected files. Or, am i missing something? secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249968#comment-13249968 ] Zhihong Yu commented on HBASE-5727: --- @Devaraj: I should have explained in clearer terms. Since Nicolas removed the getConf() method in HBASE-5335, I would expect the patch in this JIRA to be rebased. Another option is to ask Nicolas to put the getConf() method back. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249975#comment-13249975 ] Devaraj Das commented on HBASE-5727: bq. Another option is to ask Nicolas to put the getConf() method back. Zhihong, that will be better IMHO, given that this patch didn't touch anything close to do with getConf(). In other words the build would have been broken even without the commit of the patch on this jira. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249980#comment-13249980 ] stack commented on HBASE-5727: -- Or just add it back w/ addendum on this patch. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249981#comment-13249981 ] stack commented on HBASE-5727: -- Sorry. Scratch the above suggestion. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5335) Dynamic Schema Configurations
[ https://issues.apache.org/jira/browse/HBASE-5335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249991#comment-13249991 ] Zhihong Yu commented on HBASE-5335: --- @Nicolas: Can you comment how we can resolve the compilation error seen in HBASE-5727 ? Dynamic Schema Configurations - Key: HBASE-5335 URL: https://issues.apache.org/jira/browse/HBASE-5335 Project: HBase Issue Type: New Feature Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Labels: configuration, schema Fix For: 0.96.0 Attachments: D2247.1.patch, D2247.2.patch, D2247.3.patch, D2247.4.patch, D2247.5.patch, D2247.6.patch, D2247.7.patch, D2247.8.patch, HBASE-5335-trunk-2.patch, HBASE-5335-trunk-3.patch, HBASE-5335-trunk-3.patch, HBASE-5335-trunk-4.patch, HBASE-5335-trunk.patch Currently, the ability for a core developer to add per-table per-CF configuration settings is very heavyweight. You need to add a reserved keyword all the way up the stack you have to support this variable long-term if you're going to expose it explicitly to the user. This has ended up with using Configuration.get() a lot because it is lightweight and you can tweak settings while you're trying to understand system behavior [since there are many config params that may never need to be tuned]. We need to add the ability to put read arbitrary KV settings in the HBase schema. Combined with online schema change, this will allow us to safely iterate on configuration settings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249994#comment-13249994 ] Zhihong Yu commented on HBASE-5727: --- I tend to agree with Devaraj in that HBASE-5335 should have gone through compilation against secure HBase. In AccessController.java, I don't see reference to HRegionServer through which the config can be obtained. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249998#comment-13249998 ] stack commented on HBASE-5727: -- @Ted No. Thats why we run a secure build up on jenkins to find those rare cases where a commit on insecure breaks secure. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250007#comment-13250007 ] Phabricator commented on HBASE-5744: mbautin has abandoned the revision [jira] [HBASE-5744] Thrift server metrics should be long instead of int. Unfortunately, I committed this diff under the 89-fb differential revision first (D2679) and had to edit the commit message. Here is what I did: svn propedit -r 1311167 --revprop svn:log The new commit message is as follows: svn log -r1311167 https://svn.apache.org/repos/asf/hbase/trunk r1311167 | mbautin | 2012-04-09 02:00:11 -0700 (Mon, 09 Apr 2012) | 23 lines [jira] [HBASE-5744] Thrift server metrics should be long instead of int Summary: As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. There is a bug where we truncate a nanosecond latency to int, which is a problem with RPCs that take more than 2.147483647 seconds to process. This is a trunk diff. The 89-fb version of this diff is at D2679. Test Plan: TestThriftServer is updated to test for the failure case (an RPC is artificially made to take 3 seconds). The new test case fails without the fix. Re-run all unit tests. Reviewers: stack, sc, Kannan, JIRA Reviewed By: sc Differential Revision: https://reviews.facebook.net/D2685 If you have a git-svn checkout that got confused by this, here is how to get it back to a consistent state: git svn reset 1311166 git svn rebase Sorry about the inconvenience. REVISION DETAIL https://reviews.facebook.net/D2685 Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch, D2685.1.patch, D2685.2.patch, D2685.3.patch, jira-HBASE-5744-89-fb-Thrift-server-metrics-should-b-2012-04-07_21_39_35.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5749) HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize
[ https://issues.apache.org/jira/browse/HBASE-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5749: - Attachment: heapsize.txt HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize -- Key: HBASE-5749 URL: https://issues.apache.org/jira/browse/HBASE-5749 Project: HBase Issue Type: Bug Reporter: stack Attachments: heapsize.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5750) HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize
[ https://issues.apache.org/jira/browse/HBASE-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5750: - Attachment: heapsize.txt HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize -- Key: HBASE-5750 URL: https://issues.apache.org/jira/browse/HBASE-5750 Project: HBase Issue Type: Bug Reporter: stack Attachments: heapsize.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250013#comment-13250013 ] Andrew Purtell commented on HBASE-5727: --- Removing getConf() was an action that had implications for all coprocessors; AccessController just exposed it by being the only thing in tree currently that uses it. We can unify the build by integrating RPC security in the 0.96 RPC singularity. These issues with secure versus nonsecure build should be as short a transitional phase as possible, i.e. 0.92 and 0.94 use it to bridge Hadoop core differences until 0.96 specifies Hadoop 1.0 or equivalent security shims as a minimum requirement. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250017#comment-13250017 ] Jesse Yates commented on HBASE-4336: @stack: yeah, I think that is a bit more reasonable. Now that I'm thinking a little more about it, I think an hbase-common, hbase-security and hbase-core packages would be a reasonable (and not overly consuming) step. Sets up the division between things and eventually might lead to splits between client and server (and maybe between RS and master?). I'll try to take a look this week at getting something simpler up. Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250023#comment-13250023 ] Enis Soztutar commented on HBASE-4336: -- bq. I think an hbase-common, hbase-security and hbase-core packages would be a reasonable +1 to that. At least getting the module structure in place will help us a lot. After that we can gradually attack client-server separation. Since 0.96 is the singularity, should we make client side module a blocker? It seems we might have to break BC for this as well. Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250024#comment-13250024 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/#review6764 --- Jimmy, Looks good to me. Mostly some questions and a little bit of of follow up to make it easier to read and to provide user more info. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment14916 nice catch. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment15130 Maybe this one is at info level? src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment14917 nit: typo (old was correct) src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment14918 Is this n per group or globally? please make it clear in usage src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment14919 nit: should be like previous -maxMerge's comments. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15129 To find? Would returning a sort make sense? if there are no overlaps will this return the specified number (or less?) Could this return several regions that now leave holes? nit: It might make testing clearer to return the overlapCount - regions map and check that. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15132 Add comment about structure contents. something like: Integer key is overlap count and List is regions that have that many overlaps. maybe rename to overlapRangeMap? src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15135 add comment: calculates the # of overlaps for each region and populates rangeOverlapMap src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15133 add comment? We always overlap with ourselves -- which is why 1 is the condition below.. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15136 what is the intuition here? src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15140 nit: it would easer to read if this was called ai, the next ae, and the one after that ac. Also would help if you put the expected overlap count in comments! - jmhsieh On 2012-04-06 18:27:34, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4649/ bq. --- bq. bq. (Updated 2012-04-06 18:27:34) bq. bq. bq. Review request for hbase and jmhsieh. bq. bq. bq. Summary bq. --- bq. bq. Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. bq. bq. The regions chose to sideline are those which overlap with most other regions. bq. bq. bq. This addresses bug HBASE-5719. bq. https://issues.apache.org/jira/browse/HBASE-5719 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 bq.src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd bq. src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java ac3b225 bq. bq. Diff: https://reviews.apache.org/r/4649/diff bq. bq. bq. Testing bq. --- bq. bq. mvn -PlocalTests -Dtest=TestHBaseFsck* clean test bq. bq. Also tested in real system to fix inconsistencies. bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250028#comment-13250028 ] Nicolas Spiegelberg commented on HBASE-5727: I agree with Stack, this isn't a common occurrence and is pretty trivial to fix. getConf() was, sadly, a hack API that was public because one of the tests wasn't in the same namespace originally. It shouldn't have been public because the config has traditionally been at a per-server granularity. Now, with HBASE-5335, the per-server config object can be per-server or per-region/per-CF depending upon how it's used and the functionality you want. Now: HRegion.conf != HRegionServer.conf != Store.conf. If your coprocessor uses the HRegionServer.conf, it should work the same as it always has. If your corprocessor uses HRegion.conf, a compilation error is better than a subtle runtime error that you have to spend a week hunting down. Of course, this is part of why we tell people they are playing with fire if they use a coprocessor (vs. MySQL triggers). Taking a look at the security code, it looks like you want to use HRegionServer.conf because both calling classes seem to access global security data. See HRegionServer.getConfiguration(), which is used in other sections of the AccessController class. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5750) HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize
[ https://issues.apache.org/jira/browse/HBASE-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250033#comment-13250033 ] Nicolas Spiegelberg commented on HBASE-5750: I'm interested in how this wasn't caught by Hadoop QA HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize -- Key: HBASE-5750 URL: https://issues.apache.org/jira/browse/HBASE-5750 Project: HBase Issue Type: Bug Reporter: stack Attachments: heapsize.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250031#comment-13250031 ] Jesse Yates commented on HBASE-4336: bq. BC Build compatibility? I don't think we necessarily need to break that - just because its in another module doesn't mean it changes the package. Should be fine :) Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250037#comment-13250037 ] Andrew Purtell commented on HBASE-5727: --- bq. If your coprocessor uses the HRegionServer.conf, it should work the same as it always has. If your corprocessor uses HRegion.conf, a compilation error is better than a subtle runtime error that you have to spend a week hunting down. Yes I agree. I didn't mean to imply it was an incorrect change in some way, just that the change had implications (obviously...) bq. Taking a look at the security code, it looks like you want to use HRegionServer.conf because both calling classes seem to access global security data. Yes. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250038#comment-13250038 ] Alejandro Abdelnur commented on HBASE-4336: --- @Jeese, having a client module with the minimum set of dependencies would a big thing for apps built on top of hbase. thxs Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250044#comment-13250044 ] Matt Corgan commented on HBASE-4336: Sometimes helps to think of modules as things that can be pulled out of the main project and turned into black boxes that just work. After you pull them out, the main project becomes simpler. Eventually, I think the HRegion is a good candidate for becoming a module. It's really a database on its own which will inevitably grow more and more complex. Then the purpose of the master and region servers are to coordinate and host bunches of HRegions without worrying what's going on inside the region. Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5751) hbase master stop does not bring down backup masters
[ https://issues.apache.org/jira/browse/HBASE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-5751: Assignee: Gregory Chanan hbase master stop does not bring down backup masters -- Key: HBASE-5751 URL: https://issues.apache.org/jira/browse/HBASE-5751 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Gregory Chanan Fix For: 0.90.7 Carry forward the discussion from parent for 0.90 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5727: -- Attachment: AccessController.diff The attached patch allows TestRowProcessorEndpoint to pass. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch, AccessController.diff If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5751) hbase master stop does not bring down backup masters
[ https://issues.apache.org/jira/browse/HBASE-5751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250056#comment-13250056 ] Lars Hofhansl commented on HBASE-5751: -- Tail of discussion from parent: stack added a comment - 06/Apr/12 15:03 This seems to have broken 0.90 builds. Can we revert from 0.90 trunk till fixed. See here: https://builds.apache.org/view/G-L/view/HBase/job/hbase-0.90/ Build #458 is where it got committed and thereafter the build fails rolling the wal test. David S. Wang added a comment - 06/Apr/12 15:18 Greg is out of town for the next week or so ... IMHO if it's breaking 0.90, we should revert for now and Greg can look at it when he gets back. Zhihong Yu added a comment - 06/Apr/12 16:11 TestLogRolling hangs in 0.90 Jonathan Hsieh added a comment - 06/Apr/12 16:21 I'll revert the 0.90 version. Sorry about this fellas. Lars Hofhansl added a comment - 09/Apr/12 03:31 So from the discussion the test is only a problem in 0.90? Do we know why this is? Asking because we're close to another RC attempt for 0.94.0. Jonathan Hsieh added a comment - 09/Apr/12 03:37 The issue is apparently only in 0.90. How about we close this issue for 0.92/0.94/trunk and create a follow on issue for 0.90? This will unblock this for 0.94 and Greg can address this when he gets back. hbase master stop does not bring down backup masters -- Key: HBASE-5751 URL: https://issues.apache.org/jira/browse/HBASE-5751 Project: HBase Issue Type: Sub-task Reporter: Lars Hofhansl Assignee: Gregory Chanan Fix For: 0.90.7 Carry forward the discussion from parent for 0.90 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250066#comment-13250066 ] Enis Soztutar commented on HBASE-4336: -- bq. BC BC = backwards compatibility. Sorry for the confusion. I assume that for separating the client side, we have to make some changes to core client APIs (like splitting the KeyValue, Bytes, ZK classes, etc). Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5615) the master never does balance because of balancing the parent region
[ https://issues.apache.org/jira/browse/HBASE-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250069#comment-13250069 ] Lars Hofhansl commented on HBASE-5615: -- Thanks Ram... I'll hold up 0.94 for this (unless you think that unnecessary). the master never does balance because of balancing the parent region Key: HBASE-5615 URL: https://issues.apache.org/jira/browse/HBASE-5615 Project: HBase Issue Type: Bug Affects Versions: 0.90.7 Reporter: xufeng Assignee: xufeng Priority: Critical Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 Attachments: 5615-trunk.txt, HBASE-5615-90.patch, HBASE-5615.patch, NoPatched-surefire-report-5615-90.html, Patched_surefire-report-5615-90.html the master never do balance becauseof when master do rebuildUserRegions()，it will add the parent region into AssignmentManager#servers, if balancer let the parent region to move,the parent will in RIT forever.thus balance will never be executed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250067#comment-13250067 ] stack commented on HBASE-4336: -- Following on from Matt, i see this issue in a new light of late as an aid to untangling our hairball. I propose that all thats needed to close this issue is a modularization that has one module only in it and that module is hbase-common. In subsequent issues we can break out other modules; compressors, hregion, client, security, server, leather or vinyl, etc. Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5750) HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize
[ https://issues.apache.org/jira/browse/HBASE-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250074#comment-13250074 ] Hadoop QA commented on HBASE-5750: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521976/heapsize.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1456//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1456//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1456//console This message is automatically generated. HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize -- Key: HBASE-5750 URL: https://issues.apache.org/jira/browse/HBASE-5750 Project: HBase Issue Type: Bug Reporter: stack Attachments: heapsize.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250077#comment-13250077 ] stack commented on HBASE-5727: -- Patch looks right to me; its providing the region context configuration making access determination to a particular region. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch, AccessController.diff If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250087#comment-13250087 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 1869 bq. https://reviews.apache.org/r/4649/diff/2/?file=100672#file100672line1869 bq. bq. Maybe this one is at info level? Will change to info level. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2828 bq. https://reviews.apache.org/r/4649/diff/2/?file=100672#file100672line2828 bq. bq. nit: typo (old was correct) Changed it back. It must be because the mouse moved when I was typing something. I didn't remember I changed it. :) bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2895 bq. https://reviews.apache.org/r/4649/diff/2/?file=100672#file100672line2895 bq. bq. Is this n per group or globally? please make it clear in usage It is per group. Will fix it. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2981 bq. https://reviews.apache.org/r/4649/diff/2/?file=100672#file100672line2981 bq. bq. nit: should be like previous -maxMerge's comments. Will fix it. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java, line 360 bq. https://reviews.apache.org/r/4649/diff/2/?file=100674#file100674line360 bq. bq. nit: it would easer to read if this was called ai, the next ae, and the one after that ac. bq. bq. Also would help if you put the expected overlap count in comments! Sure. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 212 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line212 bq. bq. what is the intuition here? Added comment bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 189 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line189 bq. bq. add comment? bq. bq. We always overlap with ourselves -- which is why 1 is the condition below.. Will do. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 178 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line178 bq. bq. add comment: calculates the # of overlaps for each region and populates rangeOverlapMap Sure bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 177 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line177 bq. bq. Add comment about structure contents. something like: Integer key is overlap count and List is regions that have that many overlaps. bq. maybe rename to overlapRangeMap? Sure. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 171 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line171 bq. bq. To find? Would returning a sort make sense? if there are no overlaps will this return the specified number (or less?) bq. bq. Could this return several regions that now leave holes? bq. bq. nit: It might make testing clearer to return the overlapCount - regions map and check that. Added comment. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/#review6764 --- On 2012-04-06 18:27:34, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4649/ bq. --- bq. bq. (Updated 2012-04-06 18:27:34) bq. bq. bq. Review request for hbase and jmhsieh. bq. bq. bq. Summary bq. --- bq. bq. Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. bq. bq. The regions chose to sideline are those which overlap with most other regions. bq. bq. bq. This addresses bug HBASE-5719. bq. https://issues.apache.org/jira/browse/HBASE-5719 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 bq.
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250092#comment-13250092 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/ --- (Updated 2012-04-09 19:27:53.361210) Review request for hbase and jmhsieh. Changes --- Addressed Jon's second review comments. Summary --- Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. The regions chose to sideline are those which overlap with most other regions. This addresses bug HBASE-5719. https://issues.apache.org/jira/browse/HBASE-5719 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java ac3b225 Diff: https://reviews.apache.org/r/4649/diff Testing --- mvn -PlocalTests -Dtest=TestHBaseFsck* clean test Also tested in real system to fix inconsistencies. Thanks, Jimmy Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some regions in the group and break the overlapping to fix the inconsistency. Later on, sidelined regions can be bulk loaded manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5752) Blank line in SPLITS_FILE causes Master to crash
Blank line in SPLITS_FILE causes Master to crash
Key: HBASE-5752
URL: https://issues.apache.org/jira/browse/HBASE-5752
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 0.92.1
Reporter: Jeremy Carroll
Priority: Minor
When creating a new table with the hbase shell, and specifying a SPLITS_FILE
with a blank line in it will cause the master to crash.
Uploading a sample splits file, here are the commands to test the split.
create 'testTable', {NAME = 'a', BLOOMFILTER = 'NONE', REPLICATION_SCOPE =
'0', COMPRESSION = 'NONE', MIN_VERSIONS = '3', TTL = '2147483647', BLOCKSIZE
= '65536', IN_MEMORY = 'false', BLOCKCACHE = 'true'}, {SPLITS_FILE =
'/tmp/test.txt'}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5753) [book] adding listdata schema design case-study to Use Cases chapter
[book] adding listdata schema design case-study to Use Cases chapter Key: HBASE-5753 URL: https://issues.apache.org/jira/browse/HBASE-5753 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Ported a discussion from the dist-list about designing a schema to support list data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5753) [book] adding listdata schema design case-study to Use Cases chapter
[ https://issues.apache.org/jira/browse/HBASE-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5753: - Description: case_studies.xml * Ported a discussion from the dist-list about designing a schema to support list data t * created Schema Design and Performance/Troubleshooting sections in this chapter (previously all the entries were perf/trouble related) was: Ported a discussion from the dist-list about designing a schema to support list data. [book] adding listdata schema design case-study to Use Cases chapter Key: HBASE-5753 URL: https://issues.apache.org/jira/browse/HBASE-5753 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: case_studies_hbase_5753.xml.patch case_studies.xml * Ported a discussion from the dist-list about designing a schema to support list data t * created Schema Design and Performance/Troubleshooting sections in this chapter (previously all the entries were perf/trouble related) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5753) [book] adding listdata schema design case-study to Use Cases chapter
[ https://issues.apache.org/jira/browse/HBASE-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5753: - Attachment: case_studies_hbase_5753.xml.patch [book] adding listdata schema design case-study to Use Cases chapter Key: HBASE-5753 URL: https://issues.apache.org/jira/browse/HBASE-5753 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: case_studies_hbase_5753.xml.patch case_studies.xml * Ported a discussion from the dist-list about designing a schema to support list data t * created Schema Design and Performance/Troubleshooting sections in this chapter (previously all the entries were perf/trouble related) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5753) [book] adding listdata schema design case-study to Use Cases chapter
[ https://issues.apache.org/jira/browse/HBASE-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5753: - Status: Patch Available (was: Open) [book] adding listdata schema design case-study to Use Cases chapter Key: HBASE-5753 URL: https://issues.apache.org/jira/browse/HBASE-5753 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: case_studies_hbase_5753.xml.patch case_studies.xml * Ported a discussion from the dist-list about designing a schema to support list data t * created Schema Design and Performance/Troubleshooting sections in this chapter (previously all the entries were perf/trouble related) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5753) [book] adding listdata schema design case-study to Use Cases chapter
[ https://issues.apache.org/jira/browse/HBASE-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Doug Meil updated HBASE-5753: - Resolution: Fixed Status: Resolved (was: Patch Available) [book] adding listdata schema design case-study to Use Cases chapter Key: HBASE-5753 URL: https://issues.apache.org/jira/browse/HBASE-5753 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: case_studies_hbase_5753.xml.patch case_studies.xml * Ported a discussion from the dist-list about designing a schema to support list data t * created Schema Design and Performance/Troubleshooting sections in this chapter (previously all the entries were perf/trouble related) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5727) secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs'
[ https://issues.apache.org/jira/browse/HBASE-5727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250140#comment-13250140 ] Zhihong Yu commented on HBASE-5727: --- If there is no objection, I will integrate the addendum in 3 hours. secure hbase build broke because of 'HBASE-5451 Switch RPC call envelope/headers to PBs' Key: HBASE-5727 URL: https://issues.apache.org/jira/browse/HBASE-5727 Project: HBase Issue Type: Bug Reporter: stack Assignee: Devaraj Das Priority: Blocker Fix For: 0.96.0 Attachments: 5727.1.patch, 5727.2.patch, 5727.patch, AccessController.diff If you build with the security profile -- i.e. add '-P security' on the command line -- you'll see that the secure build is broke since we messed in rpc. Assigning Deveraj to take a look. If you can't work on this now DD, just give it back to me and I'll have a go at it. Thanks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5744) Thrift server metrics should be long instead of int
[ https://issues.apache.org/jira/browse/HBASE-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin updated HBASE-5744: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thrift server metrics should be long instead of int --- Key: HBASE-5744 URL: https://issues.apache.org/jira/browse/HBASE-5744 Project: HBase Issue Type: Bug Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D2679.1.patch, D2685.1.patch, D2685.2.patch, D2685.3.patch, jira-HBASE-5744-89-fb-Thrift-server-metrics-should-b-2012-04-07_21_39_35.patch As we measure our Thrift call latencies in nanoseconds, we need to make latencies long instead of int everywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2223) Handle 10min+ network partitions between clusters
[ https://issues.apache.org/jira/browse/HBASE-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250160#comment-13250160 ] Himanshu Vashishtha commented on HBASE-2223: I was skimming through the patch; and it looks like its not there in the current trunk? Will be great if anyone can provide its current status, or follow up jiras? Handle 10min+ network partitions between clusters - Key: HBASE-2223 URL: https://issues.apache.org/jira/browse/HBASE-2223 Project: HBase Issue Type: Sub-task Reporter: Jean-Daniel Cryans Assignee: Jean-Daniel Cryans Fix For: 0.90.0 Attachments: HBASE-2223.patch We need a nice way of handling long network partitions without impacting a master cluster (which pushes the data). Currently it will just retry over and over again. I think we could: - Stop replication to a slave cluster if it didn't respond for more than 10 minutes - Keep track of the duration of the partition - When the slave cluster comes back, initiate a MR job like HBASE-2221 Maybe we want less than 10 minutes, maybe we want this to be all automatic or just the first 2 parts. Discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4336) Convert source tree into maven modules
[ https://issues.apache.org/jira/browse/HBASE-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250170#comment-13250170 ] Jesse Yates commented on HBASE-4336: bq. client module with the minimum set of dependencies would a big thing Yup, that was part of the my original plan. Its just painful to do. :) Agree that follow on tickets would/should be useful for this. My original intention with this patch to do it wholesale (master, region, client, etc.), but I think we can get away with just a couple small ones (see my above comment) without too much work. bq. BC = backwards compatibility. Still think my argument holds above for preserving this as well. Convert source tree into maven modules -- Key: HBASE-4336 URL: https://issues.apache.org/jira/browse/HBASE-4336 Project: HBase Issue Type: Task Components: build Reporter: Gary Helmling Priority: Critical Fix For: 0.96.0 When we originally converted the build to maven we had a single core module defined, but later reverted this to a module-less build for the sake of simplicity. It now looks like it's time to re-address this, as we have an actual need for modules to: * provide a trimmed down client library that applications can make use of * more cleanly support building against different versions of Hadoop, in place of some of the reflection machinations currently required * incorporate the secure RPC engine that depends on some secure Hadoop classes I propose we start simply by refactoring into two initial modules: * core - common classes and utilities, and client-side code and interfaces * server - master and region server implementations and supporting code This would also lay the groundwork for incorporating the HBase security features that have been developed. Once the module structure is in place, security-related features could then be incorporated into a third module -- security -- after normal review and approval. The security module could then depend on secure Hadoop, without modifying the dependencies of the rest of the HBase code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5754) data lost with gora continuous ingest test (goraci)
[ https://issues.apache.org/jira/browse/HBASE-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-5754: Assignee: stack data lost with gora continuous ingest test (goraci) --- Key: HBASE-5754 URL: https://issues.apache.org/jira/browse/HBASE-5754 Project: HBase Issue Type: Bug Affects Versions: 0.92.1 Environment: 10 node test cluster Reporter: Eric Newton Assignee: stack Keith Turner re-wrote the accumulo continuous ingest test using gora, which has both hbase and accumulo back-ends. I put a billion entries into HBase, and ran the Verify map/reduce job. The verification failed because about 21K entries were missing. The goraci [README|https://github.com/keith-turner/goraci] explains the test, and how it detects missing data. I re-ran the test with 100 million entries, and it verified successfully. Both of the times I tested using a billion entries, the verification failed. If I run the verification step twice, the results are consistent, so the problem is probably not on the verify step. Here's the versions of the various packages: ||package||version|| |hadoop|0.20.205.0| |hbase|0.92.1| |gora|http://svn.apache.org/repos/asf/gora/trunk r1311277| |goraci|https://github.com/ericnewton/goraci tagged 2012-04-08| The change I made to goraci was to configure it for hbase and to allow it to build properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5754) data lost with gora continuous ingest test (goraci)
[ https://issues.apache.org/jira/browse/HBASE-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250195#comment-13250195 ] stack commented on HBASE-5754: -- Let me take this one. I've been playing w/ goraci. I did 25M for my first test and that seemed to work. Let me do the bigger numbers. Thanks Eric. data lost with gora continuous ingest test (goraci) --- Key: HBASE-5754 URL: https://issues.apache.org/jira/browse/HBASE-5754 Project: HBase Issue Type: Bug Affects Versions: 0.92.1 Environment: 10 node test cluster Reporter: Eric Newton Assignee: stack Keith Turner re-wrote the accumulo continuous ingest test using gora, which has both hbase and accumulo back-ends. I put a billion entries into HBase, and ran the Verify map/reduce job. The verification failed because about 21K entries were missing. The goraci [README|https://github.com/keith-turner/goraci] explains the test, and how it detects missing data. I re-ran the test with 100 million entries, and it verified successfully. Both of the times I tested using a billion entries, the verification failed. If I run the verification step twice, the results are consistent, so the problem is probably not on the verify step. Here's the versions of the various packages: ||package||version|| |hadoop|0.20.205.0| |hbase|0.92.1| |gora|http://svn.apache.org/repos/asf/gora/trunk r1311277| |goraci|https://github.com/ericnewton/goraci tagged 2012-04-08| The change I made to goraci was to configure it for hbase and to allow it to build properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5753) [book] adding listdata schema design case-study to Use Cases chapter
[ https://issues.apache.org/jira/browse/HBASE-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250223#comment-13250223 ] Hudson commented on HBASE-5753: --- Integrated in HBase-TRUNK #2732 (See [https://builds.apache.org/job/HBase-TRUNK/2732/]) hbase-5753.xml - adding schema design case study in Case Studies chapter (Revision 1311411) Result = SUCCESS [book] adding listdata schema design case-study to Use Cases chapter Key: HBASE-5753 URL: https://issues.apache.org/jira/browse/HBASE-5753 Project: HBase Issue Type: Improvement Reporter: Doug Meil Assignee: Doug Meil Priority: Minor Attachments: case_studies_hbase_5753.xml.patch case_studies.xml * Ported a discussion from the dist-list about designing a schema to support list data t * created Schema Design and Performance/Troubleshooting sections in this chapter (previously all the entries were perf/trouble related) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5752) Blank line in SPLITS_FILE causes Master to crash
[
https://issues.apache.org/jira/browse/HBASE-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250230#comment-13250230
]
stack commented on HBASE-5752:
--
Something like this Jeremy?
{code}
Index: src/main/ruby/hbase/admin.rb
===
--- src/main/ruby/hbase/admin.rb(revision 1311378)
+++ src/main/ruby/hbase/admin.rb(working copy)
@@ -194,7 +194,9 @@
end
arg[SPLITS] = []
File.foreach(arg[SPLITS_FILE]) do |line|
-arg[SPLITS].push(line.strip())
+stripped = line.strip()
+next unless stripped.length 0
+arg[SPLITS].push(stripped)
end
end
{code}
I've not tried it.
Blank line in SPLITS_FILE causes Master to crash
Key: HBASE-5752
URL: https://issues.apache.org/jira/browse/HBASE-5752
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 0.92.1
Reporter: Jeremy Carroll
Priority: Minor
Attachments: test.txt
When creating a new table with the hbase shell, and specifying a SPLITS_FILE
with a blank line in it will cause the master to crash.
Uploading a sample splits file, here are the commands to test the split.
create 'testTable', {NAME = 'a', BLOOMFILTER = 'NONE', REPLICATION_SCOPE =
'0', COMPRESSION = 'NONE', MIN_VERSIONS = '3', TTL = '2147483647',
BLOCKSIZE = '65536', IN_MEMORY = 'false', BLOCKCACHE = 'true'},
{SPLITS_FILE = '/tmp/test.txt'}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5749) HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize
[ https://issues.apache.org/jira/browse/HBASE-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5749: - Status: Patch Available (was: Open) HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize -- Key: HBASE-5749 URL: https://issues.apache.org/jira/browse/HBASE-5749 Project: HBase Issue Type: Bug Reporter: stack Attachments: heapsize.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5748) Enable lib directory in jar file for coprocessor
[ https://issues.apache.org/jira/browse/HBASE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5748: - Resolution: Fixed Fix Version/s: (was: 0.96.0) 0.92.2 Status: Resolved (was: Patch Available) Committed to 0.92, 0.94 and trunk. Thanks for the patch Takuya. Enable lib directory in jar file for coprocessor Key: HBASE-5748 URL: https://issues.apache.org/jira/browse/HBASE-5748 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.1, 0.94.0 Reporter: Takuya Ueshin Assignee: Takuya Ueshin Fix For: 0.92.2, 0.94.0 Attachments: HBASE-5748.patch Hadoop MapReduce job can use external libraries in 'lib' directory in the job.jar file. It is useful that jar files for coprocessor can use external libraries in the same way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5740) Compaction interruption may be due to balacing
[ https://issues.apache.org/jira/browse/HBASE-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250262#comment-13250262 ] stack commented on HBASE-5740: -- The first change is fine -- saying it was interrupted rather than it was stopped by user. The second change is not so clean; says user or system. Maybe this should be generic too... just 'compaction interrupted'? If you are good w/ the latter I can make this change on commit. Compaction interruption may be due to balacing -- Key: HBASE-5740 URL: https://issues.apache.org/jira/browse/HBASE-5740 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Trivial Fix For: 0.96.0 Attachments: hbase-5740.patch, hbase-5740_v2.patch Currently, the log shows Aborting compaction of store LOG in region because user requested stop. But it is actually because of balancing. Currently, there is no way to figure out who closed the region. So it is better to change the message to say it is because of user, or balancing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5748) Enable lib directory in jar file for coprocessor
[ https://issues.apache.org/jira/browse/HBASE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250268#comment-13250268 ] Takuya Ueshin commented on HBASE-5748: -- Thank you very much! Enable lib directory in jar file for coprocessor Key: HBASE-5748 URL: https://issues.apache.org/jira/browse/HBASE-5748 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.1, 0.94.0 Reporter: Takuya Ueshin Assignee: Takuya Ueshin Fix For: 0.92.2, 0.94.0 Attachments: HBASE-5748.patch Hadoop MapReduce job can use external libraries in 'lib' directory in the job.jar file. It is useful that jar files for coprocessor can use external libraries in the same way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5740) Compaction interruption may be due to balacing
[ https://issues.apache.org/jira/browse/HBASE-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250267#comment-13250267 ] Jimmy Xiang commented on HBASE-5740: @Stack, I am fine with the generic message. Please make the change on commit. Thanks a lot. We don't know for sure who interrupted it anyway for now. Compaction interruption may be due to balacing -- Key: HBASE-5740 URL: https://issues.apache.org/jira/browse/HBASE-5740 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Trivial Fix For: 0.96.0 Attachments: hbase-5740.patch, hbase-5740_v2.patch Currently, the log shows Aborting compaction of store LOG in region because user requested stop. But it is actually because of balancing. Currently, there is no way to figure out who closed the region. So it is better to change the message to say it is because of user, or balancing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5666) RegionServer doesn't retry to check if base node is available
[
https://issues.apache.org/jira/browse/HBASE-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250269#comment-13250269
]
stack commented on HBASE-5666:
--
Patch looks good.
Logs '{
+LOG.warn(zkw.prefix(Unable to set watcher on znode ( + znode +
)), keeperEx);
'
... but the method says its checkExists w/o setting watch.
I think this a bad idea; i.e. sleeping w/o interrupt. How long is
SOCKET_RETRY_WAIT_MS? What if we try to stop the hosting server in meantime?
We have to wait on this to come up out of this loop?
+Threads.sleepWithoutInterrupt(HConstants.SOCKET_RETRY_WAIT_MS);
Passing 0, are we supposed to try once only? My guess is that we could try
more than once given how the loop runs; i.e. we may loop multiple times in same
millisecond.. you might want to exit loop if timeout is zero.
What happens if a client comes in during this time? It will crash out
immediately because no base node?
Thanks Matteo.
RegionServer doesn't retry to check if base node is available
-
Key: HBASE-5666
URL: https://issues.apache.org/jira/browse/HBASE-5666
Project: HBase
Issue Type: Bug
Components: regionserver, zookeeper
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Attachments: HBASE-5666-v1.patch, HBASE-5666-v2.patch,
HBASE-5666-v3.patch, HBASE-5666-v4.patch, HBASE-5666-v5.patch,
HBASE-5666-v6.patch, hbase-1-regionserver.log, hbase-2-regionserver.log,
hbase-3-regionserver.log, hbase-master.log, hbase-regionserver.log,
hbase-zookeeper.log
I've a script that starts hbase and a couple of region servers in distributed
mode (hbase.cluster.distributed = true)
{code}
$HBASE_HOME/bin/start-hbase.sh
$HBASE_HOME/bin/local-regionservers.sh start 1 2 3
{code}
but the region servers are not able to start...
It seems that during the RS start the the znode is still not available, and
HRegionServer.initializeZooKeeper() check just once if the base not is
available.
{code}
2012-03-28 21:54:05,013 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Check the value
configured in 'zookeeper.znode.parent'. There could be a mismatch with the
one configured in the master.
2012-03-28 21:54:08,598 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
localhost,60202,133296824: Initialization of RS failed. Hence aborting
RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:626)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:596)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:558)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:672)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5740) Compaction interruption may be due to balacing
[ https://issues.apache.org/jira/browse/HBASE-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5740: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the patch Jimmy. Compaction interruption may be due to balacing -- Key: HBASE-5740 URL: https://issues.apache.org/jira/browse/HBASE-5740 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Trivial Fix For: 0.96.0 Attachments: hbase-5740.patch, hbase-5740_v2.patch Currently, the log shows Aborting compaction of store LOG in region because user requested stop. But it is actually because of balancing. Currently, there is no way to figure out who closed the region. So it is better to change the message to say it is because of user, or balancing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5748) Enable lib directory in jar file for coprocessor
[ https://issues.apache.org/jira/browse/HBASE-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250276#comment-13250276 ] stack commented on HBASE-5748: -- No. Thank you (smile). Enable lib directory in jar file for coprocessor Key: HBASE-5748 URL: https://issues.apache.org/jira/browse/HBASE-5748 Project: HBase Issue Type: Improvement Components: coprocessors Affects Versions: 0.92.1, 0.94.0 Reporter: Takuya Ueshin Assignee: Takuya Ueshin Fix For: 0.92.2, 0.94.0 Attachments: HBASE-5748.patch Hadoop MapReduce job can use external libraries in 'lib' directory in the job.jar file. It is useful that jar files for coprocessor can use external libraries in the same way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5749) HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize
[ https://issues.apache.org/jira/browse/HBASE-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250282#comment-13250282 ] Nicolas Spiegelberg commented on HBASE-5749: +1 if you haven't committed/resolved already HBASE-5335 Dynamic Schema Config broke build, missing change to HRegion heapsize -- Key: HBASE-5749 URL: https://issues.apache.org/jira/browse/HBASE-5749 Project: HBase Issue Type: Bug Reporter: stack Attachments: heapsize.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
