[jira] [Commented] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
[
https://issues.apache.org/jira/browse/HBASE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262823#comment-14262823
]
Hudson commented on HBASE-12716:
FAILURE: Integrated in HBase-0.98 #772 (See
[https://builds.apache.org/job/HBase-0.98/772/])
HBASE-12716 A bug in RegionSplitter.UniformSplit algorithm (Weichen Ye)
(apurtell: rev 74f303ba2482b2f0c81260ee41cd0f94ef3105d8)
* hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
A bug in RegionSplitter.UniformSplit algorithm
--
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.0.0, 0.98.6
Reporter: Weichen Ye
Assignee: Weichen Ye
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12716-v2.patch, HBASE-12716.patch
Welcome to the review board: https://reviews.apache.org/r/29424/diff/#
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm
meet an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
{code}
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
{code}
We will get the ERROR:
{code}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
{code}
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
review board:https://reviews.apache.org/r/29424/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12641) Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster
[ https://issues.apache.org/jira/browse/HBASE-12641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262822#comment-14262822 ] Hudson commented on HBASE-12641: FAILURE: Integrated in HBase-0.98 #772 (See [https://builds.apache.org/job/HBase-0.98/772/]) HBASE-12641 Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster (Liu Shaohui) (apurtell: rev efc49a745fa198e8f5ed9abe76392e6fff836d75) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster Key: HBASE-12641 URL: https://issues.apache.org/jira/browse/HBASE-12641 Project: HBase Issue Type: Improvement Components: Zookeeper Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12641-v1.diff Currently in a secure cluster, only the master/regionserver kerberos user can manage the znode of hbase. But he master/regionserver kerberos user is for rpc connection and we usually use another super user to manage the cluster. In some special scenarios, we need to manage the data of znode with the supper user. eg: a, To get the data of the znode for debugging. b, HBASE-8253: We need to delete the znode for the corrupted hlog to avoid it block the replication. So we grant all permissions of hbase zookeeper node to hbase superuser during creating these znodes. Suggestions are welcomed. [~apurtell] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12028) Abort the RegionServer, when it's handler threads die
[ https://issues.apache.org/jira/browse/HBASE-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263348#comment-14263348 ] Hudson commented on HBASE-12028: SUCCESS: Integrated in HBase-1.0 #626 (See [https://builds.apache.org/job/HBase-1.0/626/]) HBASE-12028 Abort the RegionServer, when it's handler threads die (Alicia Ying Shu) (enis: rev f960f2a9062a4ab3bccdcd2718f001eed54c9d18) * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SimpleRpcSchedulerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RpcSchedulerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BalancedQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestRpcHandlerException.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java Abort the RegionServer, when it's handler threads die - Key: HBASE-12028 URL: https://issues.apache.org/jira/browse/HBASE-12028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Sudarshan Kadambi Assignee: Alicia Ying Shu Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: Hbase-12028-v3.patch, Hbase-12028.patch, hbase-12028-v4.patch, hbase-12028-v5-branch-1.patch, hbase-12028-v5-master.patch, hbase-12028-v5.patch Over in HBase-11813, a user identified an issue where in all the RPC handler threads would exit with StackOverflow errors due to an unchecked recursion-terminating condition. Our clusters demonstrated the same trace. While the patch posted for HBASE-11813 got our clusters to be merry again, the breakdown surfaced some larger issues. When the RegionServer had all it's RPC handler threads dead, it continued to have regions assigned it. Clearly, it wouldn't be able to serve reads and writes on those regions. A second issue was that when a user tried to disable or drop a table, the master would try to communicate to the regionserver for region unassignment. Since the same handler threads seem to be used for master - RS communication as well, the master ended up hanging on the RS indefinitely. Eventually, the master stopped responding to all table meta-operations. A handler thread should never exit, and if it does, it seems like the more prudent thing to do would be for the RS to abort. This way, at least recovery can be undertaken and the regions could be reassigned elsewhere. I also think that the master-RS communication should get its own exclusive threadpool, but I'll wait until this issue has been sufficiently discussed before opening an issue ticket for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12735) Refactor TAG so it can live as unit test and as an integration test
[ https://issues.apache.org/jira/browse/HBASE-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263344#comment-14263344 ] Enis Soztutar commented on HBASE-12735: --- I've pushed this to 1.0.0 as well. Refactor TAG so it can live as unit test and as an integration test --- Key: HBASE-12735 URL: https://issues.apache.org/jira/browse/HBASE-12735 Project: HBase Issue Type: Sub-task Components: test Reporter: stack Assignee: Jonathan Hsieh Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: hbase-12735.patch Parent task moved TAG to IT wholesale. This is about keeping a bit of ACID coverage going in UT. Jon offered to refactor TAG so can live in IT and UT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12718) Convert TestAcidGuarantees from a unit test to an integration test
[ https://issues.apache.org/jira/browse/HBASE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12718: -- Fix Version/s: 1.0.0 Convert TestAcidGuarantees from a unit test to an integration test -- Key: HBASE-12718 URL: https://issues.apache.org/jira/browse/HBASE-12718 Project: HBase Issue Type: Bug Components: hbase, integration tests, test Affects Versions: 1.0.0, 2.0.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 1.0.0, 2.0.0, hbase-11339, 1.1.0 Attachments: hbase-12718.patch TestAcidGurantees has a main method so that it can be executed via the commandline. In the past this was run and we'd use external tools to inject faults while it executed for an extended period of time. We've had the IT framework we'd like to use the ChaosMonkey automation with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12735) Refactor TAG so it can live as unit test and as an integration test
[ https://issues.apache.org/jira/browse/HBASE-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12735: -- Fix Version/s: 1.0.0 Refactor TAG so it can live as unit test and as an integration test --- Key: HBASE-12735 URL: https://issues.apache.org/jira/browse/HBASE-12735 Project: HBase Issue Type: Sub-task Components: test Reporter: stack Assignee: Jonathan Hsieh Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: hbase-12735.patch Parent task moved TAG to IT wholesale. This is about keeping a bit of ACID coverage going in UT. Jon offered to refactor TAG so can live in IT and UT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12799) ITAG fails with java.lang.RuntimeException if table does not exist
[
https://issues.apache.org/jira/browse/HBASE-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12799:
--
Attachment: hbase-12799_v1.patch
Simple patch. We check for whether it is a mini cluster to check AM status.
ITAG fails with java.lang.RuntimeException if table does not exist
--
Key: HBASE-12799
URL: https://issues.apache.org/jira/browse/HBASE-12799
Project: HBase
Issue Type: Test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0, 2.0.0, 1.1.0
Attachments: hbase-12799_v1.patch
If the table does not exists:
{code}
2015-01-02 16:20:35,764 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.RuntimeException:
org.apache.hadoop.hbase.DistributedHBaseCluster@7e02856c not an instance of
org.apache.hadoop.hbase.MiniHBaseCluster
at
org.apache.hadoop.hbase.HBaseTestingUtility.getMiniHBaseCluster(HBaseTestingUtility.java:1022)
at
org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseCluster(HBaseTestingUtility.java:2569)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3055)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3011)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1304)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1334)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1254)
at
org.apache.hadoop.hbase.TestAcidGuarantees.createTableIfMissing(TestAcidGuarantees.java:80)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:264)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.runTestFromCommandLine(IntegrationTestAcidGuarantees.java:58)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:112)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.main(IntegrationTestAcidGuarantees.java:115)
{code}
[~jmhsieh] FYI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12799) ITAG fails with java.lang.RuntimeException if table does not exist
[
https://issues.apache.org/jira/browse/HBASE-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12799:
--
Status: Patch Available (was: Open)
ITAG fails with java.lang.RuntimeException if table does not exist
--
Key: HBASE-12799
URL: https://issues.apache.org/jira/browse/HBASE-12799
Project: HBase
Issue Type: Test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0, 2.0.0, 1.1.0
Attachments: hbase-12799_v1.patch
If the table does not exists:
{code}
2015-01-02 16:20:35,764 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.RuntimeException:
org.apache.hadoop.hbase.DistributedHBaseCluster@7e02856c not an instance of
org.apache.hadoop.hbase.MiniHBaseCluster
at
org.apache.hadoop.hbase.HBaseTestingUtility.getMiniHBaseCluster(HBaseTestingUtility.java:1022)
at
org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseCluster(HBaseTestingUtility.java:2569)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3055)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3011)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1304)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1334)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1254)
at
org.apache.hadoop.hbase.TestAcidGuarantees.createTableIfMissing(TestAcidGuarantees.java:80)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:264)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.runTestFromCommandLine(IntegrationTestAcidGuarantees.java:58)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:112)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.main(IntegrationTestAcidGuarantees.java:115)
{code}
[~jmhsieh] FYI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12799) ITAG fails with java.lang.RuntimeException if table does not exist
[
https://issues.apache.org/jira/browse/HBASE-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12799:
--
Summary: ITAG fails with java.lang.RuntimeException if table does not exist
(was: ITAG fails java.lang.RuntimeException:
org.apache.hadoop.hbase.DistributedHBaseCluster@49ef4b31 not an instance of
org.apache.hadoop.hbase.MiniHBaseCluster)
ITAG fails with java.lang.RuntimeException if table does not exist
--
Key: HBASE-12799
URL: https://issues.apache.org/jira/browse/HBASE-12799
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0, 2.0.0, 1.1.0
If the table does not exists:
{code}
2015-01-02 16:20:35,764 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.RuntimeException:
org.apache.hadoop.hbase.DistributedHBaseCluster@7e02856c not an instance of
org.apache.hadoop.hbase.MiniHBaseCluster
at
org.apache.hadoop.hbase.HBaseTestingUtility.getMiniHBaseCluster(HBaseTestingUtility.java:1022)
at
org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseCluster(HBaseTestingUtility.java:2569)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3055)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3011)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1304)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1334)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1254)
at
org.apache.hadoop.hbase.TestAcidGuarantees.createTableIfMissing(TestAcidGuarantees.java:80)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:264)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.runTestFromCommandLine(IntegrationTestAcidGuarantees.java:58)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:112)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.main(IntegrationTestAcidGuarantees.java:115)
{code}
[~jmhsieh] FYI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12799) ITAG fails with java.lang.RuntimeException if table does not exist
[
https://issues.apache.org/jira/browse/HBASE-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12799:
--
Issue Type: Test (was: Bug)
ITAG fails with java.lang.RuntimeException if table does not exist
--
Key: HBASE-12799
URL: https://issues.apache.org/jira/browse/HBASE-12799
Project: HBase
Issue Type: Test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0, 2.0.0, 1.1.0
If the table does not exists:
{code}
2015-01-02 16:20:35,764 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.RuntimeException:
org.apache.hadoop.hbase.DistributedHBaseCluster@7e02856c not an instance of
org.apache.hadoop.hbase.MiniHBaseCluster
at
org.apache.hadoop.hbase.HBaseTestingUtility.getMiniHBaseCluster(HBaseTestingUtility.java:1022)
at
org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseCluster(HBaseTestingUtility.java:2569)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3055)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3011)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1304)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1334)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1254)
at
org.apache.hadoop.hbase.TestAcidGuarantees.createTableIfMissing(TestAcidGuarantees.java:80)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:264)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.runTestFromCommandLine(IntegrationTestAcidGuarantees.java:58)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:112)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.main(IntegrationTestAcidGuarantees.java:115)
{code}
[~jmhsieh] FYI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12718) Convert TestAcidGuarantees from a unit test to an integration test
[ https://issues.apache.org/jira/browse/HBASE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263346#comment-14263346 ] Enis Soztutar commented on HBASE-12718: --- We now need HBASE-12799. Convert TestAcidGuarantees from a unit test to an integration test -- Key: HBASE-12718 URL: https://issues.apache.org/jira/browse/HBASE-12718 Project: HBase Issue Type: Bug Components: hbase, integration tests, test Affects Versions: 1.0.0, 2.0.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 1.0.0, 2.0.0, hbase-11339, 1.1.0 Attachments: hbase-12718.patch TestAcidGurantees has a main method so that it can be executed via the commandline. In the past this was run and we'd use external tools to inject faults while it executed for an extended period of time. We've had the IT framework we'd like to use the ChaosMonkey automation with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12718) Convert TestAcidGuarantees from a unit test to an integration test
[ https://issues.apache.org/jira/browse/HBASE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263345#comment-14263345 ] Enis Soztutar commented on HBASE-12718: --- I've pushed this (and subtask) to 1.0.0 as well. Convert TestAcidGuarantees from a unit test to an integration test -- Key: HBASE-12718 URL: https://issues.apache.org/jira/browse/HBASE-12718 Project: HBase Issue Type: Bug Components: hbase, integration tests, test Affects Versions: 1.0.0, 2.0.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 1.0.0, 2.0.0, hbase-11339, 1.1.0 Attachments: hbase-12718.patch TestAcidGurantees has a main method so that it can be executed via the commandline. In the past this was run and we'd use external tools to inject faults while it executed for an extended period of time. We've had the IT framework we'd like to use the ChaosMonkey automation with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12788) Promote Abortable to LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-12788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12788: -- Attachment: hbase-12788_v1.patch Thanks Andrew. Attaching simple patch. Promote Abortable to LimitedPrivate --- Key: HBASE-12788 URL: https://issues.apache.org/jira/browse/HBASE-12788 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Enis Soztutar Labels: phoenix Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: hbase-12788_v1.patch After HBASE-12028, we will be passing Abortable around in a LimitedPrivate interface, with the intent of its use in implementors and subclasses of related types (RPC schedulers). Therefore we should promote Abortable to LimitedPrivate as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12788) Promote Abortable to LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-12788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12788: -- Status: Patch Available (was: Open) Promote Abortable to LimitedPrivate --- Key: HBASE-12788 URL: https://issues.apache.org/jira/browse/HBASE-12788 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Enis Soztutar Labels: phoenix Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: hbase-12788_v1.patch After HBASE-12028, we will be passing Abortable around in a LimitedPrivate interface, with the intent of its use in implementors and subclasses of related types (RPC schedulers). Therefore we should promote Abortable to LimitedPrivate as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12711) Fix new findbugs warnings in hbase-thrift module
[ https://issues.apache.org/jira/browse/HBASE-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12711: -- Fix Version/s: 1.0.0 Fix new findbugs warnings in hbase-thrift module Key: HBASE-12711 URL: https://issues.apache.org/jira/browse/HBASE-12711 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Srikanth Srungarapu Priority: Minor Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-12711.patch, HBASE-12711_v2.patch, HBASE-12711_v3.patch From https://builds.apache.org/job/PreCommit-HBASE-Build/12121/artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html , there were 5 findbugs warnings introduced. This issue fixes the new warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reopened HBASE-12270: TestCacheOnWrite has been failing in 0.98 builds since https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/737/testReport/org.apache.hadoop.hbase.io.hfile/TestCacheOnWrite/testStoreFileCacheOnWrite_105_/ Please fix or revert this change. A bug in the bucket cache, with cache blocks on write enabled - Key: HBASE-12270 URL: https://issues.apache.org/jira/browse/HBASE-12270 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6.1 Environment: I can reproduce it on a simple 2 node cluster, one running the master and another running a RS. I was testing on ec2. I used the following configurations for the cluster. hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log hbase-site: hbase.bucketcache.ioengine=offheap hbase.bucketcache.size=4196 hbase.rs.cacheblocksonwrite=true hfile.block.index.cacheonwrite=true hfile.block.bloom.cacheonwrite=true Reporter: Khaled Elmeleegy Assignee: Liu Shaohui Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java In my experiments, I have writers streaming their output to HBase. The reader powers a web page and does this scatter/gather, where it reads 1000 keys written last and passes them the the front end. With this workload, I get the exception below at the region server. Again, I am using HBAse (0.98.6.1). Any help is appreciated. 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:236) at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12791) HBase does not attempt to clean up an aborted split when the regionserver shutting down
[
https://issues.apache.org/jira/browse/HBASE-12791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263119#comment-14263119
]
Enis Soztutar commented on HBASE-12791:
---
In offline discussion with Rajesh, it seems that not cleaning up after
SplitTransaction abort might be ok, since in case of RS hard fail (kill -9,
etc), the SplitTransaction will still leave dirty state around (mainly hdfs
directories for daughters). I think we should make master clean up the daughter
dirs in case a SPLIT fails.
bq. And also while fixing meta holes in hbck we should check for overlaps
beforehand and need not add them to meta in such case.
+1.
HBase does not attempt to clean up an aborted split when the regionserver
shutting down
---
Key: HBASE-12791
URL: https://issues.apache.org/jira/browse/HBASE-12791
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.98.0
Reporter: Rajeshbabu Chintaguntla
Assignee: Rajeshbabu Chintaguntla
Priority: Critical
Fix For: 2.0.0, 0.98.10, 1.0.1
HBase not cleaning the daughter region directories from HDFS if region
server shut down after creating the daughter region directories during the
split.
Here the logs.
- RS shutdown after creating the daughter regions.
{code}
2014-12-31 09:05:41,406 DEBUG [regionserver60020-splits-1419996941385]
zookeeper.ZKAssign: regionserver:60020-0x14a9701e53100d1,
quorum=localhost:2181, baseZNode=/hbase Transitioned node
80c665138d4fa32da4d792d8ed13206f from RS_ZK_REQUEST_REGION_SPLIT to
RS_ZK_REQUEST_REGION_SPLIT
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closing
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.: disabling compactions
flushes
2014-12-31 09:05:41,514 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Updates disabled for region
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:41,516 INFO
[StoreCloserThread-t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.-1]
regionserver.HStore: Closed f
2014-12-31 09:05:41,518 INFO [regionserver60020-splits-1419996941385]
regionserver.HRegion: Closed
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f.
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl
for table t dd9731ee43b104da565257ca1539aa8c
2014-12-31 09:05:49,922 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,,1419996941401.dd9731ee43b104da565257ca1539aa8c.
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl
for table t 2e40a44511c0e187d357d651f13a1dab
2014-12-31 09:05:49,929 DEBUG [regionserver60020-splits-1419996941385]
regionserver.HRegion: Instantiated
t,row2,1419996941401.2e40a44511c0e187d357d651f13a1dab.
Wed Dec 31 09:06:30 IST 2014 Terminating regionserver
2014-12-31 09:06:30,465 INFO [Thread-8] regionserver.ShutdownHook: Shutdown
hook starting; hbase.shutdown.hook=true;
fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@42d2282e
{code}
- Skipping rollback if RS stopped or stopping so we end up in dirty daughter
regions in HDFS.
{code}
2014-12-31 09:07:49,547 INFO [regionserver60020-splits-1419996941385]
regionserver.SplitRequest: Skip rollback/cleanup of failed split of
t,,1419996880699.80c665138d4fa32da4d792d8ed13206f. because server is stopped
java.io.InterruptedIOException: Interrupted after 0 tries on 350
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:156)
{code}
Because of this hbck always showing inconsistencies.
{code}
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/2e40a44511c0e187d357d651f13a1dab,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any
region server
ERROR: Region { meta = null, hdfs =
hdfs://localhost:9000/hbase/data/default/t/dd9731ee43b104da565257ca1539aa8c,
deployed = } on HDFS, but not listed in hbase:meta or deployed on any
region server
{code}
If we try to repair then we end up in overlap regions in hbase:meta. and both
daughter regions and parent are online.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[
https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263115#comment-14263115
]
Ted Yu commented on HBASE-12783:
I can reproduce the following failure locally:
{code}
testExcludeMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2)
Time elapsed: 94.116 sec ERROR!
org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the
locations
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:309)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:137)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:186)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:296)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:271)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:143)
at
org.apache.hadoop.hbase.client.ClientScanner.init(ClientScanner.java:138)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:797)
at
org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:593)
at
org.apache.hadoop.hbase.MetaTableAccessor.getTableRegionsAndLocations(MetaTableAccessor.java:514)
at
org.apache.hadoop.hbase.MetaTableAccessor.getTableRegions(MetaTableAccessor.java:396)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getTableRegions(HBaseAdmin.java:2586)
at
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2.testExcludeMinorCompaction(TestHFileOutputFormat2.java:976)
{code}
There're a lot of KeeperException$ConnectionLossException's in test output.
Digging ...
Create efficient RegionLocator implementation
-
Key: HBASE-12783
URL: https://issues.apache.org/jira/browse/HBASE-12783
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: HBASE-12783.patch, HBASE-12783E.patch,
HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch,
HBASE-12783I.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch
A new HRegionLocator that only implements RegionLocator functionality will be
more efficient to instantiate than a full HTable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
[
https://issues.apache.org/jira/browse/HBASE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263087#comment-14263087
]
Hudson commented on HBASE-12716:
SUCCESS: Integrated in HBase-1.0 #625 (See
[https://builds.apache.org/job/HBase-1.0/625/])
HBASE-12716 A bug in RegionSplitter.UniformSplit algorithm (Weichen Ye) (tedyu:
rev 0256cdd1b408d431170cc46711f2d9ad561b3ef2)
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
* hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java
A bug in RegionSplitter.UniformSplit algorithm
--
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.0.0, 0.98.6
Reporter: Weichen Ye
Assignee: Weichen Ye
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12716-v2.patch, HBASE-12716.patch
Welcome to the review board: https://reviews.apache.org/r/29424/diff/#
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm
meet an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
{code}
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
{code}
We will get the ERROR:
{code}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
{code}
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
review board:https://reviews.apache.org/r/29424/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12786) List the zookeeper quorum nodes on different lines in the Master web ui
[ https://issues.apache.org/jira/browse/HBASE-12786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kainan jiang updated HBASE-12786: - Status: Patch Available (was: Open) List the zookeeper quorum nodes on different lines in the Master web ui --- Key: HBASE-12786 URL: https://issues.apache.org/jira/browse/HBASE-12786 Project: HBase Issue Type: Task Components: master Affects Versions: 0.89-fb Reporter: Rishit Shroff Priority: Trivial Attachments: 0001-HBASE-12786-Change-ZK-Quorum-display-list-each-serve.patch HBase uses zookeeper to store the current state of the cluster and uses it to maintain the overall consistency of the cluster. The zookeeper is quorum is hosted on set of nodes (5 typically) in the same hbase cluster. The names of these hosts are accessible in the HMaster web ui page. However, they are all ',' separated. It would be nice to list them on different lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[ https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12796: -- Attachment: HBASE-12796-v2.patch Fixed issues. Most of them were because MetaScanner only want to use managed connections. Now it creates its own HTable again. Clean up HTable and HBaseAdmin deprecated constructor usage --- Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous Attachments: HBASE-12796-v1.patch, HBASE-12796-v2.patch, HBASE-12796.patch The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12786) List the zookeeper quorum nodes on different lines in the Master web ui
[ https://issues.apache.org/jira/browse/HBASE-12786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kainan jiang updated HBASE-12786: - Attachment: 0001-HBASE-12786-Change-ZK-Quorum-display-list-each-serve.patch List the zookeeper quorum nodes on different lines in the Master web ui --- Key: HBASE-12786 URL: https://issues.apache.org/jira/browse/HBASE-12786 Project: HBase Issue Type: Task Components: master Affects Versions: 0.89-fb Reporter: Rishit Shroff Priority: Trivial Attachments: 0001-HBASE-12786-Change-ZK-Quorum-display-list-each-serve.patch HBase uses zookeeper to store the current state of the cluster and uses it to maintain the overall consistency of the cluster. The zookeeper is quorum is hosted on set of nodes (5 typically) in the same hbase cluster. The names of these hosts are accessible in the HMaster web ui page. However, they are all ',' separated. It would be nice to list them on different lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12783: --- Attachment: 12783-10.patch Moved call to util.startMiniCluster() outside try block and the test passes. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: 12783-10.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[
https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262865#comment-14262865
]
Hadoop QA commented on HBASE-12796:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689784/HBASE-12796.patch
against master branch at commit a90e64c6373437e109e6d84adbf11ea8bf296e53.
ATTACHMENT ID: 12689784
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 555
new or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java:[527,37]
incompatible types
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/target/generated-sources/java/org/apache/hadoop/hbase/generated/master/table_jsp.java:[194,47]
cannot find symbol
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on
project hbase-server: Compilation failure: Compilation failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java:[527,37]
incompatible types
[ERROR] required: org.apache.hadoop.hbase.client.HBaseAdmin
[ERROR] found:org.apache.hadoop.hbase.client.Admin
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/target/generated-sources/java/org/apache/hadoop/hbase/generated/master/table_jsp.java:[194,47]
cannot find symbol
[ERROR] symbol: method getRegionLocations()
[ERROR] location: variable table of type org.apache.hadoop.hbase.client.Table
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn goals -rf :hbase-server
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12283//console
This message is automatically generated.
Clean up HTable and HBaseAdmin deprecated constructor usage
---
Key: HBASE-12796
URL: https://issues.apache.org/jira/browse/HBASE-12796
Project: HBase
Issue Type: Improvement
Reporter: Jurriaan Mous
Attachments: HBASE-12796.patch
The are a lot of calls to the deprecated calls to HBaseAdmin and HTable
constructors in many tests and even some in production code.
This issue is about removing most of them so most code does not create
redundant Connections and underlying pools and RPC clients. Hopefully making
the tests more lightweight.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[
https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262978#comment-14262978
]
Hadoop QA commented on HBASE-12783:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689793/HBASE-12783I.patch
against master branch at commit a90e64c6373437e109e6d84adbf11ea8bf296e53.
ATTACHMENT ID: 12689793
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12286//console
This message is automatically generated.
Create efficient RegionLocator implementation
-
Key: HBASE-12783
URL: https://issues.apache.org/jira/browse/HBASE-12783
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: HBASE-12783.patch, HBASE-12783E.patch,
HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch,
HBASE-12783I.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch
A new HRegionLocator that only implements RegionLocator functionality will be
more efficient to instantiate than a full HTable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
[
https://issues.apache.org/jira/browse/HBASE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-12716:
---
Fix Version/s: 1.0.0
Integrated to 1.0 branch as well - otherwise this would be a regression from
0.98.10
A bug in RegionSplitter.UniformSplit algorithm
--
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.0.0, 0.98.6
Reporter: Weichen Ye
Assignee: Weichen Ye
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12716-v2.patch, HBASE-12716.patch
Welcome to the review board: https://reviews.apache.org/r/29424/diff/#
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm
meet an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
{code}
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
{code}
We will get the ERROR:
{code}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
{code}
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
review board:https://reviews.apache.org/r/29424/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12798) Map Reduce jobs should not create Tables in setConf()
[ https://issues.apache.org/jira/browse/HBASE-12798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12798: --- Description: setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableOutputFormatBase Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner -TableInputFormatBase (this requires changing documentation and further thought about how users ought to close connections) was: setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableInputFormatBase - MultiTableInputFormat Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner -TableInputFormatBase (this requires changing documentation and further thought about how users ought to close connections) Map Reduce jobs should not create Tables in setConf() - Key: HBASE-12798 URL: https://issues.apache.org/jira/browse/HBASE-12798 Project: HBase Issue Type: Bug Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 1.0.0, 2.0.0 setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableOutputFormatBase Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner -TableInputFormatBase (this requires changing documentation and further thought about how users ought to close connections) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12798) Map Reduce jobs should not create Tables in setConf()
[ https://issues.apache.org/jira/browse/HBASE-12798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12798: --- Description: setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableOutputFormatBase - TableInputFormatBase Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner -TableInputFormatBase (this requires changing documentation and further thought about how users ought to close connections) was: setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableOutputFormatBase Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner -TableInputFormatBase (this requires changing documentation and further thought about how users ought to close connections) Map Reduce jobs should not create Tables in setConf() - Key: HBASE-12798 URL: https://issues.apache.org/jira/browse/HBASE-12798 Project: HBase Issue Type: Bug Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 1.0.0, 2.0.0 setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableOutputFormatBase - TableInputFormatBase Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner -TableInputFormatBase (this requires changing documentation and further thought about how users ought to close connections) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12797) ReplicationSyncUp will return before recovered queue transfered with default configuration
cuijianwei created HBASE-12797:
--
Summary: ReplicationSyncUp will return before recovered queue
transfered with default configuration
Key: HBASE-12797
URL: https://issues.apache.org/jira/browse/HBASE-12797
Project: HBase
Issue Type: Improvement
Components: Replication
Affects Versions: 0.99.2
Reporter: cuijianwei
Priority: Minor
ReplicationSyncUp will check all old sources has been removed every 10 seconds
(ReplicationSyncUp.SLEEP_TIME):
{code}
int numberOfOldSource = 1; // default wait once
while (numberOfOldSource 0) {
Thread.sleep(SLEEP_TIME);
numberOfOldSource = manager.getOldSources().size();
}
{code}
However, the default replication.sleep.before.failover is 30 seconds, which
will make NodeFailoverWorker sleep at least 30 seconds before transferring
recover queue. Therefore, ReplicationSyncUp will do nothing and return directly
with default configuration.
When using ReplicationSyncUp, the goal tends to be syncing-up the
not-replicated data to peer cluster as soon as possible, is it reasonable to
set replication.sleep.before.failover to 0 in this situation, and also make
sure ReplicationQueues.getListOfReplicators().size() = 0 before returned?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12798) Map Reduce jobs should not create Tables in setConf()
Solomon Duskis created HBASE-12798: -- Summary: Map Reduce jobs should not create Tables in setConf() Key: HBASE-12798 URL: https://issues.apache.org/jira/browse/HBASE-12798 Project: HBase Issue Type: Bug Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 1.0.0, 2.0.0 setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableInputFormatBase - MultiTableInputFormat Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner -TableInputFormatBase (this requires changing documentation and further thought about how users ought to close connections) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12783: --- Attachment: HBASE-12783I.patch Let's see if I got it this time. The last incarnation looks like it was created from a bad local copy of the repo. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[ https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viet Nguyen updated HBASE-12796: Issue Type: Brainstorming (was: Improvement) Clean up HTable and HBaseAdmin deprecated constructor usage --- Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Brainstorming Reporter: Jurriaan Mous Attachments: HBASE-12796-v1.patch, HBASE-12796.patch The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[ https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viet Nguyen updated HBASE-12796: Issue Type: Improvement (was: Brainstorming) Clean up HTable and HBaseAdmin deprecated constructor usage --- Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous Attachments: HBASE-12796-v1.patch, HBASE-12796.patch The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5699) Run with 1 WAL in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263355#comment-14263355 ] Enis Soztutar commented on HBASE-5699: -- Pushed to 1.0.0 as well. All new code, should not affect default stability. Run with 1 WAL in HRegionServer - Key: HBASE-5699 URL: https://issues.apache.org/jira/browse/HBASE-5699 Project: HBase Issue Type: Improvement Components: Performance, wal Reporter: binlijin Assignee: Sean Busbey Priority: Critical Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-5699.3.patch.txt, HBASE-5699.4.patch.txt, HBASE-5699_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff, HBASE-5699_disabled_and_regular_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff, HBASE-5699_write_iops_multiwal-1_1_to_200_threads.tiff, HBASE-5699_write_iops_multiwal-2_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_multiwal-4_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_multiwal-6_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_upstream_1_to_200_threads.tiff, PerfHbase.txt, hbase-5699_multiwal_400-threads_stats_sync_heavy.txt, hbase-5699_total_throughput_sync_heavy.txt, results-hbase5699-upstream.txt.bz2, results-hbase5699-wals-1.txt.bz2, results-updated-hbase5699-wals-2.txt.bz2, results-updated-hbase5699-wals-4.txt.bz2, results-updated-hbase5699-wals-6.txt.bz2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-5699) Run with 1 WAL in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-5699: - Fix Version/s: 1.0.0 Run with 1 WAL in HRegionServer - Key: HBASE-5699 URL: https://issues.apache.org/jira/browse/HBASE-5699 Project: HBase Issue Type: Improvement Components: Performance, wal Reporter: binlijin Assignee: Sean Busbey Priority: Critical Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-5699.3.patch.txt, HBASE-5699.4.patch.txt, HBASE-5699_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff, HBASE-5699_disabled_and_regular_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff, HBASE-5699_write_iops_multiwal-1_1_to_200_threads.tiff, HBASE-5699_write_iops_multiwal-2_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_multiwal-4_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_multiwal-6_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_upstream_1_to_200_threads.tiff, PerfHbase.txt, hbase-5699_multiwal_400-threads_stats_sync_heavy.txt, hbase-5699_total_throughput_sync_heavy.txt, results-hbase5699-upstream.txt.bz2, results-hbase5699-wals-1.txt.bz2, results-updated-hbase5699-wals-2.txt.bz2, results-updated-hbase5699-wals-4.txt.bz2, results-updated-hbase5699-wals-6.txt.bz2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12750) getRequestsCount() in ClusterStatus returns total number of request
[ https://issues.apache.org/jira/browse/HBASE-12750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263359#comment-14263359 ] Enis Soztutar commented on HBASE-12750: --- Pushed to 1.0.0 as well. getRequestsCount() in ClusterStatus returns total number of request --- Key: HBASE-12750 URL: https://issues.apache.org/jira/browse/HBASE-12750 Project: HBase Issue Type: Bug Components: Client Affects Versions: 2.0.0, 0.98.6 Reporter: Weichen Ye Assignee: Weichen Ye Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12750.patch getRequestsCount() is a function in org.apache.hadoop.hbase.ClusterStatus in HBase Client compoent. The comment says that it is for us to get the number of requests since last report, but the code inside this function return the number of requests since region server start. Here we should change the code count += e.getValue().getTotalNumberOfRequests(); to count += e.getValue().getNumberOfRequests(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12750) getRequestsCount() in ClusterStatus returns total number of request
[ https://issues.apache.org/jira/browse/HBASE-12750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12750: -- Fix Version/s: 1.0.0 getRequestsCount() in ClusterStatus returns total number of request --- Key: HBASE-12750 URL: https://issues.apache.org/jira/browse/HBASE-12750 Project: HBase Issue Type: Bug Components: Client Affects Versions: 2.0.0, 0.98.6 Reporter: Weichen Ye Assignee: Weichen Ye Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12750.patch getRequestsCount() is a function in org.apache.hadoop.hbase.ClusterStatus in HBase Client compoent. The comment says that it is for us to get the number of requests since last report, but the code inside this function return the number of requests since region server start. Here we should change the code count += e.getValue().getTotalNumberOfRequests(); to count += e.getValue().getNumberOfRequests(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12763) Make it so there must be WALs for a server to be marked dead
[ https://issues.apache.org/jira/browse/HBASE-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263365#comment-14263365 ] Enis Soztutar commented on HBASE-12763: --- [~saint@gmail.com] do we need this in 1.0.0? Make it so there must be WALs for a server to be marked dead Key: HBASE-12763 URL: https://issues.apache.org/jira/browse/HBASE-12763 Project: HBase Issue Type: Sub-task Components: wal Reporter: stack Assignee: stack Fix For: 2.0.0, 1.1.0 Attachments: 12746-v2-master-and-098.patch The patch for this issue is a subset of the patch attached to the parent. The parent solves a 1.0.0-specific issue but part of the patch needs applying to 0.98 and to master to fix an issue where Master on startup would think it was joining a cluster rather than undergoing a fresh start just because it came across a directory named for a server that was once running (the patch checks if the dir has WALs and if none, does not think the server a dead server). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12732) Log messages in FileLink$FileLinkInputStream#tryOpen are reversed
[
https://issues.apache.org/jira/browse/HBASE-12732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263357#comment-14263357
]
Enis Soztutar commented on HBASE-12732:
---
Pushed to 1.0.0 as well.
Log messages in FileLink$FileLinkInputStream#tryOpen are reversed
-
Key: HBASE-12732
URL: https://issues.apache.org/jira/browse/HBASE-12732
Project: HBase
Issue Type: Bug
Components: hbase
Reporter: Tobi Vollebregt
Priority: Trivial
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12737.patch
The log message in `tryOpen` is reversed:
{code:java}
if (currentPath != null) {
LOG.debug(link open path= + path);
} else {
LOG.trace(link switch from path= + currentPath + to path=
+ path);
}
{code}
By reading the logic, we can see that when switching paths (currentPath !=
null) it will log that it opened a path, and when opening a path, it will
print that it switched from path=null to the new path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12732) Log messages in FileLink$FileLinkInputStream#tryOpen are reversed
[
https://issues.apache.org/jira/browse/HBASE-12732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12732:
--
Fix Version/s: 1.0.0
Log messages in FileLink$FileLinkInputStream#tryOpen are reversed
-
Key: HBASE-12732
URL: https://issues.apache.org/jira/browse/HBASE-12732
Project: HBase
Issue Type: Bug
Components: hbase
Reporter: Tobi Vollebregt
Priority: Trivial
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12737.patch
The log message in `tryOpen` is reversed:
{code:java}
if (currentPath != null) {
LOG.debug(link open path= + path);
} else {
LOG.trace(link switch from path= + currentPath + to path=
+ path);
}
{code}
By reading the logic, we can see that when switching paths (currentPath !=
null) it will log that it opened a path, and when opening a path, it will
print that it switched from path=null to the new path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12711) Fix new findbugs warnings in hbase-thrift module
[ https://issues.apache.org/jira/browse/HBASE-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263351#comment-14263351 ] Enis Soztutar commented on HBASE-12711: --- Pushed to 1.0.0 as well. Fix new findbugs warnings in hbase-thrift module Key: HBASE-12711 URL: https://issues.apache.org/jira/browse/HBASE-12711 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Srikanth Srungarapu Priority: Minor Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-12711.patch, HBASE-12711_v2.patch, HBASE-12711_v3.patch From https://builds.apache.org/job/PreCommit-HBASE-Build/12121/artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html , there were 5 findbugs warnings introduced. This issue fixes the new warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12697) Don't use RegionLocationFinder if localityCost == 0
[ https://issues.apache.org/jira/browse/HBASE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263361#comment-14263361 ] Enis Soztutar commented on HBASE-12697: --- Pushed to 1.0.0 as well. Don't use RegionLocationFinder if localityCost == 0 --- Key: HBASE-12697 URL: https://issues.apache.org/jira/browse/HBASE-12697 Project: HBase Issue Type: Bug Affects Versions: 0.98.9 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12697.patch Clusters with lots of reference files will be un-able to balance since the RegionLocationFinder tries to open every reference file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12697) Don't use RegionLocationFinder if localityCost == 0
[ https://issues.apache.org/jira/browse/HBASE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263362#comment-14263362 ] Elliott Clark commented on HBASE-12697: --- Thanks [~enis] Don't use RegionLocationFinder if localityCost == 0 --- Key: HBASE-12697 URL: https://issues.apache.org/jira/browse/HBASE-12697 Project: HBase Issue Type: Bug Affects Versions: 0.98.9 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12697.patch Clusters with lots of reference files will be un-able to balance since the RegionLocationFinder tries to open every reference file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12697) Don't use RegionLocationFinder if localityCost == 0
[ https://issues.apache.org/jira/browse/HBASE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12697: -- Fix Version/s: 1.0.0 Don't use RegionLocationFinder if localityCost == 0 --- Key: HBASE-12697 URL: https://issues.apache.org/jira/browse/HBASE-12697 Project: HBase Issue Type: Bug Affects Versions: 0.98.9 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12697.patch Clusters with lots of reference files will be un-able to balance since the RegionLocationFinder tries to open every reference file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263420#comment-14263420 ] Hudson commented on HBASE-12783: FAILURE: Integrated in HBase-1.1 #46 (See [https://builds.apache.org/job/HBase-1.1/46/]) HBASE-12783 Create efficient RegionLocator implementation (Solomon Duskis) (tedyu: rev 633734a82d87de576f5c32ef460cfc5dd29dc655) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HRegionLocator.java * hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultiTableInputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSizeCalculator.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Task Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 1.1.0 Attachments: 12783-10.patch, 12783-11.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12783K.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263423#comment-14263423 ] Hudson commented on HBASE-12783: SUCCESS: Integrated in HBase-TRUNK #5985 (See [https://builds.apache.org/job/HBase-TRUNK/5985/]) HBASE-12783 Create efficient RegionLocator implementation (Solomon Duskis) (tedyu: rev ac95cc1fbb951bb9db96f2738f621d1d7cd45739) * hbase-server/src/test/java/org/apache/hadoop/hbase/TestRegionRebalancing.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HRegionLocator.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultiTableInputFormat.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java * hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/Import.java * hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSizeCalculator.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java * hbase-client/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestBulkLoad.java Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Task Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 1.1.0 Attachments: 12783-10.patch, 12783-11.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12783K.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12719) Add test WAL provider to quantify FSHLog overhead in the absence of HDFS.
[ https://issues.apache.org/jira/browse/HBASE-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263418#comment-14263418 ] Hudson commented on HBASE-12719: SUCCESS: Integrated in HBase-1.0 #628 (See [https://builds.apache.org/job/HBase-1.0/628/]) HBASE-12719 test provider to remove just the hdfs interactions from fshlog. (enis: rev bc464cc745cd3d021eb9072fc40c3a0543e247f1) * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/IOTestProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DisabledWALProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java Add test WAL provider to quantify FSHLog overhead in the absence of HDFS. - Key: HBASE-12719 URL: https://issues.apache.org/jira/browse/HBASE-12719 Project: HBase Issue Type: Improvement Components: test, wal Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-12719.1.patch.txt, HBASE-12719.2.patch.txt, HBASE-12719_comparison_chart.tiff Discussion in HBASE-5699 included showing the max throughput for our WAL benchmark (using the DisabledWALProvider) compared to when we are actually dealing with synchronization and talking with HDFS. What we can't see in that comparison is how much of the (sizable) gap is due to the coordination done in FSHLog to deal with multi-threading and how much of it is due to talking to HDFS. Make a test-only provider that we can use to isolate the cost for HDFS appends, HDFS flush, and file rolling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263440#comment-14263440 ] Hudson commented on HBASE-12783: FAILURE: Integrated in HBase-1.1 #47 (See [https://builds.apache.org/job/HBase-1.1/47/]) HBASE-12783 Addendum removes extraneous import (tedyu: rev 44f392a0ccb40530b16ce01e721bb9d68bfaeb55) * hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Task Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 1.1.0 Attachments: 12783-10.patch, 12783-11.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12783K.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12716) A bug in RegionSplitter.UniformSplit algorithm
[
https://issues.apache.org/jira/browse/HBASE-12716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262833#comment-14262833
]
Hudson commented on HBASE-12716:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #737 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/737/])
HBASE-12716 A bug in RegionSplitter.UniformSplit algorithm (Weichen Ye)
(apurtell: rev 74f303ba2482b2f0c81260ee41cd0f94ef3105d8)
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
* hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitter.java
A bug in RegionSplitter.UniformSplit algorithm
--
Key: HBASE-12716
URL: https://issues.apache.org/jira/browse/HBASE-12716
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.0.0, 0.98.6
Reporter: Weichen Ye
Assignee: Weichen Ye
Fix For: 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12716-v2.patch, HBASE-12716.patch
Welcome to the review board: https://reviews.apache.org/r/29424/diff/#
I`m working for another issues HBASE-12590 and trying to use the UniformSplit
algorithm in RegionSplitter. When the last bytes of start key and end key are
adjacent in alphabetical order or ASCII order, the UniformSplit algorithm
meet an NPE.
Like startkey: aaa, endkey :aab
startkey: endkey: 1112
For example, we write this simple test code:
{code}
import org.apache.hadoop.hbase.util.RegionSplitter.UniformSplit;
..
byte[] a1 = { 'a', 'a', 'a' };
byte[] a2 = { 'a', 'a', 'b' };
UniformSplit us = new UniformSplit();
byte[] mid = us.split(a1, a2);
..
{code}
We will get the ERROR:
{code}
Exception in thread main java.lang.NullPointerException
at
org.apache.hadoop.hbase.util.RegionSplitter$UniformSplit.split(RegionSplitter.java:986)
{code}
We hope this algorithm should be able to calculate the split point with an
additional byte. for example:
aaa and aab, split point= aaaP
and 1112, split point =P
review board:https://reviews.apache.org/r/29424/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12641) Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster
[ https://issues.apache.org/jira/browse/HBASE-12641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262832#comment-14262832 ] Hudson commented on HBASE-12641: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #737 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/737/]) HBASE-12641 Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster (Liu Shaohui) (apurtell: rev efc49a745fa198e8f5ed9abe76392e6fff836d75) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster Key: HBASE-12641 URL: https://issues.apache.org/jira/browse/HBASE-12641 Project: HBase Issue Type: Improvement Components: Zookeeper Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12641-v1.diff Currently in a secure cluster, only the master/regionserver kerberos user can manage the znode of hbase. But he master/regionserver kerberos user is for rpc connection and we usually use another super user to manage the cluster. In some special scenarios, we need to manage the data of znode with the supper user. eg: a, To get the data of the znode for debugging. b, HBASE-8253: We need to delete the znode for the corrupted hlog to avoid it block the replication. So we grant all permissions of hbase zookeeper node to hbase superuser during creating these znodes. Suggestions are welcomed. [~apurtell] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[ https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12796: -- Attachment: HBASE-12796.patch - Removed almost all deprecated HTable calls - Replaced HTable with Table interface were possible. - Introduced local Connection creation where necessary and closed them were necessary. - Removed almost all direct HBaseAdmin creation - Fixed a few non TableName calls where necessary and replaced some RegionLocators and TableDescription getters to Admin calls where possible to save Table creation. I will leave the few remaining calls which needs some more code change as I underestimated a bit of the task but wanted to contribute at least the major part of the work. Some Test fails are possible since I did not test them all. Clean up HTable and HBaseAdmin deprecated constructor usage --- Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous Attachments: HBASE-12796.patch The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[
https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262907#comment-14262907
]
Hadoop QA commented on HBASE-12783:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689791/HBASE-12783H.patch
against master branch at commit a90e64c6373437e109e6d84adbf11ea8bf296e53.
ATTACHMENT ID: 12689791
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2133
new or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12285//console
This message is automatically generated.
Create efficient RegionLocator implementation
-
Key: HBASE-12783
URL: https://issues.apache.org/jira/browse/HBASE-12783
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: HBASE-12783.patch, HBASE-12783E.patch,
HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch,
HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch
A new HRegionLocator that only implements RegionLocator functionality will be
more efficient to instantiate than a full HTable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[ https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12796: -- Attachment: HBASE-12796-v1.patch Seems I have created a faulty patch. New version. Clean up HTable and HBaseAdmin deprecated constructor usage --- Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous Attachments: HBASE-12796-v1.patch, HBASE-12796.patch The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[ https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jurriaan Mous updated HBASE-12796: -- Status: Patch Available (was: Open) Clean up HTable and HBaseAdmin deprecated constructor usage --- Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous Attachments: HBASE-12796.patch The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12783: --- Attachment: HBASE-12783H.patch I think I fixed the last broken test. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
Jurriaan Mous created HBASE-12796: - Summary: Clean up HTable and HBaseAdmin deprecated constructor usage Key: HBASE-12796 URL: https://issues.apache.org/jira/browse/HBASE-12796 Project: HBase Issue Type: Improvement Reporter: Jurriaan Mous The are a lot of calls to the deprecated calls to HBaseAdmin and HTable constructors in many tests and even some in production code. This issue is about removing most of them so most code does not create redundant Connections and underlying pools and RPC clients. Hopefully making the tests more lightweight. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12028) Abort the RegionServer, when it's handler threads die
[ https://issues.apache.org/jira/browse/HBASE-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263316#comment-14263316 ] Hudson commented on HBASE-12028: FAILURE: Integrated in HBase-1.1 #45 (See [https://builds.apache.org/job/HBase-1.1/45/]) HBASE-12028 Abort the RegionServer, when it's handler threads die (Alicia Ying Shu) (enis: rev ecbdc45d3d68d83ee001a56b2735b5f5dc63b3e2) * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SimpleRpcSchedulerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestRpcHandlerException.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RpcSchedulerFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/BalancedQueueRpcExecutor.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java Abort the RegionServer, when it's handler threads die - Key: HBASE-12028 URL: https://issues.apache.org/jira/browse/HBASE-12028 Project: HBase Issue Type: Bug Components: regionserver Reporter: Sudarshan Kadambi Assignee: Alicia Ying Shu Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: Hbase-12028-v3.patch, Hbase-12028.patch, hbase-12028-v4.patch, hbase-12028-v5-branch-1.patch, hbase-12028-v5-master.patch, hbase-12028-v5.patch Over in HBase-11813, a user identified an issue where in all the RPC handler threads would exit with StackOverflow errors due to an unchecked recursion-terminating condition. Our clusters demonstrated the same trace. While the patch posted for HBASE-11813 got our clusters to be merry again, the breakdown surfaced some larger issues. When the RegionServer had all it's RPC handler threads dead, it continued to have regions assigned it. Clearly, it wouldn't be able to serve reads and writes on those regions. A second issue was that when a user tried to disable or drop a table, the master would try to communicate to the regionserver for region unassignment. Since the same handler threads seem to be used for master - RS communication as well, the master ended up hanging on the RS indefinitely. Eventually, the master stopped responding to all table meta-operations. A handler thread should never exit, and if it does, it seems like the more prudent thing to do would be for the RS to abort. This way, at least recovery can be undertaken and the regions could be reassigned elsewhere. I also think that the master-RS communication should get its own exclusive threadpool, but I'll wait until this issue has been sufficiently discussed before opening an issue ticket for that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12788) Promote Abortable to LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-12788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-12788: -- Assignee: Enis Soztutar (was: Andrew Purtell) Promote Abortable to LimitedPrivate --- Key: HBASE-12788 URL: https://issues.apache.org/jira/browse/HBASE-12788 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Enis Soztutar Labels: phoenix Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 After HBASE-12028, we will be passing Abortable around in a LimitedPrivate interface, with the intent of its use in implementors and subclasses of related types (RPC schedulers). Therefore we should promote Abortable to LimitedPrivate as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12799) ITAG fails java.lang.RuntimeException: org.apache.hadoop.hbase.DistributedHBaseCluster@49ef4b31 not an instance of org.apache.hadoop.hbase.MiniHBaseCluster
Enis Soztutar created HBASE-12799:
-
Summary: ITAG fails java.lang.RuntimeException:
org.apache.hadoop.hbase.DistributedHBaseCluster@49ef4b31 not an instance of
org.apache.hadoop.hbase.MiniHBaseCluster
Key: HBASE-12799
URL: https://issues.apache.org/jira/browse/HBASE-12799
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0, 2.0.0, 1.1.0
If the table does not exists:
{code}
2015-01-02 16:20:35,764 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.RuntimeException:
org.apache.hadoop.hbase.DistributedHBaseCluster@7e02856c not an instance of
org.apache.hadoop.hbase.MiniHBaseCluster
at
org.apache.hadoop.hbase.HBaseTestingUtility.getMiniHBaseCluster(HBaseTestingUtility.java:1022)
at
org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseCluster(HBaseTestingUtility.java:2569)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3055)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3011)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1304)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1334)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1254)
at
org.apache.hadoop.hbase.TestAcidGuarantees.createTableIfMissing(TestAcidGuarantees.java:80)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:264)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.runTestFromCommandLine(IntegrationTestAcidGuarantees.java:58)
at
org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:112)
at
org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuarantees.main(IntegrationTestAcidGuarantees.java:115)
{code}
[~jmhsieh] FYI.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12071) Separate out thread pool for Master - RegionServer communication
[
https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263320#comment-14263320
]
Hadoop QA commented on HBASE-12071:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689893/HBASE-12071.v2-master.patch
against master branch at commit 820f629423f21fbd1dcc7a383955443a2595fd5d.
ATTACHMENT ID: 12689893
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12292//console
This message is automatically generated.
Separate out thread pool for Master - RegionServer communication
--
Key: HBASE-12071
URL: https://issues.apache.org/jira/browse/HBASE-12071
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Sudarshan Kadambi
Assignee: Stephen Yuan Jiang
Fix For: 2.0.0
Attachments: HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch
Over in HBASE-12028, there is a discussion about the case of a RegionServer
still being alive despite all its handler threads being dead. One outcome of
this is that the Master is left hanging on the RS for completion of various
operations - such as region un-assignment when a table is disabled. Does it
make sense to create a separate thread pool for communication between the
Master and the RS? This addresses not just the case of the RPC handler
threads terminating but also long-running queries or co-processor executions
holding up master operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12071) Separate out thread pool for Master - RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263326#comment-14263326 ] Enis Soztutar commented on HBASE-12071: --- [~syuanjiang] do you mind rebasing the patch for master? Separate out thread pool for Master - RegionServer communication -- Key: HBASE-12071 URL: https://issues.apache.org/jira/browse/HBASE-12071 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Sudarshan Kadambi Assignee: Stephen Yuan Jiang Fix For: 2.0.0 Attachments: HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch Over in HBASE-12028, there is a discussion about the case of a RegionServer still being alive despite all its handler threads being dead. One outcome of this is that the Master is left hanging on the RS for completion of various operations - such as region un-assignment when a table is disabled. Does it make sense to create a separate thread pool for communication between the Master and the RS? This addresses not just the case of the RPC handler threads terminating but also long-running queries or co-processor executions holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12071) Separate out thread pool for Master - RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263313#comment-14263313 ] Enis Soztutar commented on HBASE-12071: --- +1. Will retry hadoopqa. Separate out thread pool for Master - RegionServer communication -- Key: HBASE-12071 URL: https://issues.apache.org/jira/browse/HBASE-12071 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Sudarshan Kadambi Assignee: Stephen Yuan Jiang Fix For: 2.0.0 Attachments: HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch Over in HBASE-12028, there is a discussion about the case of a RegionServer still being alive despite all its handler threads being dead. One outcome of this is that the Master is left hanging on the RS for completion of various operations - such as region un-assignment when a table is disabled. Does it make sense to create a separate thread pool for communication between the Master and the RS? This addresses not just the case of the RPC handler threads terminating but also long-running queries or co-processor executions holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12071) Separate out thread pool for Master - RegionServer communication
[ https://issues.apache.org/jira/browse/HBASE-12071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12071: -- Attachment: HBASE-12071.v2-master.patch Reattaching the patch for hadoopqa. Separate out thread pool for Master - RegionServer communication -- Key: HBASE-12071 URL: https://issues.apache.org/jira/browse/HBASE-12071 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Sudarshan Kadambi Assignee: Stephen Yuan Jiang Fix For: 2.0.0 Attachments: HBASE-12071.v2-master.patch, HBASE-12071.v2-master.patch Over in HBASE-12028, there is a discussion about the case of a RegionServer still being alive despite all its handler threads being dead. One outcome of this is that the Master is left hanging on the RS for completion of various operations - such as region un-assignment when a table is disabled. Does it make sense to create a separate thread pool for communication between the Master and the RS? This addresses not just the case of the RPC handler threads terminating but also long-running queries or co-processor executions holding up master operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12783: --- Attachment: 12783-11.patch Patch v11 fixes javadoc warnings Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: 12783-10.patch, 12783-11.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12783K.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12788) Promote Abortable to LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-12788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263324#comment-14263324 ] Enis Soztutar commented on HBASE-12788: --- Do you mind me hijacking this? I already had a patch. Promote Abortable to LimitedPrivate --- Key: HBASE-12788 URL: https://issues.apache.org/jira/browse/HBASE-12788 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Andrew Purtell Labels: phoenix Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 After HBASE-12028, we will be passing Abortable around in a LimitedPrivate interface, with the intent of its use in implementors and subclasses of related types (RPC schedulers). Therefore we should promote Abortable to LimitedPrivate as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12788) Promote Abortable to LimitedPrivate
[ https://issues.apache.org/jira/browse/HBASE-12788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263335#comment-14263335 ] Andrew Purtell commented on HBASE-12788: Go for it Promote Abortable to LimitedPrivate --- Key: HBASE-12788 URL: https://issues.apache.org/jira/browse/HBASE-12788 Project: HBase Issue Type: Task Reporter: Andrew Purtell Assignee: Enis Soztutar Labels: phoenix Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 After HBASE-12028, we will be passing Abortable around in a LimitedPrivate interface, with the intent of its use in implementors and subclasses of related types (RPC schedulers). Therefore we should promote Abortable to LimitedPrivate as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12697) Don't use RegionLocationFinder if localityCost == 0
[ https://issues.apache.org/jira/browse/HBASE-12697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263400#comment-14263400 ] Hudson commented on HBASE-12697: SUCCESS: Integrated in HBase-1.0 #627 (See [https://builds.apache.org/job/HBase-1.0/627/]) HBASE-12697 Don't use RegionLocationFinder if localityCost == 0 (enis: rev 1c4da662788676c010d17da6e020171b78868573) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java Don't use RegionLocationFinder if localityCost == 0 --- Key: HBASE-12697 URL: https://issues.apache.org/jira/browse/HBASE-12697 Project: HBase Issue Type: Bug Affects Versions: 0.98.9 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12697.patch Clusters with lots of reference files will be un-able to balance since the RegionLocationFinder tries to open every reference file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12641) Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster
[ https://issues.apache.org/jira/browse/HBASE-12641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263399#comment-14263399 ] Hudson commented on HBASE-12641: SUCCESS: Integrated in HBase-1.0 #627 (See [https://builds.apache.org/job/HBase-1.0/627/]) HBASE-12641 Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster (Liu Shaohui) (enis: rev efc3a85473fdd21eadc3e7916907bcff6196f225) * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster Key: HBASE-12641 URL: https://issues.apache.org/jira/browse/HBASE-12641 Project: HBase Issue Type: Improvement Components: Zookeeper Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12641-v1.diff Currently in a secure cluster, only the master/regionserver kerberos user can manage the znode of hbase. But he master/regionserver kerberos user is for rpc connection and we usually use another super user to manage the cluster. In some special scenarios, we need to manage the data of znode with the supper user. eg: a, To get the data of the znode for debugging. b, HBASE-8253: We need to delete the znode for the corrupted hlog to avoid it block the replication. So we grant all permissions of hbase zookeeper node to hbase superuser during creating these znodes. Suggestions are welcomed. [~apurtell] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12732) Log messages in FileLink$FileLinkInputStream#tryOpen are reversed
[
https://issues.apache.org/jira/browse/HBASE-12732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263403#comment-14263403
]
Hudson commented on HBASE-12732:
SUCCESS: Integrated in HBase-1.0 #627 (See
[https://builds.apache.org/job/HBase-1.0/627/])
HBASE-12732 Log messages in FileLink#tryOpen are reversed (Tobi Vollebregt)
(enis: rev 15c43815ecf250fe0a7c36a410ce41beba2ef93d)
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java
Log messages in FileLink$FileLinkInputStream#tryOpen are reversed
-
Key: HBASE-12732
URL: https://issues.apache.org/jira/browse/HBASE-12732
Project: HBase
Issue Type: Bug
Components: hbase
Reporter: Tobi Vollebregt
Priority: Trivial
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12737.patch
The log message in `tryOpen` is reversed:
{code:java}
if (currentPath != null) {
LOG.debug(link open path= + path);
} else {
LOG.trace(link switch from path= + currentPath + to path=
+ path);
}
{code}
By reading the logic, we can see that when switching paths (currentPath !=
null) it will log that it opened a path, and when opening a path, it will
print that it switched from path=null to the new path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12718) Convert TestAcidGuarantees from a unit test to an integration test
[ https://issues.apache.org/jira/browse/HBASE-12718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263397#comment-14263397 ] Hudson commented on HBASE-12718: SUCCESS: Integrated in HBase-1.0 #627 (See [https://builds.apache.org/job/HBase-1.0/627/]) HBASE-12718 Convert TestAcidGuarantees from a unit test to an integration test (enis: rev b2b3b55dfbcfd4d5561570d019a1ebdf15fb52bd) * hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestAcidGuarantees.java * hbase-server/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java Convert TestAcidGuarantees from a unit test to an integration test -- Key: HBASE-12718 URL: https://issues.apache.org/jira/browse/HBASE-12718 Project: HBase Issue Type: Bug Components: hbase, integration tests, test Affects Versions: 1.0.0, 2.0.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 1.0.0, 2.0.0, hbase-11339, 1.1.0 Attachments: hbase-12718.patch TestAcidGurantees has a main method so that it can be executed via the commandline. In the past this was run and we'd use external tools to inject faults while it executed for an extended period of time. We've had the IT framework we'd like to use the ChaosMonkey automation with it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12711) Fix new findbugs warnings in hbase-thrift module
[ https://issues.apache.org/jira/browse/HBASE-12711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263402#comment-14263402 ] Hudson commented on HBASE-12711: SUCCESS: Integrated in HBase-1.0 #627 (See [https://builds.apache.org/job/HBase-1.0/627/]) HBASE-12711 Fix new findbugs warnings in hbase-thrift module (Srikanth) (enis: rev 4083306c3d8a0300c940859a17a55616afd4150c) * hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java * hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java * hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java Fix new findbugs warnings in hbase-thrift module Key: HBASE-12711 URL: https://issues.apache.org/jira/browse/HBASE-12711 Project: HBase Issue Type: Bug Reporter: Ted Yu Assignee: Srikanth Srungarapu Priority: Minor Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-12711.patch, HBASE-12711_v2.patch, HBASE-12711_v3.patch From https://builds.apache.org/job/PreCommit-HBASE-Build/12121/artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html , there were 5 findbugs warnings introduced. This issue fixes the new warnings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12750) getRequestsCount() in ClusterStatus returns total number of request
[ https://issues.apache.org/jira/browse/HBASE-12750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263404#comment-14263404 ] Hudson commented on HBASE-12750: SUCCESS: Integrated in HBase-1.0 #627 (See [https://builds.apache.org/job/HBase-1.0/627/]) HBASE-12750 getRequestsCount() in ClusterStatus returns total number of request (Weichen Ye) (enis: rev 9403a1b640eda5f292a5bcd2594f37f986802618) * hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java getRequestsCount() in ClusterStatus returns total number of request --- Key: HBASE-12750 URL: https://issues.apache.org/jira/browse/HBASE-12750 Project: HBase Issue Type: Bug Components: Client Affects Versions: 2.0.0, 0.98.6 Reporter: Weichen Ye Assignee: Weichen Ye Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12750.patch getRequestsCount() is a function in org.apache.hadoop.hbase.ClusterStatus in HBase Client compoent. The comment says that it is for us to get the number of requests since last report, but the code inside this function return the number of requests since region server start. Here we should change the code count += e.getValue().getTotalNumberOfRequests(); to count += e.getValue().getNumberOfRequests(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-5699) Run with 1 WAL in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263398#comment-14263398 ] Hudson commented on HBASE-5699: --- SUCCESS: Integrated in HBase-1.0 #627 (See [https://builds.apache.org/job/HBase-1.0/627/]) HBASE-5699 Adds multiple WALs per Region Server based on groups of regions. (enis: rev 2ebeddfc4276898e42e9d7cadb8092e9f72ed421) * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/RegionGroupingProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/BoundedRegionGroupingProvider.java * hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestBoundedRegionGroupingProvider.java Run with 1 WAL in HRegionServer - Key: HBASE-5699 URL: https://issues.apache.org/jira/browse/HBASE-5699 Project: HBase Issue Type: Improvement Components: Performance, wal Reporter: binlijin Assignee: Sean Busbey Priority: Critical Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-5699.3.patch.txt, HBASE-5699.4.patch.txt, HBASE-5699_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff, HBASE-5699_disabled_and_regular_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff, HBASE-5699_write_iops_multiwal-1_1_to_200_threads.tiff, HBASE-5699_write_iops_multiwal-2_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_multiwal-4_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_multiwal-6_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_upstream_1_to_200_threads.tiff, PerfHbase.txt, hbase-5699_multiwal_400-threads_stats_sync_heavy.txt, hbase-5699_total_throughput_sync_heavy.txt, results-hbase5699-upstream.txt.bz2, results-hbase5699-wals-1.txt.bz2, results-updated-hbase5699-wals-2.txt.bz2, results-updated-hbase5699-wals-4.txt.bz2, results-updated-hbase5699-wals-6.txt.bz2 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263419#comment-14263419
]
Hudson commented on HBASE-12781:
SUCCESS: Integrated in HBase-1.0 #628 (See
[https://builds.apache.org/job/HBase-1.0/628/])
HBASE-12781 thrift2 listen port will bind always to the passed command line
address (Pankaj Kumar) (enis: rev 8995d19049191ff4f6883f3d609a1aa2fddf880d)
* hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12735) Refactor TAG so it can live as unit test and as an integration test
[ https://issues.apache.org/jira/browse/HBASE-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263401#comment-14263401 ] Hudson commented on HBASE-12735: SUCCESS: Integrated in HBase-1.0 #627 (See [https://builds.apache.org/job/HBase-1.0/627/]) HBASE-12735 Refactor TestAcidGuarantees so it can live as unit test and as an integration test (enis: rev f9ce8f98e45c7b39b108384796281c0a2f9e) * hbase-server/src/test/java/org/apache/hadoop/hbase/TestAcidGuarantees.java * hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestAcidGuarantees.java Refactor TAG so it can live as unit test and as an integration test --- Key: HBASE-12735 URL: https://issues.apache.org/jira/browse/HBASE-12735 Project: HBase Issue Type: Sub-task Components: test Reporter: stack Assignee: Jonathan Hsieh Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: hbase-12735.patch Parent task moved TAG to IT wholesale. This is about keeping a bit of ACID coverage going in UT. Jon offered to refactor TAG so can live in IT and UT. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12783: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch, Solomon. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 1.1.0 Attachments: 12783-10.patch, 12783-11.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12783K.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12783: --- Fix Version/s: 1.1.0 2.0.0 Hadoop Flags: Reviewed Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 1.1.0 Attachments: 12783-10.patch, 12783-11.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12783K.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-12783: --- Issue Type: Task (was: Bug) Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Task Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 2.0.0, 1.1.0 Attachments: 12783-10.patch, 12783-11.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12783K.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[
https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262920#comment-14262920
]
Hadoop QA commented on HBASE-12796:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689787/HBASE-12796-v1.patch
against master branch at commit a90e64c6373437e109e6d84adbf11ea8bf296e53.
ATTACHMENT ID: 12689787
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 555
new or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ public int loadTable(final Table t, final byte[][] f, byte[] value,
boolean writeToWAL) throws IOException {
+HRegionLocation hloc =
connection.getRegionLocator(TableName.META_TABLE_NAME).getRegionLocation(Bytes.toBytes());
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.security.access.TestNamespaceCommands
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildBase
org.apache.hadoop.hbase.security.access.TestCellACLWithMultipleVersions
org.apache.hadoop.hbase.security.visibility.TestEnforcingScanLabelGenerator
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDistributedLogReplay
org.apache.hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFiles
org.apache.hadoop.hbase.security.access.TestScanEarlyTermination
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesUseSecurityEndPoint
org.apache.hadoop.hbase.replication.TestReplicationSmallTests
org.apache.hadoop.hbase.replication.TestMasterReplication
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat2
org.apache.hadoop.hbase.security.access.TestCellACLs
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildHole
org.apache.hadoop.hbase.util.TestHBaseFsck
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithACL
org.apache.hadoop.hbase.security.visibility.TestDefaultScanLabelGeneratorStack
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
org.apache.hadoop.hbase.security.access.TestAccessController
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
org.apache.hadoop.hbase.security.visibility.TestVisibilityWithCheckAuths
org.apache.hadoop.hbase.security.access.TestAccessControlFilter
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12284//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12284//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12284//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12284//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12284//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12284//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12284//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12284//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
[jira] [Commented] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263228#comment-14263228 ] Hudson commented on HBASE-12270: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #738 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/738/]) Revert HBASE-12270 A bug in the bucket cache, with cache blocks on write enabled (Liu Shaohui) -- ADDENDUM (stack: rev bb7074b3d83a87b1e0dd4c8cfe0702e621f23ea6) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java Revert HBASE-12270 A bug in the bucket cache, with cache blocks on write enabled (Liu Shaohui) (stack: rev 9b25e850df5b915305194e21ae4a47d0ab029034) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java A bug in the bucket cache, with cache blocks on write enabled - Key: HBASE-12270 URL: https://issues.apache.org/jira/browse/HBASE-12270 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6.1 Environment: I can reproduce it on a simple 2 node cluster, one running the master and another running a RS. I was testing on ec2. I used the following configurations for the cluster. hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log hbase-site: hbase.bucketcache.ioengine=offheap hbase.bucketcache.size=4196 hbase.rs.cacheblocksonwrite=true hfile.block.index.cacheonwrite=true hfile.block.bloom.cacheonwrite=true Reporter: Khaled Elmeleegy Assignee: Liu Shaohui Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java In my experiments, I have writers streaming their output to HBase. The reader powers a web page and does this scatter/gather, where it reads 1000 keys written last and passes them the the front end. With this workload, I get the exception below at the region server. Again, I am using HBAse (0.98.6.1). Any help is appreciated. 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:236) at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263231#comment-14263231 ] Hudson commented on HBASE-12270: FAILURE: Integrated in HBase-0.98 #774 (See [https://builds.apache.org/job/HBase-0.98/774/]) Revert HBASE-12270 A bug in the bucket cache, with cache blocks on write enabled (Liu Shaohui) -- ADDENDUM (stack: rev bb7074b3d83a87b1e0dd4c8cfe0702e621f23ea6) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java Revert HBASE-12270 A bug in the bucket cache, with cache blocks on write enabled (Liu Shaohui) (stack: rev 9b25e850df5b915305194e21ae4a47d0ab029034) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheOnWrite.java A bug in the bucket cache, with cache blocks on write enabled - Key: HBASE-12270 URL: https://issues.apache.org/jira/browse/HBASE-12270 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6.1 Environment: I can reproduce it on a simple 2 node cluster, one running the master and another running a RS. I was testing on ec2. I used the following configurations for the cluster. hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log hbase-site: hbase.bucketcache.ioengine=offheap hbase.bucketcache.size=4196 hbase.rs.cacheblocksonwrite=true hfile.block.index.cacheonwrite=true hfile.block.bloom.cacheonwrite=true Reporter: Khaled Elmeleegy Assignee: Liu Shaohui Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java In my experiments, I have writers streaming their output to HBase. The reader powers a web page and does this scatter/gather, where it reads 1000 keys written last and passes them the the front end. With this workload, I get the exception below at the region server. Again, I am using HBAse (0.98.6.1). Any help is appreciated. 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:236) at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12786) List the zookeeper quorum nodes on different lines in the Master web ui
[
https://issues.apache.org/jira/browse/HBASE-12786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263240#comment-14263240
]
Hadoop QA commented on HBASE-12786:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689870/0001-HBASE-12786-Change-ZK-Quorum-display-list-each-serve.patch
against master branch at commit a90e64c6373437e109e6d84adbf11ea8bf296e53.
ATTACHMENT ID: 12689870
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testRegionTransitionOperations(TestMasterObserver.java:1605)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12288//console
This message is automatically generated.
List the zookeeper quorum nodes on different lines in the Master web ui
---
Key: HBASE-12786
URL: https://issues.apache.org/jira/browse/HBASE-12786
Project: HBase
Issue Type: Task
Components: master
Affects Versions: 0.89-fb
Reporter: Rishit Shroff
Priority: Trivial
Attachments:
0001-HBASE-12786-Change-ZK-Quorum-display-list-each-serve.patch
HBase uses zookeeper to store the current state of the cluster and uses it to
maintain the overall consistency of the cluster. The zookeeper is quorum is
hosted on set of nodes (5 typically) in the same hbase cluster. The names of
these hosts are accessible in the HMaster web ui page. However, they are all
',' separated. It would be nice to list them on different lines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[
https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263210#comment-14263210
]
Hadoop QA commented on HBASE-12783:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689876/HBASE-12783J.patch
against master branch at commit a90e64c6373437e109e6d84adbf11ea8bf296e53.
ATTACHMENT ID: 12689876
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java:[126,11]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java:[721,18]
cannot find symbol
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java:[366,24]
cannot find symbol
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on
project hbase-client: Compilation failure: Compilation failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java:[126,11]
cannot find symbol
[ERROR] symbol: class HRegionLocator
[ERROR] location: class org.apache.hadoop.hbase.client.HTable
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java:[721,18]
cannot find symbol
[ERROR] symbol: class HRegionLocator
[ERROR] location: class
org.apache.hadoop.hbase.client.ConnectionManager.HConnectionImplementation
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java:[366,24]
cannot find symbol
[ERROR] symbol: class HRegionLocator
[ERROR] location: class org.apache.hadoop.hbase.client.HTable
[ERROR] - [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn goals -rf :hbase-client
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12290//console
This message is automatically generated.
Create efficient RegionLocator implementation
-
Key: HBASE-12783
URL: https://issues.apache.org/jira/browse/HBASE-12783
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: 12783-10.patch, HBASE-12783.patch, HBASE-12783E.patch,
HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch,
HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12873B.patch,
HBASE-12873C.patch, HBASE-12873D.patch
A new HRegionLocator that only implements RegionLocator functionality will be
more efficient to instantiate than a full HTable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12796) Clean up HTable and HBaseAdmin deprecated constructor usage
[
https://issues.apache.org/jira/browse/HBASE-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263234#comment-14263234
]
Hadoop QA commented on HBASE-12796:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689868/HBASE-12796-v2.patch
against master branch at commit a90e64c6373437e109e6d84adbf11ea8bf296e53.
ATTACHMENT ID: 12689868
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 555
new or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.security.visibility.TestVisibilityWithCheckAuths
org.apache.hadoop.hbase.replication.TestReplicationSyncUpTool
org.apache.hadoop.hbase.master.TestRestartCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12287//console
This message is automatically generated.
Clean up HTable and HBaseAdmin deprecated constructor usage
---
Key: HBASE-12796
URL: https://issues.apache.org/jira/browse/HBASE-12796
Project: HBase
Issue Type: Improvement
Reporter: Jurriaan Mous
Attachments: HBASE-12796-v1.patch, HBASE-12796-v2.patch,
HBASE-12796.patch
The are a lot of calls to the deprecated calls to HBaseAdmin and HTable
constructors in many tests and even some in production code.
This issue is about removing most of them so most code does not create
redundant Connections and underlying pools and RPC clients. Hopefully making
the tests more lightweight.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12782) ITBLL fails for me if generator does anything but 5M per maptask
[ https://issues.apache.org/jira/browse/HBASE-12782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12782: -- Fix Version/s: 1.0.0 ITBLL fails for me if generator does anything but 5M per maptask Key: HBASE-12782 URL: https://issues.apache.org/jira/browse/HBASE-12782 Project: HBase Issue Type: Bug Components: integration tests Affects Versions: 1.0.0 Reporter: stack Priority: Critical Fix For: 1.0.0 Anyone else seeing this? If I do an ITBLL with generator doing 5M rows per maptask, all is good -- verify passes. I've been running 5 servers and had one splot per server. So below works: HADOOP_CLASSPATH=/home/stack/conf_hbase:`/home/stack/hbase/bin/hbase classpath` ./hadoop/bin/hadoop --config ~/conf_hadoop org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList --monkey serverKilling Generator 5 500 g1.tmp or if I double the map tasks, it works: HADOOP_CLASSPATH=/home/stack/conf_hbase:`/home/stack/hbase/bin/hbase classpath` ./hadoop/bin/hadoop --config ~/conf_hadoop org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList --monkey serverKilling Generator 10 500 g2.tmp ...but if I change the 5M to 50M or 25M, Verify fails. Looking into it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12798) Map Reduce jobs should not create Tables in setConf()
[ https://issues.apache.org/jira/browse/HBASE-12798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12798: --- Description: setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableOutputFormatBase - TableInputFormatBase - TableInputFormat Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner was: setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableOutputFormatBase - TableInputFormatBase Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner -TableInputFormatBase (this requires changing documentation and further thought about how users ought to close connections) Map Reduce jobs should not create Tables in setConf() - Key: HBASE-12798 URL: https://issues.apache.org/jira/browse/HBASE-12798 Project: HBase Issue Type: Bug Reporter: Solomon Duskis Assignee: Solomon Duskis Fix For: 1.0.0, 2.0.0 setConf() gets called in many places along the Map/Reduce chain. HBase creates Tables and etc in setConf in a few places. There should be a better place to create the Table, often while creating a TableRecordReader. This issue will tackle the following classes: - TableOutputFormatBase - TableInputFormatBase - TableInputFormat Other classes that could use a look over: - HRegionPartitioner - ReplicationLogCleaner -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12783: --- Attachment: HBASE-12783K.patch Ok. HRegionLocator was added this time, so the compilation errors should be gone. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: 12783-10.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12783K.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12783) Create efficient RegionLocator implementation
[ https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Solomon Duskis updated HBASE-12783: --- Attachment: HBASE-12783J.patch Thanks for the catch [~tedyu] for the catch. I fixed it. Hopefully this is the last attempt. Create efficient RegionLocator implementation - Key: HBASE-12783 URL: https://issues.apache.org/jira/browse/HBASE-12783 Project: HBase Issue Type: Bug Affects Versions: 1.0.0, 2.0.0 Reporter: Solomon Duskis Assignee: Solomon Duskis Attachments: 12783-10.patch, HBASE-12783.patch, HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch, HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch A new HRegionLocator that only implements RegionLocator functionality will be more efficient to instantiate than a full HTable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[
https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263366#comment-14263366
]
Ted Yu commented on HBASE-12270:
Test failed in the following builds as well:
{code}
https://builds.apache.org/job/HBase-1.1/41/testReport/org.apache.hadoop.hbase.io.hfile/TestCacheOnWrite/testStoreFileCacheOnWrite_80_/
https://builds.apache.org/job/HBase-TRUNK/5983/testReport/junit/org.apache.hadoop.hbase.io.hfile/TestCacheOnWrite/testStoreFileCacheOnWrite_95_/
{code}
A bug in the bucket cache, with cache blocks on write enabled
-
Key: HBASE-12270
URL: https://issues.apache.org/jira/browse/HBASE-12270
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.11, 0.98.6.1
Environment: I can reproduce it on a simple 2 node cluster, one
running the master and another running a RS. I was testing on ec2.
I used the following configurations for the cluster.
hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G
-XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog
gc:/tmp/hbase-regionserver-gc.log
hbase-site:
hbase.bucketcache.ioengine=offheap
hbase.bucketcache.size=4196
hbase.rs.cacheblocksonwrite=true
hfile.block.index.cacheonwrite=true
hfile.block.bloom.cacheonwrite=true
Reporter: Khaled Elmeleegy
Assignee: Liu Shaohui
Priority: Critical
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff,
HBASE-12270-v2.patch, TestHBase.java, TestKey.java
In my experiments, I have writers streaming their output to HBase. The reader
powers a web page and does this scatter/gather, where it reads 1000 keys
written last and passes them the the front end. With this workload, I get the
exception below at the region server. Again, I am using HBAse (0.98.6.1). Any
help is appreciated.
2014-10-10 15:06:44,173 ERROR
[B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected
throwable object
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:236)
at
org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936)
at
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12799) ITAG fails with java.lang.RuntimeException if table does not exist
[
https://issues.apache.org/jira/browse/HBASE-12799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263394#comment-14263394
]
Hadoop QA commented on HBASE-12799:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689900/hbase-12799_v1.patch
against master branch at commit 820f629423f21fbd1dcc7a383955443a2595fd5d.
ATTACHMENT ID: 12689900
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12295//console
This message is automatically generated.
ITAG fails with java.lang.RuntimeException if table does not exist
--
Key: HBASE-12799
URL: https://issues.apache.org/jira/browse/HBASE-12799
Project: HBase
Issue Type: Test
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 1.0.0, 2.0.0, 1.1.0
Attachments: hbase-12799_v1.patch
If the table does not exists:
{code}
2015-01-02 16:20:35,764 ERROR [main] util.AbstractHBaseTool: Error running
command-line tool
java.lang.RuntimeException:
org.apache.hadoop.hbase.DistributedHBaseCluster@7e02856c not an instance of
org.apache.hadoop.hbase.MiniHBaseCluster
at
org.apache.hadoop.hbase.HBaseTestingUtility.getMiniHBaseCluster(HBaseTestingUtility.java:1022)
at
org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseCluster(HBaseTestingUtility.java:2569)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3055)
at
org.apache.hadoop.hbase.HBaseTestingUtility.waitUntilAllRegionsAssigned(HBaseTestingUtility.java:3011)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1304)
at
org.apache.hadoop.hbase.HBaseTestingUtility.createTable(HBaseTestingUtility.java:1334)
at
[jira] [Commented] (HBASE-12788) Promote Abortable to LimitedPrivate
[
https://issues.apache.org/jira/browse/HBASE-12788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263388#comment-14263388
]
Hadoop QA commented on HBASE-12788:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689898/hbase-12788_v1.patch
against master branch at commit 820f629423f21fbd1dcc7a383955443a2595fd5d.
ATTACHMENT ID: 12689898
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12294//console
This message is automatically generated.
Promote Abortable to LimitedPrivate
---
Key: HBASE-12788
URL: https://issues.apache.org/jira/browse/HBASE-12788
Project: HBase
Issue Type: Task
Reporter: Andrew Purtell
Assignee: Enis Soztutar
Labels: phoenix
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: hbase-12788_v1.patch
After HBASE-12028, we will be passing Abortable around in a LimitedPrivate
interface, with the intent of its use in implementors and subclasses of
related types (RPC schedulers). Therefore we should promote Abortable to
LimitedPrivate as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12783) Create efficient RegionLocator implementation
[
https://issues.apache.org/jira/browse/HBASE-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263381#comment-14263381
]
Hadoop QA commented on HBASE-12783:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12689896/12783-11.patch
against master branch at commit 820f629423f21fbd1dcc7a383955443a2595fd5d.
ATTACHMENT ID: 12689896
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/12293//console
This message is automatically generated.
Create efficient RegionLocator implementation
-
Key: HBASE-12783
URL: https://issues.apache.org/jira/browse/HBASE-12783
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.0, 2.0.0
Reporter: Solomon Duskis
Assignee: Solomon Duskis
Attachments: 12783-10.patch, 12783-11.patch, HBASE-12783.patch,
HBASE-12783E.patch, HBASE-12783F.patch, HBASE-12783G.patch,
HBASE-12783H.patch, HBASE-12783I.patch, HBASE-12783J.patch,
HBASE-12783K.patch, HBASE-12873B.patch, HBASE-12873C.patch, HBASE-12873D.patch
A new HRegionLocator that only implements RegionLocator functionality will be
more efficient to instantiate than a full HTable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12641) Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster
[ https://issues.apache.org/jira/browse/HBASE-12641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12641: -- Fix Version/s: 1.0.0 Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster Key: HBASE-12641 URL: https://issues.apache.org/jira/browse/HBASE-12641 Project: HBase Issue Type: Improvement Components: Zookeeper Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12641-v1.diff Currently in a secure cluster, only the master/regionserver kerberos user can manage the znode of hbase. But he master/regionserver kerberos user is for rpc connection and we usually use another super user to manage the cluster. In some special scenarios, we need to manage the data of znode with the supper user. eg: a, To get the data of the znode for debugging. b, HBASE-8253: We need to delete the znode for the corrupted hlog to avoid it block the replication. So we grant all permissions of hbase zookeeper node to hbase superuser during creating these znodes. Suggestions are welcomed. [~apurtell] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12641) Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster
[ https://issues.apache.org/jira/browse/HBASE-12641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263367#comment-14263367 ] Enis Soztutar commented on HBASE-12641: --- I've pushed this to 1.0.0 as well. Is reading the configuration everytime we create a znode expensive? Grant all permissions of hbase zookeeper node to hbase superuser in a secure cluster Key: HBASE-12641 URL: https://issues.apache.org/jira/browse/HBASE-12641 Project: HBase Issue Type: Improvement Components: Zookeeper Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12641-v1.diff Currently in a secure cluster, only the master/regionserver kerberos user can manage the znode of hbase. But he master/regionserver kerberos user is for rpc connection and we usually use another super user to manage the cluster. In some special scenarios, we need to manage the data of znode with the supper user. eg: a, To get the data of the znode for debugging. b, HBASE-8253: We need to delete the znode for the corrupted hlog to avoid it block the replication. So we grant all permissions of hbase zookeeper node to hbase superuser during creating these znodes. Suggestions are welcomed. [~apurtell] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263380#comment-14263380
]
Enis Soztutar commented on HBASE-12781:
---
Pushed to 1.0.0 as well.
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12781) thrift2 listen port will bind always to the passed command line address
[
https://issues.apache.org/jira/browse/HBASE-12781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-12781:
--
Fix Version/s: 1.0.0
thrift2 listen port will bind always to the passed command line address
---
Key: HBASE-12781
URL: https://issues.apache.org/jira/browse/HBASE-12781
Project: HBase
Issue Type: Bug
Components: Thrift
Affects Versions: 0.98.3
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0
Attachments: 12781-V1.patch
In Thrift server, listen port will bind always to the address which is
passed through command line argument.
{code}
InetSocketAddress inetSocketAddress = bindToPort(cmd.getOptionValue(bind),
listenPort);
{code}
{code}
private static InetSocketAddress bindToPort(String bindValue, int listenPort)
throws UnknownHostException {
try {
if (bindValue == null) {
return new InetSocketAddress(listenPort);
} else {
return new InetSocketAddress(InetAddress.getByName(bindValue),
listenPort);
}
} catch (UnknownHostException e) {
throw new RuntimeException(Could not bind to provided ip address, e);
}
}
{code}
In case when bind address is not passed through argument then it is binding
with any local address. It should read hbase.thrift.info.bindAddress value
from configuration first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12719) Add test WAL provider to quantify FSHLog overhead in the absence of HDFS.
[ https://issues.apache.org/jira/browse/HBASE-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263384#comment-14263384 ] Enis Soztutar commented on HBASE-12719: --- Pushed to 1.0.0 as well. Add test WAL provider to quantify FSHLog overhead in the absence of HDFS. - Key: HBASE-12719 URL: https://issues.apache.org/jira/browse/HBASE-12719 Project: HBase Issue Type: Improvement Components: test, wal Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-12719.1.patch.txt, HBASE-12719.2.patch.txt, HBASE-12719_comparison_chart.tiff Discussion in HBASE-5699 included showing the max throughput for our WAL benchmark (using the DisabledWALProvider) compared to when we are actually dealing with synchronization and talking with HDFS. What we can't see in that comparison is how much of the (sizable) gap is due to the coordination done in FSHLog to deal with multi-threading and how much of it is due to talking to HDFS. Make a test-only provider that we can use to isolate the cost for HDFS appends, HDFS flush, and file rolling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12719) Add test WAL provider to quantify FSHLog overhead in the absence of HDFS.
[ https://issues.apache.org/jira/browse/HBASE-12719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12719: -- Fix Version/s: 1.0.0 Add test WAL provider to quantify FSHLog overhead in the absence of HDFS. - Key: HBASE-12719 URL: https://issues.apache.org/jira/browse/HBASE-12719 Project: HBase Issue Type: Improvement Components: test, wal Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Fix For: 1.0.0, 2.0.0, 1.1.0 Attachments: HBASE-12719.1.patch.txt, HBASE-12719.2.patch.txt, HBASE-12719_comparison_chart.tiff Discussion in HBASE-5699 included showing the max throughput for our WAL benchmark (using the DisabledWALProvider) compared to when we are actually dealing with synchronization and talking with HDFS. What we can't see in that comparison is how much of the (sizable) gap is due to the coordination done in FSHLog to deal with multi-threading and how much of it is due to talking to HDFS. Make a test-only provider that we can use to isolate the cost for HDFS appends, HDFS flush, and file rolling. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12728) buffered writes substantially less useful after removal of HTablePool
[
https://issues.apache.org/jira/browse/HBASE-12728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263110#comment-14263110
]
Carter commented on HBASE-12728:
Okay, here's another pass, scratching out the HTableMultiplexer idea. Instead
we'll create a new class called {{AsyncPutter}}. (Not a huge fan of the name,
so if you have a better one, please share.)
First off, here are our basic requirements in this refactor:
# Handle the M/R case where a user wants to batch and flush in a single thread
# Handle the case Aaron described where we batch across multiple threads
# Provide a way to do this through the new Table interface for convenience
# Buffering/batching limits based on size in bytes, not queue length
# Move towards [~lhofhansl]'s suggestion of HTable as cheap proxies to tables
only
# While durability can't be guaranteed in case of a crash, avoid losing data
otherwise.
So here are our classes:
{code:java}
// BufferedTable is lightweight and single-threaded. Many of them can share a
single AsyncPutter.
public class BufferedTable implements Table {
public BufferedTable(Table t, AsyncPutter ap);
public void flush();
}
// Thread-safe handler of puts for one or more BufferedTable instances.
public class AsyncPutter implements Closeable {
public AsyncPutter(Connection c, ExecutorService pool, ExceptionListener e,
PutBuffer pb);
public synchronized add(Put put); // Synchronization adds nanoseconds in
the single-threaded case. No biggie.
public synchronized flush();
public synchronized close();
}
// Simple single-threaded data holder.
public class PutBuffer {
public PutBuffer(long maxBufferSize); // In bytes. This makes more sense
than queue length for memory management.
// maxBufferSize = totalBufferMem / numberOfExecutorPoolThreads
public void add(Put p);
public boolean isBatchAvailable();
public ListPut removeBatch();
}
// To make sure exceptions don't get swallowed.
public interface ExceptionListener {
void onException(RetriesExhaustedWithDetailsException e);
}
{code}
We also proposed a {{BufferedConnection}} factory, simply to make it easier to
switch between Table and BufferTable implementations without much refactoring.
When used, it would own the AsyncPutter. Pros/cons for this idea? It's not
essential.
Asynchronous exception handling takes place through an {{ExceptionListener}}
observer provided by the user. This means that exceptions are not thrown for
simple put failures; they are passed to the listener. The thought here is I
find the current behavior non-deterministic:
{code:java}
table.put(put1); // This put causes an exception
table.put(put2); // But we don't see the exception until we get here ...
table.put(put3); // ... or maybe(?) here. put3 succeeded, but I got an
exception thrown. That's counter-intuitive.
{code}
An ExceptionListener is a pretty standard pattern for asynchronous error
handling. M/R or other cases might rely on an exception being thrown
synchronously to rollback appropriately, but it's easy enough to mimic that
behavior with the listener approach.
{{BufferedTable#close}} does not flush since we need to support batching across
multiple threads. {{AsyncPutter#close}} does flush. (Will JavaDoc this.) If
we decide to provide a BufferedConnection, then closing that would also flush,
since it owns the AsyncPutter.
Do we need a timeout-based flush? I don't see one in the current HTable
implementation, but if it's important we could add it to the AsyncPutter.
Seems a good way to limit lost mutations during slow periods of writes into a
big buffer.
buffered writes substantially less useful after removal of HTablePool
-
Key: HBASE-12728
URL: https://issues.apache.org/jira/browse/HBASE-12728
Project: HBase
Issue Type: Bug
Components: hbase
Affects Versions: 0.98.0
Reporter: Aaron Beppu
In previous versions of HBase, when use of HTablePool was encouraged, HTable
instances were long-lived in that pool, and for that reason, if autoFlush was
set to false, the table instance could accumulate a full buffer of writes
before a flush was triggered. Writes from the client to the cluster could
then be substantially larger and less frequent than without buffering.
However, when HTablePool was deprecated, the primary justification seems to
have been that creating HTable instances is cheap, so long as the connection
and executor service being passed to it are pre-provided. A use pattern was
encouraged where users should create a new HTable instance for every
operation, using an existing connection and executor service, and then close
the table. In this pattern, buffered writes are substantially less useful;
writes are as small and as
[jira] [Commented] (HBASE-12270) A bug in the bucket cache, with cache blocks on write enabled
[ https://issues.apache.org/jira/browse/HBASE-12270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14263248#comment-14263248 ] stack commented on HBASE-12270: --- Was watching the builds and I reverted from 0.98 only this morning since it is having most trouble here. Was unable to fill in the issue because JIRA seemed down. Will put it back later after we figure the issue. Sorry about that [~apurtell] A bug in the bucket cache, with cache blocks on write enabled - Key: HBASE-12270 URL: https://issues.apache.org/jira/browse/HBASE-12270 Project: HBase Issue Type: Bug Affects Versions: 0.94.11, 0.98.6.1 Environment: I can reproduce it on a simple 2 node cluster, one running the master and another running a RS. I was testing on ec2. I used the following configurations for the cluster. hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx2G -XX:MaxDirectMemorySize=5G -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog gc:/tmp/hbase-regionserver-gc.log hbase-site: hbase.bucketcache.ioengine=offheap hbase.bucketcache.size=4196 hbase.rs.cacheblocksonwrite=true hfile.block.index.cacheonwrite=true hfile.block.bloom.cacheonwrite=true Reporter: Khaled Elmeleegy Assignee: Liu Shaohui Priority: Critical Fix For: 1.0.0, 2.0.0, 0.98.10, 1.1.0 Attachments: HBASE-12270-v1.diff, HBASE-12270-v2.diff, HBASE-12270-v2.patch, TestHBase.java, TestKey.java In my experiments, I have writers streaming their output to HBase. The reader powers a web page and does this scatter/gather, where it reads 1000 keys written last and passes them the the front end. With this workload, I get the exception below at the region server. Again, I am using HBAse (0.98.6.1). Any help is appreciated. 2014-10-10 15:06:44,173 ERROR [B.DefaultRpcServer.handler=62,queue=2,port=60020] ipc.RpcServer: Unexpected throwable object java.lang.IllegalArgumentException at java.nio.Buffer.position(Buffer.java:236) at org.apache.hadoop.hbase.util.ByteBufferUtils.skip(ByteBufferUtils.java:434) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:849) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:760) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:248) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:152) at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:317) at org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:176) at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1780) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3758) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1950) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1936) at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1913) at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3157) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2027) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
