[jira] [Commented] (HBASE-9754) Eliminate threadlocal from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800371#comment-13800371
]
Lars Hofhansl commented on HBASE-9754:
--
would have felt better if a test had caught it :)
> Eliminate threadlocal from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-0.94.txt, 9754-0.94-v2.txt, 9754-addendum.txt,
> 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 9754-rp-hregion-v2.txt,
> 9754-rp-hregion-v3.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9522) Allow region opening even if creation of some HFile Readers fail.
[ https://issues.apache.org/jira/browse/HBASE-9522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800357#comment-13800357 ] ramkrishna.s.vasudevan commented on HBASE-9522: --- I will give a patch this week. > Allow region opening even if creation of some HFile Readers fail. > - > > Key: HBASE-9522 > URL: https://issues.apache.org/jira/browse/HBASE-9522 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.95.2, 0.94.11, 0.96.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.14 > > > In some scenarios, when am sure that the Reader creation while region opening > would fail, it would be better if the region still opens with a warning. > This would ensure that atleast the data that is available can be read instead > of failing the region assignment. > Agree that, this should be based on a configuration. If you feel this is ok, > I can give a patch. The patch would just collect a list of store files for > which the region opening failed and just log those files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9810) Global memstore size will be calculated wrongly if replaying recovered edits throws exception
[
https://issues.apache.org/jira/browse/HBASE-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800356#comment-13800356
]
ramkrishna.s.vasudevan commented on HBASE-9810:
---
I think this should go in 0.96 also. So should we also call
clearRegionReplayEditsSize() in case of errors and the property
HREGION_EDITS_REPLAY_SKIP_ERRORS is set to true?
> Global memstore size will be calculated wrongly if replaying recovered edits
> throws exception
> -
>
> Key: HBASE-9810
> URL: https://issues.apache.org/jira/browse/HBASE-9810
> Project: HBase
> Issue Type: Bug
>Reporter: chunhui shen
>Assignee: chunhui shen
>Priority: Critical
> Attachments: hbase-9810-trunk.patch
>
>
> Recently we encountered such a case in 0.94-version:
> Flush is triggered frequently because:
> {noformat}DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush
> thread woke up because memory above low water=14.4g
> {noformat}
> But, the real global memstore size is about 1g.
> It seems the global memstore size has been calculated wrongly.
> Through the logs, I find the following root cause log:
> {noformat}
> ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> open of region=notifysub2_index,\x83\xDC^\xCD\xA3\x8A<\x
> E2\x8E\xE6\xAD!\xDC\xE8t\xED,1379148697072.46be7c2d71c555379278a7494df3015e.,
> starting to roll back the global memstore size.
> java.lang.NegativeArraySizeException
> at org.apache.hadoop.hbase.KeyValue.getFamily(KeyValue.java:1096)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:2933)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:2811)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:583)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:499)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3939)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3887)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Browse the code of this part, seems a critial bug about global memstore size
> when replaying recovered edits.
> (RegionServerAccounting#clearRegionReplayEditsSize is called for each edit
> file, it means the roll back size is smaller than actual when calling
> RegionServerAccounting#rollbackRegionReplayEditsSize)
> Anyway, the solution is easy as the patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9754) Eliminate threadlocal from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800332#comment-13800332
]
Ted Yu commented on HBASE-9754:
---
I went over the patch again and found that for this particular ctor, we should
not use smallestReadPoint as the readpoint.
Readpoint from the region should be used.
> Eliminate threadlocal from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-0.94.txt, 9754-0.94-v2.txt, 9754-addendum.txt,
> 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 9754-rp-hregion-v2.txt,
> 9754-rp-hregion-v3.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800329#comment-13800329
]
Lars Hofhansl commented on HBASE-4811:
--
Some gets (to measure any increase setup costs for scanning) and some scans (to
measure any new scan overhead).
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v3.txt, 4811-trunk-v10.txt,
> 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
> hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
> hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
> hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
> hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
> hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> NOTE: when setting reversed as true for a client scan, you must set the start
> row, else will throw exception. Through {@link
> Scan#createBiggestByteArray(int)},you could get a big enough byte array as
> the start row
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9754) Eliminate threadlocal from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800328#comment-13800328
]
Lars Hofhansl commented on HBASE-9754:
--
How did you figure you need the addendum?
> Eliminate threadlocal from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-0.94.txt, 9754-0.94-v2.txt, 9754-addendum.txt,
> 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 9754-rp-hregion-v2.txt,
> 9754-rp-hregion-v3.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9802) A new failover test framework for HBase
[ https://issues.apache.org/jira/browse/HBASE-9802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800324#comment-13800324 ] Liang Xie commented on HBASE-9802: -- Dihao, you can assign this jira to yourself:) > A new failover test framework for HBase > --- > > Key: HBASE-9802 > URL: https://issues.apache.org/jira/browse/HBASE-9802 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.3 >Reporter: chendihao >Priority: Minor > > Currently HBase uses ChaosMonkey for IT test and fault injection. It will > restart regionserver, force balancer and perform other actions randomly and > periodically. However, we need a more extensible and full-featured framework > for our failover test and we find ChaosMonkey cant' suit our needs since it > has the following drawbacks. > 1) Only process-level actions can be simulated, not support > machine-level/hardware-level/network-level actions. > 2) No data validation before and after the test, the fatal bugs such as that > can cause data inconsistent may be overlook. > 3) When failure occurs, we can't repro the problem and hard to figure out the > reason. > Therefore, we have developed a new framework to satisfy the need of failover > test. We extended ChaosMonkey and implement the function to validate data and > to replay failed actions. Here are the features we add. > 1) Policy/Task/Action abstraction, seperating Task from Policy and Action > makes it easier to manage and replay a set of actions. > 2) Make action configurable. We have implemented some actions to cause > machine failure and defined the same interface as original actions. > 3) We should validate the date consistent before and after failover test to > ensure the availability and data correctness. > 4) After performing a set of actions, we also check the consistency of table > as well. > 5) The set of actions that caused test failure can be replayed, and the > reproducibility of actions can help fixing the exposed bugs. > Our team has developed this framework and run for a while. Some bugs were > exposed and fixed by running this test framework. Moreover, we have a monitor > program which shows the progress of failover test and make sure our cluster > is as stable as we want. Now we are trying to make it more general and will > opensource it later. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9810) Global memstore size will be calculated wrongly if replaying recovered edits throws exception
[
https://issues.apache.org/jira/browse/HBASE-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-9810:
Attachment: hbase-9810-trunk.patch
> Global memstore size will be calculated wrongly if replaying recovered edits
> throws exception
> -
>
> Key: HBASE-9810
> URL: https://issues.apache.org/jira/browse/HBASE-9810
> Project: HBase
> Issue Type: Bug
>Reporter: chunhui shen
>Assignee: chunhui shen
>Priority: Critical
> Attachments: hbase-9810-trunk.patch
>
>
> Recently we encountered such a case in 0.94-version:
> Flush is triggered frequently because:
> {noformat}DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush
> thread woke up because memory above low water=14.4g
> {noformat}
> But, the real global memstore size is about 1g.
> It seems the global memstore size has been calculated wrongly.
> Through the logs, I find the following root cause log:
> {noformat}
> ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
> open of region=notifysub2_index,\x83\xDC^\xCD\xA3\x8A<\x
> E2\x8E\xE6\xAD!\xDC\xE8t\xED,1379148697072.46be7c2d71c555379278a7494df3015e.,
> starting to roll back the global memstore size.
> java.lang.NegativeArraySizeException
> at org.apache.hadoop.hbase.KeyValue.getFamily(KeyValue.java:1096)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:2933)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:2811)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:583)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:499)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3939)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3887)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> Browse the code of this part, seems a critial bug about global memstore size
> when replaying recovered edits.
> (RegionServerAccounting#clearRegionReplayEditsSize is called for each edit
> file, it means the roll back size is smaller than actual when calling
> RegionServerAccounting#rollbackRegionReplayEditsSize)
> Anyway, the solution is easy as the patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9810) Global memstore size will be calculated wrongly if replaying recovered edits throws exception
[
https://issues.apache.org/jira/browse/HBASE-9810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-9810:
Description:
Recently we encountered such a case in 0.94-version:
Flush is triggered frequently because:
{noformat}DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush
thread woke up because memory above low water=14.4g
{noformat}
But, the real global memstore size is about 1g.
It seems the global memstore size has been calculated wrongly.
Through the logs, I find the following root cause log:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
open of region=notifysub2_index,\x83\xDC^\xCD\xA3\x8A<\x
E2\x8E\xE6\xAD!\xDC\xE8t\xED,1379148697072.46be7c2d71c555379278a7494df3015e.,
starting to roll back the global memstore size.
java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.KeyValue.getFamily(KeyValue.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:2933)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:2811)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:583)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:499)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3939)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3887)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
Browse the code of this part, seems a critial bug about global memstore size
when replaying recovered edits.
(RegionServerAccounting#clearRegionReplayEditsSize is called for each edit
file, it means the roll back size is smaller than actual when calling
RegionServerAccounting#rollbackRegionReplayEditsSize)
Anyway, the solution is easy as the patch.
was:
Recently we encountered such a case in 0.94-version:
Flush is triggered frequently because:
{format}DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush
thread woke up because memory above low water=14.4g
{format}
But, the real global memstore size is about 1g.
It seems the global memstore size has been calculated wrongly.
Through the logs, I find the following root cause log:
{format}
ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
open of region=notifysub2_index,\x83\xDC^\xCD\xA3\x8A<\x
E2\x8E\xE6\xAD!\xDC\xE8t\xED,1379148697072.46be7c2d71c555379278a7494df3015e.,
starting to roll back the global memstore size.
java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.KeyValue.getFamily(KeyValue.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:2933)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:2811)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:583)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:499)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3939)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3887)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{format}
Browse the code of this part, seems a critial bug about global memstore size
when replaying recovered edits.
(RegionServerAccounting#clearRegionReplayEditsSize is called for each edit
file, it means the roll back size is smaller than actual when calling
RegionServerAccounting#rollbackRegionReplayEditsSize)
Anyway, the solution is easy as the patch.
> Global memstore size will be calculated wrongly if replaying recovered edits
> throws exception
> -
>
> Key: HBASE-9810
>
[jira] [Commented] (HBASE-9754) Eliminate threadlocal from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800322#comment-13800322
]
Hadoop QA commented on HBASE-9754:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609351/9754-addendum.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7591//console
This message is automatically generated.
> Eliminate threadlocal from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-0.94.txt, 9754-0.94-v2.txt, 9754-addendum.txt,
> 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 9754-rp-hregion-v2.txt,
> 9754-rp-hregion-v3.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9810) Global memstore size will be calculated wrongly if replaying recovered edits throws exception
chunhui shen created HBASE-9810:
---
Summary: Global memstore size will be calculated wrongly if
replaying recovered edits throws exception
Key: HBASE-9810
URL: https://issues.apache.org/jira/browse/HBASE-9810
Project: HBase
Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Recently we encountered such a case in 0.94-version:
Flush is triggered frequently because:
{format}DEBUG org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush
thread woke up because memory above low water=14.4g
{format}
But, the real global memstore size is about 1g.
It seems the global memstore size has been calculated wrongly.
Through the logs, I find the following root cause log:
{format}
ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed
open of region=notifysub2_index,\x83\xDC^\xCD\xA3\x8A<\x
E2\x8E\xE6\xAD!\xDC\xE8t\xED,1379148697072.46be7c2d71c555379278a7494df3015e.,
starting to roll back the global memstore size.
java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.KeyValue.getFamily(KeyValue.java:1096)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:2933)
at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:2811)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:583)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:499)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3939)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3887)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{format}
Browse the code of this part, seems a critial bug about global memstore size
when replaying recovered edits.
(RegionServerAccounting#clearRegionReplayEditsSize is called for each edit
file, it means the roll back size is smaller than actual when calling
RegionServerAccounting#rollbackRegionReplayEditsSize)
Anyway, the solution is easy as the patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9775) Client write path perf issues
[ https://issues.apache.org/jira/browse/HBASE-9775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800320#comment-13800320 ] Jean-Marc Spaggiari commented on HBASE-9775: [~nkeywal] to make sure I understand, I just run my test suite again 0.96.0 RC5 and 0.94.12? I don't apply any patch on top of that? I can do that for sure. I have a server which is doing nothing, just waiting for that ;) Also, next week-end I will be able to run the same test on a 4 nodes cluster if required. Will wait for your confirmation to start. > Client write path perf issues > - > > Key: HBASE-9775 > URL: https://issues.apache.org/jira/browse/HBASE-9775 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.96.0 >Reporter: Elliott Clark >Priority: Critical > Attachments: Charts Search Cloudera Manager - ITBLL.png, Charts > Search Cloudera Manager.png, hbase-9775.patch, job_run.log, short_ycsb.png, > ycsb_insert_94_vs_96.png, ycsb.png > > > Testing on larger clusters has not had the desired throughput increases. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9754) Eliminate threadlocal from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800318#comment-13800318
]
chunhui shen commented on HBASE-9754:
-
Addendum is ok for me
> Eliminate threadlocal from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-0.94.txt, 9754-0.94-v2.txt, 9754-addendum.txt,
> 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 9754-rp-hregion-v2.txt,
> 9754-rp-hregion-v3.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800317#comment-13800317
]
Jean-Marc Spaggiari commented on HBASE-4811:
started with:
filterScan
randomSeekScan
scanRange10
scanRange1000
for both 0.94 branch and 0.94+4811... Will post the results as soon as it's
done. Running on SSD drive. I can also run it on a server with SATA if required.
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v3.txt, 4811-trunk-v10.txt,
> 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
> hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
> hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
> hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
> hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
> hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> NOTE: when setting reversed as true for a client scan, you must set the start
> row, else will throw exception. Through {@link
> Scan#createBiggestByteArray(int)},you could get a big enough byte array as
> the start row
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800314#comment-13800314
]
Jean-Marc Spaggiari commented on HBASE-4811:
Do you want all the tests?
Random read/write
Sequence read/write
scans.
Or just the scans? The more tests, the longer ;)
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v3.txt, 4811-trunk-v10.txt,
> 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
> hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
> hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
> hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
> hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
> hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> NOTE: when setting reversed as true for a client scan, you must set the start
> row, else will throw exception. Through {@link
> Scan#createBiggestByteArray(int)},you could get a big enough byte array as
> the start row
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9754) Eliminate threadlocal from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9754:
--
Attachment: 9754-addendum.txt
> Eliminate threadlocal from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-0.94.txt, 9754-0.94-v2.txt, 9754-addendum.txt,
> 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 9754-rp-hregion-v2.txt,
> 9754-rp-hregion-v3.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9754) Eliminate threadlocal from MVCC code
[
https://issues.apache.org/jira/browse/HBASE-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9754:
--
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Eliminate threadlocal from MVCC code
>
>
> Key: HBASE-9754
> URL: https://issues.apache.org/jira/browse/HBASE-9754
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Ted Yu
> Fix For: 0.98.0
>
> Attachments: 9754-0.94.txt, 9754-0.94-v2.txt, 9754-addendum.txt,
> 9754-rp-0.txt, 9754-rp-1.txt, 9754-rp-hregion.txt, 9754-rp-hregion-v2.txt,
> 9754-rp-hregion-v3.txt
>
>
> Brought up by [~vrodionov] and [~yuzhih...@gmail.com].
> Currently we use ThreadLocals to communicate the current readpoint between a
> RegionScanner and the Store\{File}Scanner's down the stack.
> Since ThreadLocals are not cheap we should consider whether it is possible to
> pass the readpoint through the call stack instead.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800296#comment-13800296
]
Lars Hofhansl commented on HBASE-4811:
--
Yep. If you have time, that's be great (I just would like an independent
performance review here :) ).
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v3.txt, 4811-trunk-v10.txt,
> 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
> hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
> hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
> hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
> hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
> hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> NOTE: when setting reversed as true for a client scan, you must set the start
> row, else will throw exception. Through {@link
> Scan#createBiggestByteArray(int)},you could get a big enough byte array as
> the start row
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8143) HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
[
https://issues.apache.org/jira/browse/HBASE-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800295#comment-13800295
]
Lars Hofhansl commented on HBASE-8143:
--
This should to be added to hdfs-site.xml as seen at the region server, right?
> HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
> --
>
> Key: HBASE-8143
> URL: https://issues.apache.org/jira/browse/HBASE-8143
> Project: HBase
> Issue Type: Bug
> Components: hadoop2
>Affects Versions: 0.98.0, 0.94.7, 0.95.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: 8143doc.txt, OpenFileTest.java
>
>
> We've run into an issue with HBase 0.94 on Hadoop2, with SSR turned on that
> the memory usage of the HBase process grows to 7g, on an -Xmx3g, after some
> time, this causes OOM for the RSs.
> Upon further investigation, I've found out that we end up with 200 regions,
> each having 3-4 store files open. Under hadoop2 SSR, BlockReaderLocal
> allocates DirectBuffers, which is unlike HDFS 1 where there is no direct
> buffer allocation.
> It seems that there is no guards against the memory used by local buffers in
> hdfs 2, and having a large number of open files causes multiple GB of memory
> to be consumed from the RS process.
> This issue is to further investigate what is going on. Whether we can limit
> the memory usage in HDFS, or HBase, and/or document the setup.
> Possible mitigation scenarios are:
> - Turn off SSR for Hadoop 2
> - Ensure that there is enough unallocated memory for the RS based on
> expected # of store files
> - Ensure that there is lower number of regions per region server (hence
> number of open files)
> Stack trace:
> {code}
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1560)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1439)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1380)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:449)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:215)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:63)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:632)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:97)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> at
> org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(DirectBufferPool.java:70)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:315)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:208)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:790)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:888)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:455)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:645)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:312)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:543)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:589)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1261)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:512)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:603)
> at
> org.apache.hadoop.hbase.regionserver.Store.validateStoreFile(Store.java:1568)
> at
> org.apache.hadoop.hbase.regionserver.Store.commitFile(Store.java:845)
> at
> org.apache.hadoop.hbase.regionserver.Store.access$500(Store.java:109)
> at
> org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.commit(Store.java:2209)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.inte
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800293#comment-13800293
]
Jean-Marc Spaggiari commented on HBASE-4811:
Sure I can.
So I run the last 0.94 branch and 0.94 branch+ this patch and compare. Is that
correct?
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v3.txt, 4811-trunk-v10.txt,
> 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
> hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
> hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
> hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
> hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
> hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> NOTE: when setting reversed as true for a client scan, you must set the start
> row, else will throw exception. Through {@link
> Scan#createBiggestByteArray(int)},you could get a big enough byte array as
> the start row
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8803) region_mover.rb should move multiple regions at a time
[ https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800291#comment-13800291 ] Jean-Marc Spaggiari commented on HBASE-8803: It's definitively on my todo. Just lacking some time. I will have time to work on all pending JIRAs (8803, 9694, 9695, etc.) I have the week of November 4rd... Sorry for the delay :( > region_mover.rb should move multiple regions at a time > -- > > Key: HBASE-8803 > URL: https://issues.apache.org/jira/browse/HBASE-8803 > Project: HBase > Issue Type: Bug > Components: Usability >Affects Versions: 0.98.0, 0.94.8, 0.95.1 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 8803v5.txt, HBASE-8803-v0-trunk.patch, > HBASE-8803-v1-0.94.patch, HBASE-8803-v1-trunk.patch, > HBASE-8803-v2-0.94.patch, HBASE-8803-v2-0.94.patch, HBASE-8803-v3-0.94.patch, > HBASE-8803-v4-0.94.patch, HBASE-8803-v4-trunk.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > When there is many regions in a cluster, rolling_restart can take hours > because region_mover is moving the regions one by one. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9807) scanning with block encoding unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800287#comment-13800287
]
Hadoop QA commented on HBASE-9807:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609340/9807-trunk-v1.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7590//console
This message is automatically generated.
> scanning with block encoding unnecessarily copies the key for each reseek
> -
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates to ByteBuffers in ScannerV2 and makes a deep copy of the
> key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-8143) HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
[
https://issues.apache.org/jira/browse/HBASE-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-8143:
-
Attachment: 8143doc.txt
Its a client-side config no? Not for hdfs-side.
Here is a bit of doc for the reference guide that recommends setting this down
from its default size. Does this do? If so, I'll commit (I try to clean up
the stale SSR section a little too).
> HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
> --
>
> Key: HBASE-8143
> URL: https://issues.apache.org/jira/browse/HBASE-8143
> Project: HBase
> Issue Type: Bug
> Components: hadoop2
>Affects Versions: 0.98.0, 0.94.7, 0.95.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: 8143doc.txt, OpenFileTest.java
>
>
> We've run into an issue with HBase 0.94 on Hadoop2, with SSR turned on that
> the memory usage of the HBase process grows to 7g, on an -Xmx3g, after some
> time, this causes OOM for the RSs.
> Upon further investigation, I've found out that we end up with 200 regions,
> each having 3-4 store files open. Under hadoop2 SSR, BlockReaderLocal
> allocates DirectBuffers, which is unlike HDFS 1 where there is no direct
> buffer allocation.
> It seems that there is no guards against the memory used by local buffers in
> hdfs 2, and having a large number of open files causes multiple GB of memory
> to be consumed from the RS process.
> This issue is to further investigate what is going on. Whether we can limit
> the memory usage in HDFS, or HBase, and/or document the setup.
> Possible mitigation scenarios are:
> - Turn off SSR for Hadoop 2
> - Ensure that there is enough unallocated memory for the RS based on
> expected # of store files
> - Ensure that there is lower number of regions per region server (hence
> number of open files)
> Stack trace:
> {code}
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1560)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1439)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1380)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:449)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:215)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:63)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:632)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:97)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> at
> org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(DirectBufferPool.java:70)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:315)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:208)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:790)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:888)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:455)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:645)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:312)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:543)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:589)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1261)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:512)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:603)
> at
> org.apache.hadoop.hbase.regionserver.Store.validateStoreFile(Store.java:1568)
> at
> org.apache.hadoop.hbase.regionserver.Store.commitFile(Store.java:845)
> at
> org.apache.hadoop.hbase.regionserver.Store.access$500(Store.java:109)
> at
> org.apache.hadoop.hbase.re
[jira] [Commented] (HBASE-9807) scanning with block encoding unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800270#comment-13800270
]
Lars Hofhansl commented on HBASE-9807:
--
[~mcorgan], wanna have a look? Can you think of a way of avoiding this copy? I
guess we could partial compares, since we know the scanner won't progress
throughout this call.
> scanning with block encoding unnecessarily copies the key for each reseek
> -
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates to ByteBuffers in ScannerV2 and makes a deep copy of the
> key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9807) scanning with block encoding unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9807:
-
Attachment: 9807-0.94-v3.txt
Somehow {{throws IOException}} sneaked in there. Not needed.
> scanning with block encoding unnecessarily copies the key for each reseek
> -
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates to ByteBuffers in ScannerV2 and makes a deep copy of the
> key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HBASE-9807) scanning with block encoding unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl reassigned HBASE-9807:
Assignee: Lars Hofhansl
> scanning with block encoding unnecessarily copies the key for each reseek
> -
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates to ByteBuffers in ScannerV2 and makes a deep copy of the
> key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9807) scanning with block encoding unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9807:
-
Status: Patch Available (was: Open)
> scanning with block encoding unnecessarily copies the key for each reseek
> -
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates to ByteBuffers in ScannerV2 and makes a deep copy of the
> key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9807) scanning with block encoding unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9807:
-
Attachment: 9807-trunk-v1.txt
And a patch for trunk. Note that prefix tree encoding cannot be optimized this
way. So this patch won't help there.
> scanning with block encoding unnecessarily copies the key for each reseek
> -
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt, 9807-0.94-v3.txt,
> 9807-trunk-v1.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates to ByteBuffers in ScannerV2 and makes a deep copy of the
> key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9783) Validate column families early in o.a.h.h.regionserver.HRegion.mutateRowsWithLocks()
[
https://issues.apache.org/jira/browse/HBASE-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800260#comment-13800260
]
Lars Hofhansl commented on HBASE-9783:
--
+1
Will commit later today unless I hear objections.
> Validate column families early in
> o.a.h.h.regionserver.HRegion.mutateRowsWithLocks()
>
>
> Key: HBASE-9783
> URL: https://issues.apache.org/jira/browse/HBASE-9783
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.12
>Reporter: Aditya Kishore
>Assignee: Aditya Kishore
>Priority: Minor
> Fix For: 0.94.13
>
> Attachments: HBASE-9783.patch, HBASE-9783.patch, HBASE-9783.patch,
> HBASE-9783.patch
>
>
> h6. 0.94 branch only
> Calling {{HTable.mutateRow()}} with non-existent column families result in a
> NPE on the region server which is wrapped into {{RemoteException}} and
> returned to the client. Since NPE is not a {{DoNotRetryIOException}}, client
> keeps repeating it until the number of retries are exhausted.
> Negative case, hence creating as minor.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9783) Validate column families early in o.a.h.h.regionserver.HRegion.mutateRowsWithLocks()
[
https://issues.apache.org/jira/browse/HBASE-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aditya Kishore updated HBASE-9783:
--
Attachment: HBASE-9783.patch
I see it now, thanks :) Patch updated.
> Validate column families early in
> o.a.h.h.regionserver.HRegion.mutateRowsWithLocks()
>
>
> Key: HBASE-9783
> URL: https://issues.apache.org/jira/browse/HBASE-9783
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.12
>Reporter: Aditya Kishore
>Assignee: Aditya Kishore
>Priority: Minor
> Fix For: 0.94.13
>
> Attachments: HBASE-9783.patch, HBASE-9783.patch, HBASE-9783.patch,
> HBASE-9783.patch
>
>
> h6. 0.94 branch only
> Calling {{HTable.mutateRow()}} with non-existent column families result in a
> NPE on the region server which is wrapped into {{RemoteException}} and
> returned to the client. Since NPE is not a {{DoNotRetryIOException}}, client
> keeps repeating it until the number of retries are exhausted.
> Negative case, hence creating as minor.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9221) Provide interface for getting a User in the client
[ https://issues.apache.org/jira/browse/HBASE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800238#comment-13800238 ] Lars Hofhansl commented on HBASE-9221: -- Is this done [~jesse_yates]? > Provide interface for getting a User in the client > -- > > Key: HBASE-9221 > URL: https://issues.apache.org/jira/browse/HBASE-9221 > Project: HBase > Issue Type: Improvement > Components: Client, hadoop2, security >Affects Versions: 0.98.0, 0.95.2, 0.94.12 >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: hbase-9221-0.94-v0.patch, hbase-9221-0.94-v1.patch, > hbase-9221-trunk-v0.patch, hbase-9221-trunk-v1.patch, > hbase-9221-trunk-v2.patch, hbase-9221-trunk-v3.patch > > > Sometimes, users will want to provide their own User class depending on the > type of security they will support in their local environment. For instance, > running Hadoop1 vs Hadoop2 vs CDH means potentially different ways of getting > the UserGroupInformation. > This issue abstracts out the mechanism by which we obtain an > o.a.h.hbase.security.User to a UserProvider. This UserProvider can then be > extented as a Hadoop 1/2 shim as well as supporting custom authentication > code. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9645) Regionserver halt because of HLog's "Logic Error Snapshot seq id from earlier flush still present!"
[
https://issues.apache.org/jira/browse/HBASE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9645:
-
Fix Version/s: 0.98.0
> Regionserver halt because of HLog's "Logic Error Snapshot seq id from earlier
> flush still present!"
> ---
>
> Key: HBASE-9645
> URL: https://issues.apache.org/jira/browse/HBASE-9645
> Project: HBase
> Issue Type: Bug
> Components: regionserver, wal
>Affects Versions: 0.94.10
> Environment: Linux 2.6.32-el5.x86_64
>Reporter: Victor Xu
>Priority: Critical
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: HBASE_9645-0.94.10.patch
>
>
> I upgrade my hbase cluster to 0.94.10 three weeks ago, and this case happened
> several days after that. I change the bug's priority to 'Critical' because
> every time it happens, a regionserver halt down. All of them have the same
> log:
> {noformat}
> ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Logic Error Snapshot seq
> id from earlier flush still present! for region
> c0d88db4ce3606842fbec9d34c38f707 overwritten oldseq=80114270537with new
> seq=80115066829
> {noformat}
> I check the code finding that it locates at HLog.startCacheFlush method. The
> 'lastSeqWritten' has been locked. Maybe something wrong happened outside the
> HLog that change it by mistake.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9645) Regionserver halt because of HLog's "Logic Error Snapshot seq id from earlier flush still present!"
[
https://issues.apache.org/jira/browse/HBASE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800237#comment-13800237
]
Lars Hofhansl commented on HBASE-9645:
--
Looks good. Should be documented a bit more.
> Regionserver halt because of HLog's "Logic Error Snapshot seq id from earlier
> flush still present!"
> ---
>
> Key: HBASE-9645
> URL: https://issues.apache.org/jira/browse/HBASE-9645
> Project: HBase
> Issue Type: Bug
> Components: regionserver, wal
>Affects Versions: 0.94.10
> Environment: Linux 2.6.32-el5.x86_64
>Reporter: Victor Xu
>Priority: Critical
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: HBASE_9645-0.94.10.patch
>
>
> I upgrade my hbase cluster to 0.94.10 three weeks ago, and this case happened
> several days after that. I change the bug's priority to 'Critical' because
> every time it happens, a regionserver halt down. All of them have the same
> log:
> {noformat}
> ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Logic Error Snapshot seq
> id from earlier flush still present! for region
> c0d88db4ce3606842fbec9d34c38f707 overwritten oldseq=80114270537with new
> seq=80115066829
> {noformat}
> I check the code finding that it locates at HLog.startCacheFlush method. The
> 'lastSeqWritten' has been locked. Maybe something wrong happened outside the
> HLog that change it by mistake.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9522) Allow region opening even if creation of some HFile Readers fail.
[ https://issues.apache.org/jira/browse/HBASE-9522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9522: - Fix Version/s: (was: 0.94.13) 0.94.14 Have a patch [~ram_krish]? > Allow region opening even if creation of some HFile Readers fail. > - > > Key: HBASE-9522 > URL: https://issues.apache.org/jira/browse/HBASE-9522 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.95.2, 0.94.11, 0.96.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.14 > > > In some scenarios, when am sure that the Reader creation while region opening > would fail, it would be better if the region still opens with a warning. > This would ensure that atleast the data that is available can be read instead > of failing the region assignment. > Agree that, this should be based on a configuration. If you feel this is ok, > I can give a patch. The patch would just collect a list of store files for > which the region opening failed and just log those files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9783) Validate column families early in o.a.h.h.regionserver.HRegion.mutateRowsWithLocks()
[
https://issues.apache.org/jira/browse/HBASE-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800233#comment-13800233
]
Lars Hofhansl commented on HBASE-9783:
--
memstoreUpdated should be set to true before we call applyFamilyMapToMemstore,
so we roll back any partial memstore updates.
It's only really important to set this after we call checkFamily...
> Validate column families early in
> o.a.h.h.regionserver.HRegion.mutateRowsWithLocks()
>
>
> Key: HBASE-9783
> URL: https://issues.apache.org/jira/browse/HBASE-9783
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.12
>Reporter: Aditya Kishore
>Assignee: Aditya Kishore
>Priority: Minor
> Fix For: 0.94.13
>
> Attachments: HBASE-9783.patch, HBASE-9783.patch, HBASE-9783.patch
>
>
> h6. 0.94 branch only
> Calling {{HTable.mutateRow()}} with non-existent column families result in a
> NPE on the region server which is wrapped into {{RemoteException}} and
> returned to the client. Since NPE is not a {{DoNotRetryIOException}}, client
> keeps repeating it until the number of retries are exhausted.
> Negative case, hence creating as minor.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8619) [HBase Client]: Improve client to retry if master is down when requesting getHTableDescriptor
[
https://issues.apache.org/jira/browse/HBASE-8619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800231#comment-13800231
]
Lars Hofhansl commented on HBASE-8619:
--
[~eclark], are you still working on this?
> [HBase Client]: Improve client to retry if master is down when requesting
> getHTableDescriptor
> -
>
> Key: HBASE-8619
> URL: https://issues.apache.org/jira/browse/HBASE-8619
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 0.94.8
>Reporter: Aleksandr Shulman
>Assignee: Elliott Clark
>Priority: Minor
> Fix For: 0.94.13
>
>
> In our rolling upgrade testing, running ImportTsv failed in the job
> submission phase when the master was down. This was when the master was
> failing over to the backup master. In this case, a retry would have been
> helpful and made sure that the job would get submitted.
> A good solution would be to refresh the master information before placing the
> call to getHTableDescriptor.
> Command:
> {code} sudo -u hbase hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
> -Dimporttsv.columns=HBASE_ROW_KEY,c1,c2,c3
> -Dimporttsv.bulk.output=/user/hbase/storeFiles2_2/import2_table1369439156
> import2_table1369439156 /user/hbase/tsv2{code}
> Here is the stack trace:
> {code} 13/05/24 16:55:49 INFO compress.CodecPool: Got brand-new compressor
> [.deflate]
> 16:45:44 Exception in thread "main"
> java.lang.reflect.UndeclaredThrowableException
> 16:45:44 at $Proxy7.getHTableDescriptors(Unknown Source)
> 16:45:44 at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:1861)
> 16:45:44 at
> org.apache.hadoop.hbase.client.HTable.getTableDescriptor(HTable.java:440)
> 16:45:44 at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureCompression(HFileOutputFormat.java:458)
> 16:45:44 at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLoad(HFileOutputFormat.java:375)
> 16:45:44 at
> org.apache.hadoop.hbase.mapreduce.ImportTsv.createSubmittableJob(ImportTsv.java:280)
> 16:45:44 at
> org.apache.hadoop.hbase.mapreduce.ImportTsv.main(ImportTsv.java:424)
> 16:45:44 Caused by: java.io.IOException: Call to
> hbase-rolling-6.ent.cloudera.com/10.20.186.99:22001 failed on local
> exception: java.io.EOFException
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:1030)
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:999)
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
> 16:45:44 ... 7 more
> 16:45:44 Caused by: java.io.EOFException
> 16:45:44 at java.io.DataInputStream.readInt(DataInputStream.java:375)
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:646)
> 16:45:44 at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:580){code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8803) region_mover.rb should move multiple regions at a time
[ https://issues.apache.org/jira/browse/HBASE-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800230#comment-13800230 ] Lars Hofhansl commented on HBASE-8803: -- Where are we with this? Should I remove this from 0.94? > region_mover.rb should move multiple regions at a time > -- > > Key: HBASE-8803 > URL: https://issues.apache.org/jira/browse/HBASE-8803 > Project: HBase > Issue Type: Bug > Components: Usability >Affects Versions: 0.98.0, 0.94.8, 0.95.1 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 8803v5.txt, HBASE-8803-v0-trunk.patch, > HBASE-8803-v1-0.94.patch, HBASE-8803-v1-trunk.patch, > HBASE-8803-v2-0.94.patch, HBASE-8803-v2-0.94.patch, HBASE-8803-v3-0.94.patch, > HBASE-8803-v4-0.94.patch, HBASE-8803-v4-trunk.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > When there is many regions in a cluster, rolling_restart can take hours > because region_mover is moving the regions one by one. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9459) Backport 8534 "Fix coverage for org.apache.hadoop.hbase.mapreduce" to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9459: - Fix Version/s: (was: 0.94.13) 0.94.14 > Backport 8534 "Fix coverage for org.apache.hadoop.hbase.mapreduce" to 0.94 > -- > > Key: HBASE-9459 > URL: https://issues.apache.org/jira/browse/HBASE-9459 > Project: HBase > Issue Type: Test > Components: mapreduce, test >Reporter: Nick Dimiduk >Assignee: Aleksey Gorshkov > Fix For: 0.94.14 > > > Do you want this test update backported? See HBASE-8534 for a 0.94 patch. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HBASE-9235) Backport HBASE-8667 "Master and Regionserver not able to communicate if both bound to different network interfaces on the same machine"
[ https://issues.apache.org/jira/browse/HBASE-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-9235. -- Resolution: Won't Fix Fix Version/s: (was: 0.94.13) Nobody showed up. No interested. Closing. > Backport HBASE-8667 "Master and Regionserver not able to communicate if both > bound to different network interfaces on the same machine" > > > Key: HBASE-9235 > URL: https://issues.apache.org/jira/browse/HBASE-9235 > Project: HBase > Issue Type: Task >Reporter: stack > > For consideration in 0.94. It could be a surprise changing how we bind. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HBASE-8353) -ROOT-/.META. regions are hanging if master restarted while closing -ROOT-/.META. regions on dead RS
[
https://issues.apache.org/jira/browse/HBASE-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl resolved HBASE-8353.
--
Resolution: Won't Fix
Fix Version/s: (was: 0.94.13)
As threatened, closing as "Won't fix" due to a lack of interest.
> -ROOT-/.META. regions are hanging if master restarted while closing
> -ROOT-/.META. regions on dead RS
>
>
> Key: HBASE-8353
> URL: https://issues.apache.org/jira/browse/HBASE-8353
> Project: HBase
> Issue Type: Bug
> Components: Region Assignment
>Affects Versions: 0.94.6
>Reporter: rajeshbabu
>Assignee: rajeshbabu
> Attachments: HBASE-8353_94_2.patch, HBASE-8353_94_3.patch,
> HBASE-8353_94_4.patch, HBASE-8353_94.patch
>
>
> ROOT/META are not getting assigned if master restarted while closing
> ROOT/META.
> Lets suppose catalog table regions in M_ZK_REGION_CLOSING state during master
> initialization and then just we are adding the them to RIT and waiting for
> TM. {code}
> if (isOnDeadServer(regionInfo, deadServers) &&
> (data.getOrigin() == null ||
> !serverManager.isServerOnline(data.getOrigin( {
> // If was on dead server, its closed now. Force to OFFLINE and this
> // will get it reassigned if appropriate
> forceOffline(regionInfo, data);
> } else {
> // Just insert region into RIT.
> // If this never updates the timeout will trigger new assignment
> regionsInTransition.put(encodedRegionName, new RegionState(
> regionInfo, RegionState.State.CLOSING,
> data.getStamp(), data.getOrigin()));
> }
> {code}
> isOnDeadServer always return false to ROOT/META because deadServers is null.
> Even TM cannot close them properly because its not available in online
> regions since its not yet assigned.
> {code}
> synchronized (this.regions) {
> // Check if this region is currently assigned
> if (!regions.containsKey(region)) {
> LOG.debug("Attempted to unassign region " +
> region.getRegionNameAsString() + " but it is not " +
> "currently assigned anywhere");
> return;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HBASE-9651) Backport HBASE-3890 'Scheduled tasks in distributed log splitting not in sync with ZK' to 0.94
[
https://issues.apache.org/jira/browse/HBASE-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl resolved HBASE-9651.
--
Resolution: Fixed
Hadoop Flags: Reviewed
Was committed some time back. Marking fixed.
> Backport HBASE-3890 'Scheduled tasks in distributed log splitting not in sync
> with ZK' to 0.94
> --
>
> Key: HBASE-9651
> URL: https://issues.apache.org/jira/browse/HBASE-9651
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.94.13
>
> Attachments: 9651.patch
>
>
> HBASE-3890 was fixed in 0.96 and trunk. This issue is to backport to 0.94
> Note that there must be more slightly off here. Although the splitlogs znode
> is now empty the master is still stuck here:
> {code}
> Doing distributed log split in
> hdfs://localhost:8020/hbase/.logs/10.0.0.65,60020,1305406356765
> - Waiting for distributed tasks to finish. scheduled=2 done=1 error=0 4380s
> Master startup
> - Splitting logs after master startup 4388s
> {code}
> There seems to be an issue with what is in ZK and what the TaskBatch holds.
> In my case it could be related to the fact that the task was already in ZK
> after many faulty restarts because of the NPE. Maybe it was added once (since
> that is keyed by path, and that is unique on my machine), but the reference
> count upped twice? Now that the real one is done, the done counter has been
> increased, but will never match the scheduled.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-8143) HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
[
https://issues.apache.org/jira/browse/HBASE-8143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-8143:
-
Fix Version/s: (was: 0.94.13)
Removing from 0.94.
Can we force HDFS default settings?
> HBase on Hadoop 2 with local short circuit reads (ssr) causes OOM
> --
>
> Key: HBASE-8143
> URL: https://issues.apache.org/jira/browse/HBASE-8143
> Project: HBase
> Issue Type: Bug
> Components: hadoop2
>Affects Versions: 0.98.0, 0.94.7, 0.95.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 0.98.0, 0.96.1
>
> Attachments: OpenFileTest.java
>
>
> We've run into an issue with HBase 0.94 on Hadoop2, with SSR turned on that
> the memory usage of the HBase process grows to 7g, on an -Xmx3g, after some
> time, this causes OOM for the RSs.
> Upon further investigation, I've found out that we end up with 200 regions,
> each having 3-4 store files open. Under hadoop2 SSR, BlockReaderLocal
> allocates DirectBuffers, which is unlike HDFS 1 where there is no direct
> buffer allocation.
> It seems that there is no guards against the memory used by local buffers in
> hdfs 2, and having a large number of open files causes multiple GB of memory
> to be consumed from the RS process.
> This issue is to further investigate what is going on. Whether we can limit
> the memory usage in HDFS, or HBase, and/or document the setup.
> Possible mitigation scenarios are:
> - Turn off SSR for Hadoop 2
> - Ensure that there is enough unallocated memory for the RS based on
> expected # of store files
> - Ensure that there is lower number of regions per region server (hence
> number of open files)
> Stack trace:
> {code}
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> IntegrationTestLoadAndVerify,yC^P\xD7\x945\xD4,1363388517630.24655343d8d356ef708732f34cfe8946.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1560)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1439)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1380)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:449)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:215)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$500(MemStoreFlusher.java:63)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:237)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:632)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:97)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> at
> org.apache.hadoop.hdfs.util.DirectBufferPool.getBuffer(DirectBufferPool.java:70)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.(BlockReaderLocal.java:315)
> at
> org.apache.hadoop.hdfs.BlockReaderLocal.newBlockReader(BlockReaderLocal.java:208)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocalBlockReader(DFSClient.java:790)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:888)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:455)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:645)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689)
> at java.io.DataInputStream.readFully(DataInputStream.java:178)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:312)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:543)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:589)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1261)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:512)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:603)
> at
> org.apache.hadoop.hbase.regionserver.Store.validateStoreFile(Store.java:1568)
> at
> org.apache.hadoop.hbase.regionserver.Store.commitFile(Store.java:845)
> at
> org.apache.hadoop.hbase.regionserver.Store.access$500(Store.java:109)
> at
> org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.commit(Store.java:2209)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1541)
> {code}
--
This
[jira] [Updated] (HBASE-9807) scanning with block encoding unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9807:
-
Fix Version/s: 0.96.1
0.94.13
0.98.0
> scanning with block encoding unnecessarily copies the key for each reseek
> -
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates to ByteBuffers in ScannerV2 and makes a deep copy of the
> key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9807) scanning with block encoding unnecessarily copies the key for each reseek
[
https://issues.apache.org/jira/browse/HBASE-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-9807:
-
Attachment: 9807-0.94-v2.txt
Simplified patch. This is only needed in AbstractScannerV2 hierarchy.
> scanning with block encoding unnecessarily copies the key for each reseek
> -
>
> Key: HBASE-9807
> URL: https://issues.apache.org/jira/browse/HBASE-9807
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
> Attachments: 9807-0.94.txt, 9807-0.94-v2.txt
>
>
> In HFileReaderV2.AbstractScannerV2.reseekTo(...) we have this:
> {code}
> ByteBuffer bb = getKey();
> compared = reader.getComparator().compare(key, offset,
> length, bb.array(), bb.arrayOffset(), bb.limit());
> {code}
> {{getKey()}} creates to ByteBuffers in ScannerV2 and makes a deep copy of the
> key in EncodedScannerV2.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9727) HBase Rest Server - DELETE scanner operation is a no-op
[ https://issues.apache.org/jira/browse/HBASE-9727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9727: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to all branches. Thanks [~adityakishore]. > HBase Rest Server - DELETE scanner operation is a no-op > --- > > Key: HBASE-9727 > URL: https://issues.apache.org/jira/browse/HBASE-9727 > Project: HBase > Issue Type: Bug > Components: REST >Affects Versions: 0.95.2, 0.94.12 >Reporter: Aditya Kishore >Assignee: Aditya Kishore > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: HBASE-9727_94.txt, HBASE-9727.patch, HBASE-9727.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-4811:
-
Attachment: 4811-0.94-v22.txt
Rebased 0.94 patch with the perf improvement Chunhui mentioned.
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v3.txt, 4811-trunk-v10.txt,
> 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
> hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
> hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
> hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
> hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
> hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> NOTE: when setting reversed as true for a client scan, you must set the start
> row, else will throw exception. Through {@link
> Scan#createBiggestByteArray(int)},you could get a big enough byte array as
> the start row
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-4811:
-
Status: Open (was: Patch Available)
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.94.7, 0.20.6
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 4811-0.94-v22.txt, 4811-0.94-v3.txt, 4811-trunk-v10.txt,
> 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
> hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
> hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
> hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
> hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
> hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> NOTE: when setting reversed as true for a client scan, you must set the start
> row, else will throw exception. Through {@link
> Scan#createBiggestByteArray(int)},you could get a big enough byte array as
> the start row
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9272) A parallel, unordered scanner
[ https://issues.apache.org/jira/browse/HBASE-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-9272: - Fix Version/s: 0.96.1 0.94.13 0.98.0 > A parallel, unordered scanner > - > > Key: HBASE-9272 > URL: https://issues.apache.org/jira/browse/HBASE-9272 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Minor > Fix For: 0.98.0, 0.94.13, 0.96.1 > > Attachments: 9272-0.94.txt, 9272-0.94-v2.txt, 9272-0.94-v3.txt, > 9272-0.94-v4.txt, 9272-trunk.txt, 9272-trunk-v2.txt, 9272-trunk-v3.txt, > ParallelClientScanner.java, ParallelClientScanner.java > > > The contract of ClientScanner is to return rows in sort order. That limits > the order in which region can be scanned. > I propose a simple ParallelScanner that does not have this requirement and > queries regions in parallel, return whatever gets returned first. > This is generally useful for scans that filter a lot of data on the server, > or in cases where the client can very quickly react to the returned data. > I have a simple prototype (doesn't do error handling right, and might be a > bit heavy on the synchronization side - it used a BlockingQueue to hand data > between the client using the scanner and the threads doing the scanning, it > also could potentially starve some scanners long enugh to time out at the > server). > On the plus side, it's only a 130 lines of code. :) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800215#comment-13800215
]
Lars Hofhansl commented on HBASE-4811:
--
[~jmspaggi], any chance to run this through your routine. Need to make sure
this does not regress the forward scanning.
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 4811-0.94-v3.txt, 4811-trunk-v10.txt,
> 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
> hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
> hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
> hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
> hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
> hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> NOTE: when setting reversed as true for a client scan, you must set the start
> row, else will throw exception. Through {@link
> Scan#createBiggestByteArray(int)},you could get a big enough byte array as
> the start row
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-4811:
-
Fix Version/s: 0.96.1
0.94.13
> Support reverse Scan
>
>
> Key: HBASE-4811
> URL: https://issues.apache.org/jira/browse/HBASE-4811
> Project: HBase
> Issue Type: New Feature
> Components: Client
>Affects Versions: 0.20.6, 0.94.7
>Reporter: John Carrino
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.94.13, 0.96.1
>
> Attachments: 4811-0.94-v3.txt, 4811-trunk-v10.txt,
> 4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
> v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
> hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
> hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
> hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
> hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
> hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
> hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
> hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
> hbase-4811-trunkv9.patch
>
>
> Reversed scan means scan the rows backward.
> And StartRow bigger than StopRow in a reversed scan.
> For example, for the following rows:
> aaa/c1:q1/value1
> aaa/c1:q2/value2
> bbb/c1:q1/value1
> bbb/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> eee/c1:q1/value1
> eee/c1:q2/value2
> you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
> Scan scan = new Scan();
> scan.setStartRow('ddd');
> scan.setStopRow('bbb');
> scan.setReversed(true);
> for(Result result:htable.getScanner(scan)){
> System.out.println(result);
> }
> Aslo you could do the reversed scan with shell like this:
> hbase> scan 'table',{REVERSED => true,STARTROW=>'ddd', STOPROW=>'bbb'}
> And the output is:
> ddd/c1:q1/value1
> ddd/c1:q2/value2
> ccc/c1:q1/value1
> ccc/c1:q2/value2
> NOTE: when setting reversed as true for a client scan, you must set the start
> row, else will throw exception. Through {@link
> Scan#createBiggestByteArray(int)},you could get a big enough byte array as
> the start row
> All the documentation I find about HBase says that if you want forward and
> reverse scans you should just build 2 tables and one be ascending and one
> descending. Is there a fundamental reason that HBase only supports forward
> Scan? It seems like a lot of extra space overhead and coding overhead (to
> keep them in sync) to support 2 tables.
> I am assuming this has been discussed before, but I can't find the
> discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9809) RegionTooBusyException should provide region name which was too busy
Ted Yu created HBASE-9809:
-
Summary: RegionTooBusyException should provide region name which
was too busy
Key: HBASE-9809
URL: https://issues.apache.org/jira/browse/HBASE-9809
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Under this thread: http://search-hadoop.com/m/WSfKp1yJOFJ, John showed log from
LoadIncrementalHFiles where the following is a snippet:
{code}
04:18:07,110 INFO LoadIncrementalHFiles:451 - Trying to load
hfile=hdfs://pc08.pool.ifis.uni-luebeck.de:8020/tmp/bulkLoadDirectory/PO_S_rowBufferHFile/Hexa/_tmp/PO_S,9.bottom
first=,"emulates drylot
births"^^ last=
[jira] [Created] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
Ted Yu created HBASE-9808:
-
Summary: org.apache.hadoop.hbase.rest.PerformanceEvaluation is out
of sync with org.apache.hadoop.hbase.PerformanceEvaluation
Key: HBASE-9808
URL: https://issues.apache.org/jira/browse/HBASE-9808
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Here is list of JIRAs whose fixes might have gone into
rest.PerformanceEvaluation :
{code}
r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
HBASE-9663 PerformanceEvaluation does not properly honor specified table name
parameter
r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
HBASE-9662 PerformanceEvaluation input do not handle tags properties
r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
HBASE-8496 - Implement tags and the internals of how a tag should look like
(Ram)
r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a dependency
to MiniDFSCluster
r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
HBASE-9521 clean clearBufferOnFail behavior and deprecate it
r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
HBASE-9330 Refactor PE to create HTable the correct way
{code}
Long term, we may consider consolidating the two PerformanceEvaluation classes
so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9806) Add PerfEval tool for BlockCache
[ https://issues.apache.org/jira/browse/HBASE-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13800124#comment-13800124 ] Jean-Marc Spaggiari commented on HBASE-9806: Can we not do this via PE? Having a PerfTest driver might be good. See https://issues.apache.org/jira/browse/HBASE-9599 . That way we know easily where to look at for perf testing? Instead of looking at 20 different classes? > Add PerfEval tool for BlockCache > > > Key: HBASE-9806 > URL: https://issues.apache.org/jira/browse/HBASE-9806 > Project: HBase > Issue Type: Test > Components: Performance, test >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk > Attachments: HBASE-9806.00.patch > > > We have at least three different block caching layers with myriad > configuration settings. Let's add a tool for evaluating memory allocations > and configuration combinations with different access patterns. -- This message was sent by Atlassian JIRA (v6.1#6144)
