[jira] [Assigned] (HBASE-21602) Procedure v2 access control
[ https://issues.apache.org/jira/browse/HBASE-21602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu reassigned HBASE-21602: Assignee: Yi Mei > Procedure v2 access control > --- > > Key: HBASE-21602 > URL: https://issues.apache.org/jira/browse/HBASE-21602 > Project: HBase > Issue Type: Task >Reporter: Yi Mei >Assignee: Yi Mei >Priority: Major > > Now the access control service (grant and revoke) is done by regionserver > with acl region. The process of grant/revoke is: client call grant, > regionserver with acl region will check and then put permission row in acl > table, and also write to the ZooKeeper acl node(/hbase/acl). Each > regionserver watch acl znode and update local acl cache when node children > changed. > Create this issus to use procedure v2 framework and reduce zk depencency. Any > suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21578) Fix wrong throttling exception for capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720985#comment-16720985
]
Guanghao Zhang commented on HBASE-21578:
+1
> Fix wrong throttling exception for capacity unit
>
>
> Key: HBASE-21578
> URL: https://issues.apache.org/jira/browse/HBASE-21578
> Project: HBase
> Issue Type: Bug
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-21578.master.001.patch,
> HBASE-21578.master.002.patch
>
>
> HBASE-21034 provides a new throttle type: capacity unit, but the throttling
> exception is confusing:
>
> {noformat}
> 2018-12-11 14:38:41,503 DEBUG [Time-limited test]
> client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=7,
> started=0 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException: write size limit
> exceeded - wait 10sec
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwThrottlingException(RpcThrottlingException.java:106)
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwWriteSizeExceeded(RpcThrottlingException.java:96)
> at
> org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:179){noformat}
> Need to make the exception more clearly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21578) Fix wrong throttling exception for capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-21578:
---
Issue Type: Sub-task (was: Bug)
Parent: HBASE-16707
> Fix wrong throttling exception for capacity unit
>
>
> Key: HBASE-21578
> URL: https://issues.apache.org/jira/browse/HBASE-21578
> Project: HBase
> Issue Type: Sub-task
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-21578.master.001.patch,
> HBASE-21578.master.002.patch
>
>
> HBASE-21034 provides a new throttle type: capacity unit, but the throttling
> exception is confusing:
>
> {noformat}
> 2018-12-11 14:38:41,503 DEBUG [Time-limited test]
> client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=7,
> started=0 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException: write size limit
> exceeded - wait 10sec
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwThrottlingException(RpcThrottlingException.java:106)
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwWriteSizeExceeded(RpcThrottlingException.java:96)
> at
> org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:179){noformat}
> Need to make the exception more clearly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720976#comment-16720976
]
Zheng Hu commented on HBASE-21520:
--
Patch.v4 : make the TestMultiColumnScanner to be an abstract class, it's no
need to run again.
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21520.v1.patch, HBASE-21520.v2.patch,
> HBASE-21520.v3.patch, HBASE-21520.v4.patch, TestMultiColumnScanner.png,
> rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21603) Move access control service from regionserver to master
Yi Mei created HBASE-21603: -- Summary: Move access control service from regionserver to master Key: HBASE-21603 URL: https://issues.apache.org/jira/browse/HBASE-21603 Project: HBase Issue Type: Sub-task Reporter: Yi Mei Create a sub task to move access control service from regionserver to master. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21602) Procedure v2 access control
Yi Mei created HBASE-21602: -- Summary: Procedure v2 access control Key: HBASE-21602 URL: https://issues.apache.org/jira/browse/HBASE-21602 Project: HBase Issue Type: Task Reporter: Yi Mei Now the access control service (grant and revoke) is done by regionserver with acl region. The process of grant/revoke is: client call grant, regionserver with acl region will check and then put permission row in acl table, and also write to the ZooKeeper acl node(/hbase/acl). Each regionserver watch acl znode and update local acl cache when node children changed. Create this issus to use procedure v2 framework and reduce zk depencency. Any suggestions are welcomed. Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21520:
-
Attachment: HBASE-21520.v4.patch
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21520.v1.patch, HBASE-21520.v2.patch,
> HBASE-21520.v3.patch, HBASE-21520.v4.patch, TestMultiColumnScanner.png,
> rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720961#comment-16720961
]
Hadoop QA commented on HBASE-21520:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} hbase-server: The patch generated 0 new + 5
unchanged - 4 fixed = 5 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
48s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 23s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 26s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestCheckTestClasses |
| | hadoop.hbase.regionserver.TestWALLockup |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21520 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951761/HBASE-21520.v3.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 5a3d92f75010 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3ff274e22e |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/15279/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15279/testReport/ |
| Max. process+thread
[jira] [Commented] (HBASE-21594) Requested block is out of range when reading hfile
[
https://issues.apache.org/jira/browse/HBASE-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720951#comment-16720951
]
Zheng Hu commented on HBASE-21594:
--
Great，will take a look later.
> Requested block is out of range when reading hfile
> --
>
> Key: HBASE-21594
> URL: https://issues.apache.org/jira/browse/HBASE-21594
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.98.10
>Reporter: ChenKai
>Priority: Major
> Attachments: 2e4db6a7f8d14f2bb846781b07750dc1_SeqId_84_.bak,
> image-2018-12-13-20-11-00-818.png
>
>
> My HFiles are generated by Spark HBaseBulkLoad. And then when i read a few of
> them(or hbase do compact), i encounter the following exceptions.
>
> {code:java}
> Exception in thread "main" java.io.IOException: Requested block is out of
> range: 77329641, lastDataBlockOffset: 77329641,
> trailer.getLoadOnOpenDataOffset: 77329641
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:396)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:734)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:859)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:854)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:871)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:891)
> at io.patamon.hbase.test.read.TestHFileRead.main(TestHFileRead.java:49)
> {code}
> Looks like `lastDataBlockOffset` is equals to
> `trailer.getLoadOnOpenDataOffset`. Could anyone help me? Thanks very much.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21587) Improve support for protobuf 3
[
https://issues.apache.org/jira/browse/HBASE-21587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720933#comment-16720933
]
Anoop Sam John commented on HBASE-21587:
For HBase 2.x we need PB 3 version.. I dont think we need to support PB 2.x
there.. Ya for this only we have done the shading work and make sure that there
wont be class conflict
> Improve support for protobuf 3
> --
>
> Key: HBASE-21587
> URL: https://issues.apache.org/jira/browse/HBASE-21587
> Project: HBase
> Issue Type: Improvement
>Reporter: Mya Pitzeruse
>Priority: Major
>
> {{HBaseZeroCopyByteString}} extends {{LiteralByteString}} which was removed
> in protobuf 3. The class was marked as package private, so hbase needed to do
> a package trick to get to the underlying class.
> [https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-protocol/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java#L18]
> ejona86 references this problem in grpc-java:
> [https://github.com/grpc/grpc-java/issues/3035#issuecomment-360851817]
> The {{HBaseZeroCopyByteString}} class appears to only be used by
> {{ByteStringer}} class.
> [https://github.com/apache/hbase/blob/025ddce868eb06b4072b5152c5ffae5a01e7ae30/hbase-protocol/src/main/java/org/apache/hadoop/hbase/util/ByteStringer.java#L42-L49]
> I think a simple change can be made to the {{ByteStringer}} class to support
> both proto2 and proto3.
> Proto3 offers an {{UnsafeByteOperations}} class that can be used in place of
> the {{HBaseZeroCopyByteString}} class.
> [https://github.com/protocolbuffers/protobuf/blob/master/java/core/src/main/java/com/google/protobuf/UnsafeByteOperations.java#L97]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21578) Fix wrong throttling exception for capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720936#comment-16720936
]
Hadoop QA commented on HBASE-21578:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
13s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 11s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
12s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}140m
31s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}201m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21578 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951751/HBASE-21578.master.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ad10c292fc0e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3ff274e22e |
|
[jira] [Commented] (HBASE-21594) Requested block is out of range when reading hfile
[
https://issues.apache.org/jira/browse/HBASE-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720927#comment-16720927
]
ChenKai commented on HBASE-21594:
-
You can read the hfile from the
attachments(2e4db6a7f8d14f2bb846781b07750dc1_SeqId_84_.bak). And the test code:
{code:java}
public static void main(String[] args) throws IOException,
ClassNotFoundException {
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
HFile.Reader reader = HFile.createReader(fs,
new
Path("file:///xxx/xxx/2e4db6a7f8d14f2bb846781b07750dc1_SeqId_84_.bak"), new
CacheConfig(conf), conf);
// 1. Trailer
FixedFileTrailer trailer = reader.getTrailer();
System.out.println(trailer.getMajorVersion());
System.out.println(trailer.getFirstDataBlockOffset());
System.out.println(trailer.getLastDataBlockOffset());
System.out.println(trailer.getLoadOnOpenDataOffset());
// Load up the index.
reader.loadFileInfo();
// Get a scanner that caches and that does not use pread.
HFileScanner scanner = reader.getScanner(false, false);
// Align scanner at start of the file.
scanner.seekTo();
// read data
while (scanner.next()) {
Cell cell = scanner.getKeyValue();
String row = Bytes.toString(CellUtil.cloneRow(cell));
String cf = Bytes.toString(CellUtil.cloneFamily(cell));
String col = Bytes.toString(CellUtil.cloneQualifier(cell));
// String value = Bytes.toString(CellUtil.cloneValue(cell));
// System.out.println(row + " => " + cf + ":" + col + " = " + value);
}{code}
Thanks a lot.
> Requested block is out of range when reading hfile
> --
>
> Key: HBASE-21594
> URL: https://issues.apache.org/jira/browse/HBASE-21594
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.98.10
>Reporter: ChenKai
>Priority: Major
> Attachments: 2e4db6a7f8d14f2bb846781b07750dc1_SeqId_84_.bak,
> image-2018-12-13-20-11-00-818.png
>
>
> My HFiles are generated by Spark HBaseBulkLoad. And then when i read a few of
> them(or hbase do compact), i encounter the following exceptions.
>
> {code:java}
> Exception in thread "main" java.io.IOException: Requested block is out of
> range: 77329641, lastDataBlockOffset: 77329641,
> trailer.getLoadOnOpenDataOffset: 77329641
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:396)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:734)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:859)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:854)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:871)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:891)
> at io.patamon.hbase.test.read.TestHFileRead.main(TestHFileRead.java:49)
> {code}
> Looks like `lastDataBlockOffset` is equals to
> `trailer.getLoadOnOpenDataOffset`. Could anyone help me? Thanks very much.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21594) Requested block is out of range when reading hfile
[
https://issues.apache.org/jira/browse/HBASE-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ChenKai updated HBASE-21594:
Attachment: 2e4db6a7f8d14f2bb846781b07750dc1_SeqId_84_.bak
> Requested block is out of range when reading hfile
> --
>
> Key: HBASE-21594
> URL: https://issues.apache.org/jira/browse/HBASE-21594
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.98.10
>Reporter: ChenKai
>Priority: Major
> Attachments: 2e4db6a7f8d14f2bb846781b07750dc1_SeqId_84_.bak,
> image-2018-12-13-20-11-00-818.png
>
>
> My HFiles are generated by Spark HBaseBulkLoad. And then when i read a few of
> them(or hbase do compact), i encounter the following exceptions.
>
> {code:java}
> Exception in thread "main" java.io.IOException: Requested block is out of
> range: 77329641, lastDataBlockOffset: 77329641,
> trailer.getLoadOnOpenDataOffset: 77329641
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:396)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:734)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:859)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:854)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:871)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:891)
> at io.patamon.hbase.test.read.TestHFileRead.main(TestHFileRead.java:49)
> {code}
> Looks like `lastDataBlockOffset` is equals to
> `trailer.getLoadOnOpenDataOffset`. Could anyone help me? Thanks very much.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21583) Insert info Level0 instead of throwing Exceptions when doing bulkload on tables with StripeCompaction enabled
[
https://issues.apache.org/jira/browse/HBASE-21583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenxu updated HBASE-21583:
---
Assignee: chenxu
Status: Patch Available (was: Open)
> Insert info Level0 instead of throwing Exceptions when doing bulkload on
> tables with StripeCompaction enabled
> -
>
> Key: HBASE-21583
> URL: https://issues.apache.org/jira/browse/HBASE-21583
> Project: HBase
> Issue Type: Bug
>Reporter: chenxu
>Assignee: chenxu
>Priority: Major
> Attachments: HBASE-21583-master-v1.patch, HBASE-21583-master-v2.patch
>
>
> Table-A is an existing table that enable StripeCompaction，when create
> Table-B(also enable StripeCompaction) and bulkload data copy from Table-A，the
> following exception will occur.
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=1, exceptions:
> Mon Dec 10 21:26:34 CST 2018,
> RpcRetryingCaller{globalStartTime=158389041, pause=100, retries=1},
> java.io.IOException: java.io.IOException: Newly created stripes do not cover
> the entire key space.
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.processNewCandidateStripes(StripeStoreFileManager.java:886)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.mergeResults(StripeStoreFileManager.java:717)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.access$100(StripeStoreFileManager.java:690)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager.insertNewFiles(StripeStoreFileManager.java:151)
> at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:891)
> at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:869)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5524)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:1980)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33650)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2208)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:116)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:135)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:110)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21583) Insert info Level0 instead of throwing Exceptions when doing bulkload on tables with StripeCompaction enabled
[
https://issues.apache.org/jira/browse/HBASE-21583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenxu updated HBASE-21583:
---
Summary: Insert info Level0 instead of throwing Exceptions when doing
bulkload on tables with StripeCompaction enabled (was: bulkload maybe failed
if StripeCompaction enabled but Region has no stripes)
> Insert info Level0 instead of throwing Exceptions when doing bulkload on
> tables with StripeCompaction enabled
> -
>
> Key: HBASE-21583
> URL: https://issues.apache.org/jira/browse/HBASE-21583
> Project: HBase
> Issue Type: Bug
>Reporter: chenxu
>Priority: Major
> Attachments: HBASE-21583-master-v1.patch, HBASE-21583-master-v2.patch
>
>
> Table-A is an existing table that enable StripeCompaction，when create
> Table-B(also enable StripeCompaction) and bulkload data copy from Table-A，the
> following exception will occur.
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=1, exceptions:
> Mon Dec 10 21:26:34 CST 2018,
> RpcRetryingCaller{globalStartTime=158389041, pause=100, retries=1},
> java.io.IOException: java.io.IOException: Newly created stripes do not cover
> the entire key space.
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.processNewCandidateStripes(StripeStoreFileManager.java:886)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.mergeResults(StripeStoreFileManager.java:717)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.access$100(StripeStoreFileManager.java:690)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager.insertNewFiles(StripeStoreFileManager.java:151)
> at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:891)
> at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:869)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5524)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:1980)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33650)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2208)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:116)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:135)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:110)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21583) bulkload maybe failed if StripeCompaction enabled but Region has no stripes
[
https://issues.apache.org/jira/browse/HBASE-21583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenxu updated HBASE-21583:
---
Attachment: HBASE-21583-master-v2.patch
> bulkload maybe failed if StripeCompaction enabled but Region has no stripes
> ---
>
> Key: HBASE-21583
> URL: https://issues.apache.org/jira/browse/HBASE-21583
> Project: HBase
> Issue Type: Bug
>Reporter: chenxu
>Priority: Major
> Attachments: HBASE-21583-master-v1.patch, HBASE-21583-master-v2.patch
>
>
> Table-A is an existing table that enable StripeCompaction，when create
> Table-B(also enable StripeCompaction) and bulkload data copy from Table-A，the
> following exception will occur.
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=1, exceptions:
> Mon Dec 10 21:26:34 CST 2018,
> RpcRetryingCaller{globalStartTime=158389041, pause=100, retries=1},
> java.io.IOException: java.io.IOException: Newly created stripes do not cover
> the entire key space.
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.processNewCandidateStripes(StripeStoreFileManager.java:886)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.mergeResults(StripeStoreFileManager.java:717)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager$CompactionOrFlushMergeCopy.access$100(StripeStoreFileManager.java:690)
> at
> org.apache.hadoop.hbase.regionserver.StripeStoreFileManager.insertNewFiles(StripeStoreFileManager.java:151)
> at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:891)
> at org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:869)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5524)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.bulkLoadHFile(RSRpcServices.java:1980)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33650)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2208)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:116)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:135)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:110)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21594) Requested block is out of range when reading hfile
[
https://issues.apache.org/jira/browse/HBASE-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720913#comment-16720913
]
Zheng Hu commented on HBASE-21594:
--
Would you mind to share you UT ? I'll try to reproduce this bug and see what's
wrong with this.
> Requested block is out of range when reading hfile
> --
>
> Key: HBASE-21594
> URL: https://issues.apache.org/jira/browse/HBASE-21594
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.98.10
>Reporter: ChenKai
>Priority: Major
> Attachments: image-2018-12-13-20-11-00-818.png
>
>
> My HFiles are generated by Spark HBaseBulkLoad. And then when i read a few of
> them(or hbase do compact), i encounter the following exceptions.
>
> {code:java}
> Exception in thread "main" java.io.IOException: Requested block is out of
> range: 77329641, lastDataBlockOffset: 77329641,
> trailer.getLoadOnOpenDataOffset: 77329641
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:396)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:734)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:859)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:854)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:871)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:891)
> at io.patamon.hbase.test.read.TestHFileRead.main(TestHFileRead.java:49)
> {code}
> Looks like `lastDataBlockOffset` is equals to
> `trailer.getLoadOnOpenDataOffset`. Could anyone help me? Thanks very much.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21594) Requested block is out of range when reading hfile
[
https://issues.apache.org/jira/browse/HBASE-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720877#comment-16720877
]
Zheng Hu commented on HBASE-21594:
--
So you used a new hbase-client ( hbase 1.2 ) to bulk load into a older
hbase-server ? The server's version is too old now, if possible, you can try
to upgrade the hbase-server to the hbase1.2 too.
> Requested block is out of range when reading hfile
> --
>
> Key: HBASE-21594
> URL: https://issues.apache.org/jira/browse/HBASE-21594
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.98.10
>Reporter: ChenKai
>Priority: Major
> Attachments: image-2018-12-13-20-11-00-818.png
>
>
> My HFiles are generated by Spark HBaseBulkLoad. And then when i read a few of
> them(or hbase do compact), i encounter the following exceptions.
>
> {code:java}
> Exception in thread "main" java.io.IOException: Requested block is out of
> range: 77329641, lastDataBlockOffset: 77329641,
> trailer.getLoadOnOpenDataOffset: 77329641
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:396)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:734)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:859)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:854)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:871)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:891)
> at io.patamon.hbase.test.read.TestHFileRead.main(TestHFileRead.java:49)
> {code}
> Looks like `lastDataBlockOffset` is equals to
> `trailer.getLoadOnOpenDataOffset`. Could anyone help me? Thanks very much.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19863) java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter is used
[
https://issues.apache.org/jira/browse/HBASE-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720909#comment-16720909
]
Lars Hofhansl commented on HBASE-19863:
---
Looks back on this... Isn't it cheaper to just detect the fake key the same way
that ScanQueryMatcher does it - by checking the timestamp for
HConstants.OLDEST_TIMESTAMP, instead of doing the compare?
If the last K/V is fake we seek, no need to compare the bytes (which is
expensive).
[~elserj] [~ram_krish] [~anoop.hbase] [~sergey.soldatov]
> java.lang.IllegalStateException: isDelete failed when SingleColumnValueFilter
> is used
> -
>
> Key: HBASE-19863
> URL: https://issues.apache.org/jira/browse/HBASE-19863
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 1.4.1
>Reporter: Sergey Soldatov
>Assignee: Sergey Soldatov
>Priority: Major
> Fix For: 1.5.0, 2.0.0-beta-2, 1.4.3, 2.0.0
>
> Attachments: HBASE-19863-branch-2.patch, HBASE-19863-branch1.patch,
> HBASE-19863-test.patch, HBASE-19863.v2-branch-2.patch,
> HBASE-19863.v3-branch-2.patch, HBASE-19863.v4-branch-2.patch,
> HBASE-19863.v4-master.patch, HBASE-19863.v5-branch-1.4.patch,
> HBASE-19863.v5-branch-1.patch, HBASE-19863.v5-branch-2.patch
>
>
> Under some circumstances scan with SingleColumnValueFilter may fail with an
> exception
> {noformat}
> java.lang.IllegalStateException: isDelete failed: deleteBuffer=C3,
> qualifier=C2, timestamp=1516433595543, comparison result: 1
> at
> org.apache.hadoop.hbase.regionserver.ScanDeleteTracker.isDeleted(ScanDeleteTracker.java:149)
> at
> org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.match(ScanQueryMatcher.java:386)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:545)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5876)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6027)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5814)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2552)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32385)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)
> {noformat}
> Conditions:
> table T with a single column family 0 that uses ROWCOL bloom filter
> (important) and column qualifiers C1,C2,C3,C4,C5.
> When we fill the table for every row we put deleted cell for C3.
> The table has a single region with two HStore:
> A: start row: 0, stop row: 99
> B: start row: 10 stop row: 99
> B has newer versions of rows 10-99. Store files have several blocks each
> (important).
> Store A is the result of major compaction, so it doesn't have any deleted
> cells (important).
> So, we are running a scan like:
> {noformat}
> scan 'T', { COLUMNS => ['0:C3','0:C5'], FILTER => "SingleColumnValueFilter
> ('0','C5',=,'binary:whatever')"}
> {noformat}
> How the scan performs:
> First, we iterate A for rows 0 and 1 without any problems.
> Next, we start to iterate A for row 10, so read the first cell and set hfs
> scanner to A :
> 10:0/C1/0/Put/x but found that we have a newer version of the cell in B :
> 10:0/C1/1/Put/x,
> so we make B as our current store scanner. Since we are looking for
> particular columns
> C3 and C5, we perform the optimization StoreScanner.seekOrSkipToNextColumn
> which
> would run reseek for all store scanners.
> For store A the following magic would happen in requestSeek:
> 1. bloom filter check passesGeneralBloomFilter would set haveToSeek to
> false because row 10 doesn't have C3 qualifier in store A.
> 2. Since we don't have to seek we just create a fake row
> 10:0/C3/OLDEST_TIMESTAMP/Maximum, an optimization that is quite important for
> us and it commented with :
> {noformat}
> // Multi-column Bloom filter optimization.
> // Create a fake key/value, so that this scanner only bubbles up to the
> top
> // of the KeyValueHeap in StoreScanner after we scanned this row/column in
> // all other store files. The query matcher will then just skip this fake
> // key/value and the store scanner will progress to the next column. This
> // is obviously not a "real real"
[jira] [Commented] (HBASE-21594) Requested block is out of range when reading hfile
[
https://issues.apache.org/jira/browse/HBASE-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720906#comment-16720906
]
ChenKai commented on HBASE-21594:
-
[~openinx] I just use version 1.2 to test reading some of these hfiles, our
production is using hbase-client (0.98.13-hadoop2) to bulk load into
hbase-server (0.98.13-hadoop2).
> Requested block is out of range when reading hfile
> --
>
> Key: HBASE-21594
> URL: https://issues.apache.org/jira/browse/HBASE-21594
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.98.10
>Reporter: ChenKai
>Priority: Major
> Attachments: image-2018-12-13-20-11-00-818.png
>
>
> My HFiles are generated by Spark HBaseBulkLoad. And then when i read a few of
> them(or hbase do compact), i encounter the following exceptions.
>
> {code:java}
> Exception in thread "main" java.io.IOException: Requested block is out of
> range: 77329641, lastDataBlockOffset: 77329641,
> trailer.getLoadOnOpenDataOffset: 77329641
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:396)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:734)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:859)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:854)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:871)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:891)
> at io.patamon.hbase.test.read.TestHFileRead.main(TestHFileRead.java:49)
> {code}
> Looks like `lastDataBlockOffset` is equals to
> `trailer.getLoadOnOpenDataOffset`. Could anyone help me? Thanks very much.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21019) Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21019: --- Status: Patch Available (was: Open) > Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler > -- > > Key: HBASE-21019 > URL: https://issues.apache.org/jira/browse/HBASE-21019 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0 >Reporter: Yi Mei >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21019.master.001.patch > > > In HBASE-20965, we propose MasterFifoRpcScheduler to handle RSReport requests > in independent handlers, other requests are handled in other handlers. > To better utilize handlers, we can use StealJobQueue to allow other handlers > to steal jobs from RSReport handlers when there is a few RSReport handlers. > And it's the same for SimpleRpcScheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21019) Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang reassigned HBASE-21019: -- Assignee: Guanghao Zhang > Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler > -- > > Key: HBASE-21019 > URL: https://issues.apache.org/jira/browse/HBASE-21019 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0 >Reporter: Yi Mei >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21019.master.001.patch > > > In HBASE-20965, we propose MasterFifoRpcScheduler to handle RSReport requests > in independent handlers, other requests are handled in other handlers. > To better utilize handlers, we can use StealJobQueue to allow other handlers > to steal jobs from RSReport handlers when there is a few RSReport handlers. > And it's the same for SimpleRpcScheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21019) Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21019: --- Attachment: HBASE-21019.master.001.patch > Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler > -- > > Key: HBASE-21019 > URL: https://issues.apache.org/jira/browse/HBASE-21019 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0 >Reporter: Yi Mei >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21019.master.001.patch > > > In HBASE-20965, we propose MasterFifoRpcScheduler to handle RSReport requests > in independent handlers, other requests are handled in other handlers. > To better utilize handlers, we can use StealJobQueue to allow other handlers > to steal jobs from RSReport handlers when there is a few RSReport handlers. > And it's the same for SimpleRpcScheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21019) Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler
[ https://issues.apache.org/jira/browse/HBASE-21019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21019: --- Summary: Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler (was: Use StealJobQueue to better utilize handlers in RpcScheduer) > Use StealJobQueue to better utilize handlers in MasterFifoRpcScheduler > -- > > Key: HBASE-21019 > URL: https://issues.apache.org/jira/browse/HBASE-21019 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0, 2.2.0 >Reporter: Yi Mei >Priority: Major > > In HBASE-20965, we propose MasterFifoRpcScheduler to handle RSReport requests > in independent handlers, other requests are handled in other handlers. > To better utilize handlers, we can use StealJobQueue to allow other handlers > to steal jobs from RSReport handlers when there is a few RSReport handlers. > And it's the same for SimpleRpcScheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21563) HBase Get Encounters java.lang.IndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HBASE-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu reassigned HBASE-21563:
Assignee: Zheng Hu
> HBase Get Encounters java.lang.IndexOutOfBoundsException
>
>
> Key: HBASE-21563
> URL: https://issues.apache.org/jira/browse/HBASE-21563
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 1.2.0
>Reporter: William Shen
>Assignee: Zheng Hu
>Priority: Major
> Attachments: 67a04bc049be4f58afecdcc0a3ba62ca.tar.gz
>
>
> We've recently encountered issue retrieving data from our HBase cluster, and
> have not had much luck troubleshooting the issue. We narrowed down our issue
> to a single GET, which appears to be caused by FastDiffDeltaEncoder.java
> running into java.lang.IndexOutOfBoundsException.
> Perhaps there is a bug on a corner case for FastDiffDeltaEncoder?
> We are running 1.2.0-cdh5.9.2, and the GET in question is:
> {noformat}
> hbase(main):004:0> get 'qa2.ADGROUPS',
> "\x05\x80\x00\x00\x00\x00\x1F\x54\x9C\x80\x00\x00\x00\x00\x1C\x7D\x45\x00\x04\x80\x00\x00\x00\x00\x1D\x0F\x19\x80\x00\x00\x00\x00\x4A\x64\x6F\x80\x00\x00\x00\x01\xD9\xDB\xCE"
> COLUMNCELL
>
>
> ERROR: java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
> Caused by: java.lang.IndexOutOfBoundsException
> at java.nio.Buffer.checkBounds(Buffer.java:567)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:149)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:465)
> at
> org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:516)
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:618)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.next(HFileReaderV2.java:1277)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:180)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:108)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:588)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:147)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:5706)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:5865)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:5643)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5620)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:5606)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6801)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6779)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2029)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33644)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> ... 3 more {noformat}
> Likewise, running {{ hbase hfile -f -p }} on the specific hfile, a subset of
> kv pairs were printed until the program hits the following exception and
> crashes:
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Unknown code 65
> at org.apache.hadoop.hbase.KeyValue$Type.codeToType(KeyValue.java:259)
> at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:1246)
> at
> org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$ClonedSeekerState.toString(BufferedDataBlockEncoder.java:506)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:382)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:316)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:255)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:677)
> {noformat}
> I have attached the HFile related to this issue for debugging.
[jira] [Updated] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21520:
-
Fix Version/s: 2.0.5
2.1.3
2.2.0
3.0.0
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21520.v1.patch, HBASE-21520.v2.patch,
> HBASE-21520.v3.patch, TestMultiColumnScanner.png, rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21520:
-
Issue Type: Improvement (was: Bug)
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21520.v1.patch, HBASE-21520.v2.patch,
> HBASE-21520.v3.patch, TestMultiColumnScanner.png, rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21520:
-
Status: Patch Available (was: Open)
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-21520.v1.patch, HBASE-21520.v2.patch,
> HBASE-21520.v3.patch, TestMultiColumnScanner.png, rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21520) TestMultiColumnScanner cost long time when using ROWCOL bloom type
[
https://issues.apache.org/jira/browse/HBASE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-21520:
-
Attachment: HBASE-21520.v3.patch
> TestMultiColumnScanner cost long time when using ROWCOL bloom type
> --
>
> Key: HBASE-21520
> URL: https://issues.apache.org/jira/browse/HBASE-21520
> Project: HBase
> Issue Type: Bug
> Components: test
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-21520.v1.patch, HBASE-21520.v2.patch,
> HBASE-21520.v3.patch, TestMultiColumnScanner.png, rowcol.txt
>
>
> The TestMultiColumnScanner is easy to be timeout, you can see HBASE-21517.
> In my localhost, when I set the parameters to be {
> Compression.Algorithm.NONE, BloomType.ROW, false }, it took about 5 seconds.
> but if I set the parameters to be { Compression.Algorithm.NONE,
> BloomType.ROWCOL, false }, it would take about 45 seconds, which means
> ROWCOL cost much more time than ROW.
> Need to find out what's wrong with this ut.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21590) Optimize trySkipToNextColumn in StoreScanner a bit
[ https://issues.apache.org/jira/browse/HBASE-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720847#comment-16720847 ] Guanghao Zhang commented on HBASE-21590: Ping [~stack] for branch-2.0 and branch-2.1. > Optimize trySkipToNextColumn in StoreScanner a bit > -- > > Key: HBASE-21590 > URL: https://issues.apache.org/jira/browse/HBASE-21590 > Project: HBase > Issue Type: Improvement > Components: Performance, Scanners >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10 > > Attachments: 21590-1.5.txt, HBASE-21590-master.txt > > > See latest comment on HBASE-17958 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21020) Determine WAL API changes for replication
[
https://issues.apache.org/jira/browse/HBASE-21020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720851#comment-16720851
]
Hadoop QA commented on HBASE-21020:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-20952 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
10s{color} | {color:green} HBASE-20952 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
40s{color} | {color:green} HBASE-20952 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
42s{color} | {color:green} HBASE-20952 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
34s{color} | {color:blue} hbase-server in HBASE-20952 has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} HBASE-20952 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 2s{color}
| {color:red} hbase-server generated 1 new + 187 unchanged - 1 fixed = 188
total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
13s{color} | {color:red} hbase-server: The patch generated 41 new + 192
unchanged - 4 fixed = 233 total (was 196) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
8s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 20s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s{color} | {color:green} hbase-server generated 0 new + 0 unchanged - 1 fixed
= 0 total (was 1) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
33s{color} | {color:red} hbase-server generated 10 new + 2 unchanged - 0 fixed
= 12 total (was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
44s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}175m 14s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}227m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.regionserver.TestSerialReplicationEndpoint |
| | hadoop.hbase.replication.TestMultiSlaveReplication |
| | hadoop.hbase.replication.TestSerialSyncReplication |
| |
[jira] [Assigned] (HBASE-21600) Investigate non-null check on FSWALIdentity
[
https://issues.apache.org/jira/browse/HBASE-21600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reid Chan reassigned HBASE-21600:
-
Assignee: Reid Chan
> Investigate non-null check on FSWALIdentity
> ---
>
> Key: HBASE-21600
> URL: https://issues.apache.org/jira/browse/HBASE-21600
> Project: HBase
> Issue Type: Task
>Reporter: Josh Elser
>Assignee: Reid Chan
>Priority: Trivial
> Fix For: HBASE-20952
>
>
> {code:java}
> public FSWALIdentity(Path path)
> public FSWALIdentity(String name)
> {code}
> bq. Can we add a pre-null check or annotation NotNullable or javadoc to raise
> attention of no-null? Passing a null object to WALIdentity makes no sense to
> me.
> Reid had the above suggestion on HBASE-21246. Should make a check throughout
> the code and make sure nothing else breaks if we start asserting that the
> path/name is always non-null (I fear something might :P).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21600) Investigate non-null check on FSWALIdentity
[
https://issues.apache.org/jira/browse/HBASE-21600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720844#comment-16720844
]
Reid Chan commented on HBASE-21600:
---
Let me finish this trivial, since i bring this up.
> Investigate non-null check on FSWALIdentity
> ---
>
> Key: HBASE-21600
> URL: https://issues.apache.org/jira/browse/HBASE-21600
> Project: HBase
> Issue Type: Task
>Reporter: Josh Elser
>Priority: Trivial
> Fix For: HBASE-20952
>
>
> {code:java}
> public FSWALIdentity(Path path)
> public FSWALIdentity(String name)
> {code}
> bq. Can we add a pre-null check or annotation NotNullable or javadoc to raise
> attention of no-null? Passing a null object to WALIdentity makes no sense to
> me.
> Reid had the above suggestion on HBASE-21246. Should make a check throughout
> the code and make sure nothing else breaks if we start asserting that the
> path/name is always non-null (I fear something might :P).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21594) Requested block is out of range when reading hfile
[
https://issues.apache.org/jira/browse/HBASE-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720090#comment-16720090
]
ChenKai edited comment on HBASE-21594 at 12/14/18 2:14 AM:
---
!image-2018-12-13-20-11-00-818.png|width=280,height=100!
here shoud be this? Because i see next block is ROOT_INDEX (no meta blocks).
{code:java}
if (dataBlockOffset < 0 || dataBlockOffset > trailerOffset) {
}{code}
was (Author: 514793...@qq.com):
!image-2018-12-13-20-11-00-818.png|width=280,height=100!
here shoud be this? Because i see next block is ROOT_INDEX.
{code:java}
if (dataBlockOffset < 0 || dataBlockOffset > trailerOffset) {
}{code}
> Requested block is out of range when reading hfile
> --
>
> Key: HBASE-21594
> URL: https://issues.apache.org/jira/browse/HBASE-21594
> Project: HBase
> Issue Type: Bug
> Components: HFile
>Affects Versions: 0.98.10
>Reporter: ChenKai
>Priority: Major
> Attachments: image-2018-12-13-20-11-00-818.png
>
>
> My HFiles are generated by Spark HBaseBulkLoad. And then when i read a few of
> them(or hbase do compact), i encounter the following exceptions.
>
> {code:java}
> Exception in thread "main" java.io.IOException: Requested block is out of
> range: 77329641, lastDataBlockOffset: 77329641,
> trailer.getLoadOnOpenDataOffset: 77329641
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:396)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.readNextDataBlock(HFileReaderV2.java:734)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.isNextBlock(HFileReaderV2.java:859)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.positionForNextBlock(HFileReaderV2.java:854)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2._next(HFileReaderV2.java:871)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:891)
> at io.patamon.hbase.test.read.TestHFileRead.main(TestHFileRead.java:49)
> {code}
> Looks like `lastDataBlockOffset` is equals to
> `trailer.getLoadOnOpenDataOffset`. Could anyone help me? Thanks very much.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21601) corrupted WAL is not handled in all places (NegativeArraySizeException)
[
https://issues.apache.org/jira/browse/HBASE-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720835#comment-16720835
]
Zheng Hu commented on HBASE-21601:
--
Could you find method to reproduce this bug ? If sure, will be easy to fix...
btw, I am trying to fix this issue: HBASE-21379
> corrupted WAL is not handled in all places (NegativeArraySizeException)
> ---
>
> Key: HBASE-21601
> URL: https://issues.apache.org/jira/browse/HBASE-21601
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> 2018-12-13 17:01:12,208 ERROR [RS_LOG_REPLAY_OPS-regionserver/...]
> executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
> java.lang.RuntimeException: java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hbase.wal.WALSplitter$PipelineController.checkForErrors(WALSplitter.java:846)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$OutputSink.finishWriting(WALSplitter.java:1203)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.finishWritingAndClose(WALSplitter.java:1267)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:349)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:196)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:178)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:90)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NegativeArraySizeException
> at org.apache.hadoop.hbase.CellUtil.cloneFamily(CellUtil.java:113)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.filterCellByStore(WALSplitter.java:1542)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.appendBuffer(WALSplitter.java:1586)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.append(WALSplitter.java:1560)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.writeBuffer(WALSplitter.java:1085)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.doRun(WALSplitter.java:1077)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.run(WALSplitter.java:1047)
> {noformat}
> Unfortunately I cannot share the file.
> The issue appears to be straightforward - for whatever reason the family
> length is negative. Not sure how such a cell got created, I suspect the file
> was corrupted.
> {code}
> byte[] output = new byte[cell.getFamilyLength()];
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21564) race condition in WAL rolling resulting in size-based rolling getting stuck
[ https://issues.apache.org/jira/browse/HBASE-21564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720830#comment-16720830 ] Sergey Shelukhin commented on HBASE-21564: -- The test failures are actually real but I'm having really hard time debugging them... will update the patch eventually. > race condition in WAL rolling resulting in size-based rolling getting stuck > --- > > Key: HBASE-21564 > URL: https://issues.apache.org/jira/browse/HBASE-21564 > Project: HBase > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HBASE-21564.master.001.patch, > HBASE-21564.master.002.patch, HBASE-21564.master.003.patch, > HBASE-21564.master.004.patch > > > Manifests at least with AsyncFsWriter. > There's a window after LogRoller replaces the writer in the WAL, but before > it sets the rollLog boolean to false in the finally, where the WAL class can > request another log roll (it can happen in particular when the logs are > getting archived in the LogRoller thread, and there's high write volume > causing the logs to roll quickly). > LogRoller will blindly reset the rollLog flag in finally and "forget" about > this request. > AsyncWAL in turn never requests it again because its own rollRequested field > is set and it expects a callback. Logs don't get rolled until a periodic roll > is triggered after that. > The acknowledgment of roll requests by LogRoller should be atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21578) Fix wrong throttling exception for capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Mei reassigned HBASE-21578:
--
Assignee: Yi Mei
> Fix wrong throttling exception for capacity unit
>
>
> Key: HBASE-21578
> URL: https://issues.apache.org/jira/browse/HBASE-21578
> Project: HBase
> Issue Type: Bug
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-21578.master.001.patch
>
>
> HBASE-21034 provides a new throttle type: capacity unit, but the throttling
> exception is confusing:
>
> {noformat}
> 2018-12-11 14:38:41,503 DEBUG [Time-limited test]
> client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=7,
> started=0 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException: write size limit
> exceeded - wait 10sec
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwThrottlingException(RpcThrottlingException.java:106)
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwWriteSizeExceeded(RpcThrottlingException.java:96)
> at
> org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:179){noformat}
> Need to make the exception more clearly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21601) corrupted WAL is not handled in all places
Sergey Shelukhin created HBASE-21601:
Summary: corrupted WAL is not handled in all places
Key: HBASE-21601
URL: https://issues.apache.org/jira/browse/HBASE-21601
Project: HBase
Issue Type: Bug
Reporter: Sergey Shelukhin
{noformat}
2018-12-13 17:01:12,208 ERROR [RS_LOG_REPLAY_OPS-regionserver/...]
executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
java.lang.RuntimeException: java.lang.NegativeArraySizeException
at
org.apache.hadoop.hbase.wal.WALSplitter$PipelineController.checkForErrors(WALSplitter.java:846)
at
org.apache.hadoop.hbase.wal.WALSplitter$OutputSink.finishWriting(WALSplitter.java:1203)
at
org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.finishWritingAndClose(WALSplitter.java:1267)
at
org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:349)
at
org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:196)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:178)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:90)
at
org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NegativeArraySizeException
at org.apache.hadoop.hbase.CellUtil.cloneFamily(CellUtil.java:113)
at
org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.filterCellByStore(WALSplitter.java:1542)
at
org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.appendBuffer(WALSplitter.java:1586)
at
org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.append(WALSplitter.java:1560)
at
org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.writeBuffer(WALSplitter.java:1085)
at
org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.doRun(WALSplitter.java:1077)
at
org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.run(WALSplitter.java:1047)
{noformat}
Unfortunately I cannot share the file.
The issue appears to be straightforward - for whatever reason the family length
is negative. Not sure how such a cell got created, I suspect the file was
corrupted.
{code}
byte[] output = new byte[cell.getFamilyLength()];
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21601) corrupted WAL is not handled in all places (NegativeArraySizeException)
[
https://issues.apache.org/jira/browse/HBASE-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HBASE-21601:
-
Summary: corrupted WAL is not handled in all places
(NegativeArraySizeException) (was: corrupted WAL is not handled in all places)
> corrupted WAL is not handled in all places (NegativeArraySizeException)
> ---
>
> Key: HBASE-21601
> URL: https://issues.apache.org/jira/browse/HBASE-21601
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> {noformat}
> 2018-12-13 17:01:12,208 ERROR [RS_LOG_REPLAY_OPS-regionserver/...]
> executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY
> java.lang.RuntimeException: java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.hbase.wal.WALSplitter$PipelineController.checkForErrors(WALSplitter.java:846)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$OutputSink.finishWriting(WALSplitter.java:1203)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.finishWritingAndClose(WALSplitter.java:1267)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:349)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:196)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:178)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.lambda$new$0(SplitLogWorker.java:90)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NegativeArraySizeException
> at org.apache.hadoop.hbase.CellUtil.cloneFamily(CellUtil.java:113)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.filterCellByStore(WALSplitter.java:1542)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.appendBuffer(WALSplitter.java:1586)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$LogRecoveredEditsOutputSink.append(WALSplitter.java:1560)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.writeBuffer(WALSplitter.java:1085)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.doRun(WALSplitter.java:1077)
> at
> org.apache.hadoop.hbase.wal.WALSplitter$WriterThread.run(WALSplitter.java:1047)
> {noformat}
> Unfortunately I cannot share the file.
> The issue appears to be straightforward - for whatever reason the family
> length is negative. Not sure how such a cell got created, I suspect the file
> was corrupted.
> {code}
> byte[] output = new byte[cell.getFamilyLength()];
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21578) Fix wrong throttling exception for capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720828#comment-16720828
]
Yi Mei commented on HBASE-21578:
The CU throttling exception after modification is:
{code:java}
2018-12-14 09:53:28,372 DEBUG [Time-limited test]
client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=7,
started=0 ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException: write capacity unit
exceeded - wait 10sec
at
org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwThrottlingException(RpcThrottlingException.java:123)
at
org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwWriteCapacityUnitExceeded(RpcThrottlingException.java:117)
at
org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:164)
{code}
> Fix wrong throttling exception for capacity unit
>
>
> Key: HBASE-21578
> URL: https://issues.apache.org/jira/browse/HBASE-21578
> Project: HBase
> Issue Type: Bug
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-21578.master.001.patch,
> HBASE-21578.master.002.patch
>
>
> HBASE-21034 provides a new throttle type: capacity unit, but the throttling
> exception is confusing:
>
> {noformat}
> 2018-12-11 14:38:41,503 DEBUG [Time-limited test]
> client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=7,
> started=0 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException: write size limit
> exceeded - wait 10sec
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwThrottlingException(RpcThrottlingException.java:106)
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwWriteSizeExceeded(RpcThrottlingException.java:96)
> at
> org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:179){noformat}
> Need to make the exception more clearly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21578) Fix wrong throttling exception for capacity unit
[
https://issues.apache.org/jira/browse/HBASE-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Mei updated HBASE-21578:
---
Attachment: HBASE-21578.master.002.patch
> Fix wrong throttling exception for capacity unit
>
>
> Key: HBASE-21578
> URL: https://issues.apache.org/jira/browse/HBASE-21578
> Project: HBase
> Issue Type: Bug
>Reporter: Yi Mei
>Assignee: Yi Mei
>Priority: Major
> Attachments: HBASE-21578.master.001.patch,
> HBASE-21578.master.002.patch
>
>
> HBASE-21034 provides a new throttle type: capacity unit, but the throttling
> exception is confusing:
>
> {noformat}
> 2018-12-11 14:38:41,503 DEBUG [Time-limited test]
> client.RpcRetryingCallerImpl(131): Call exception, tries=6, retries=7,
> started=0 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.quotas.RpcThrottlingException: write size limit
> exceeded - wait 10sec
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwThrottlingException(RpcThrottlingException.java:106)
> at
> org.apache.hadoop.hbase.quotas.RpcThrottlingException.throwWriteSizeExceeded(RpcThrottlingException.java:96)
> at
> org.apache.hadoop.hbase.quotas.TimeBasedLimiter.checkQuota(TimeBasedLimiter.java:179){noformat}
> Need to make the exception more clearly.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21599) Fix findbugs and javadoc warnings from HBASE-21246
[
https://issues.apache.org/jira/browse/HBASE-21599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720814#comment-16720814
]
Hadoop QA commented on HBASE-21599:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HBASE-20952 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} HBASE-20952 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} HBASE-20952 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} HBASE-20952 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
50s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
8s{color} | {color:blue} hbase-server in HBASE-20952 has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} HBASE-20952 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
0s{color} | {color:red} hbase-server: The patch generated 3 new + 5 unchanged -
0 fixed = 8 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
43s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 21s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} hbase-server generated 0 new + 0 unchanged - 1 fixed
= 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} hbase-server generated 0 new + 0 unchanged - 2 fixed
= 0 total (was 2) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}121m
43s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21599 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951741/HBASE-21599.HBASE-20952.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 520539eea677 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-20952 / c738e1575f |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
[jira] [Commented] (HBASE-21359) Fix build problem against Hadoop 2.8.5
[ https://issues.apache.org/jira/browse/HBASE-21359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720804#comment-16720804 ] Hudson commented on HBASE-21359: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #510 (See [https://builds.apache.org/job/HBase-1.3-IT/510/]) HBASE-21359 Fix build problem against Hadoop 2.8.5 (apurtell: rev e7563cce32d54e41d5536b4b50e703fed42b349b) * (edit) hbase-resource-bundle/src/main/resources/supplemental-models.xml > Fix build problem against Hadoop 2.8.5 > -- > > Key: HBASE-21359 > URL: https://issues.apache.org/jira/browse/HBASE-21359 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 1.5.0, 1.3.3, 1.4.9 > > Attachments: HBASE-21359-branch-1.patch > > > 1.4.8 build fails against Hadoop 2.8.5. The fix is an easy change to > supplemental-models.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19458) Allow building HBase 1.3.x against Hadoop 2.8
[ https://issues.apache.org/jira/browse/HBASE-19458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720805#comment-16720805 ] Hudson commented on HBASE-19458: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #510 (See [https://builds.apache.org/job/HBase-1.3-IT/510/]) HBASE-19458 Allow building HBase 1.3.x against Hadoop 2.8 (Lars (apurtell: rev c4ed66a05a11e196ac5f5df3436132af28db707f) * (edit) hbase-resource-bundle/src/main/resources/supplemental-models.xml > Allow building HBase 1.3.x against Hadoop 2.8 > - > > Key: HBASE-19458 > URL: https://issues.apache.org/jira/browse/HBASE-19458 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Fix For: 1.3.3 > > Attachments: 19458.txt > > > Currently the 1.3 branch cannot be built against Hadoop 2.8.2. (that latest > stable 2.x release). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
[ https://issues.apache.org/jira/browse/HBASE-20401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720784#comment-16720784 ] Hudson commented on HBASE-20401: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20401 Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext (apurtell: rev 614b5f6e724db594b37d900d5b0fa4ada636eee5) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java > Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable > -- > > Key: HBASE-20401 > URL: https://issues.apache.org/jira/browse/HBASE-20401 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 3.0.0, 1.5.0, 2.0.0-beta-1, 1.4.4, 2.0.0 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Minor > Labels: beginner > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.4.6, 2.2.0, 2.1.1, 2.0.2 > > Attachments: HBASE-20401.branch-1.001.patch, > HBASE-20401.branch-1.002.patch, HBASE-20401.branch-1.003.patch, > HBASE-20401.master.001.patch, HBASE-20401.master.002.patch, > HBASE-20401.master.003.patch, HBASE-20401.master.004.patch, > HBASE-20401.master.005.patch, HBASE-20401.master.006.patch, > HBASE-20401.master.007.patch > > > When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls > CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for > notification (notify) from the fs.delete file thread. there might be two > situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished > when LogClearner call getResult. > # fs.delete never complete (strange but possible), then we need to wait for > a max of 60 seconds. here, 60 seconds might be too long > # getResult is waiting in the period of 500 milliseconds, but the fs.delete > has completed and setFromClear is set but yet notify(). one might want to > tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21306) website should have 1.4.x release docs
[ https://issues.apache.org/jira/browse/HBASE-21306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21306: --- Affects Version/s: (was: 1.4.7) > website should have 1.4.x release docs > -- > > Key: HBASE-21306 > URL: https://issues.apache.org/jira/browse/HBASE-21306 > Project: HBase > Issue Type: Task > Components: community, documentation, website >Reporter: Mike Drob >Assignee: Andrew Purtell >Priority: Major > > Since 1.4 is the stable line now, we should include the API docs on the > website instead of (in addition to?) the 1.2 docs we have now. > See also HBASE-21114 and HBASE-21119 for the process. > FYI: [~busbey], [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19551) hbck -boundaries doesn't work correctly
[ https://issues.apache.org/jira/browse/HBASE-19551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720748#comment-16720748 ] Hudson commented on HBASE-19551: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-19551 hbck -boundaries doesn't work correctly (Toshihiro Suzuki) (apurtell: rev 217ee60fb4538392d345e581725e79dc53ed) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/HbckTestingUtil.java > hbck -boundaries doesn't work correctly > --- > > Key: HBASE-19551 > URL: https://issues.apache.org/jira/browse/HBASE-19551 > Project: HBase > Issue Type: Bug > Components: hbck >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 1.4.1, 1.5.0, 1.3.3 > > Attachments: HBASE-19551.branch-1.patch, HBASE-19551.patch > > > Currently, in HBaseFsck#checkRegionBoundaries(), it seems like keys from > reader.getFirstKey() and reader.getLastKey() are directly compared by > ByteArrayComparator: > https://github.com/apache/hbase/blob/9d0c7c6dfbcba0907cbbc2244eac570fcc4d58a5/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java#L864-L865 > https://github.com/apache/hbase/blob/9d0c7c6dfbcba0907cbbc2244eac570fcc4d58a5/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java#L869-L870 > This is not correct because the keys consist of a key length and key itself. > We should compare rowkeys or the only key itself (by removing a key length). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21582) If call HBaseAdmin#snapshotAsync but forget call isSnapshotFinished, then SnapshotHFileCleaner will skip to run every time
[
https://issues.apache.org/jira/browse/HBASE-21582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720796#comment-16720796
]
Hudson commented on HBASE-21582:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-21582 If call HBaseAdmin#snapshotAsync but forget call (apurtell: rev
d3729417462dc70a1fe3bdf5ffb3bea136987d4c)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
> If call HBaseAdmin#snapshotAsync but forget call isSnapshotFinished, then
> SnapshotHFileCleaner will skip to run every time
> --
>
> Key: HBASE-21582
> URL: https://issues.apache.org/jira/browse/HBASE-21582
> Project: HBase
> Issue Type: Bug
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.2, 1.2.10, 1.4.10, 2.0.5
>
> Attachments: HBASE-21582.branch-1.v3.patch, HBASE-21582.v1.patch,
> HBASE-21582.v2.patch, HBASE-21582.v3.patch
>
>
> This is because we remove the SnapshotSentinel from snapshotHandlers in
> SnapshotManager#cleanupSentinels. Only when the following 3 case, the
> cleanupSentinels will be called:
> 1. SnapshotManager#isSnapshotDone;
> 2. SnapshotManager#takeSnapshot;
> 3. SnapshotManager#restoreOrCloneSnapshot
> So if no isSnapshotDone called, or no further snapshot taking, or snapshot
> restore/clone. the SnapshotSentinel will always be keep in snapshotHandlers.
> But after HBASE-21387, Only when no snapshot taking, the
> SnapshotHFileCleaner will check the unref files and clean.
> I found this bug, because in our XiaoMi branch-2, we implement the soft
> delete feature, which means if someone delete a table, then master will
> create a snapshot firstly, after that, the table deletion begain. the
> implementation is quite simple, we use the snapshotManager to create a
> snapshot.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> index 8f42e4a..6da6a64 100644
> --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> @@ -2385,12 +2385,6 @@ public class HMaster extends HRegionServer implements
> MasterServices {
>protected void run() throws IOException {
> getMaster().getMasterCoprocessorHost().preDeleteTable(tableName);
>
> +if (snapshotBeforeDelete) {
> + LOG.info("Take snaposhot for " + tableName + " before deleting");
> + snapshotManager
> +
> .takeSnapshot(SnapshotDescriptionUtils.getSnapshotNameForDeletedTable(tableName));
> +}
> +
> LOG.info(getClientIdAuditPrefix() + " delete " + tableName);
>
> // TODO: We can handle/merge duplicate request
> {code}
> In the master, I found the endless log after delete a table:
> {code}
> org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache: Not checking
> unreferenced files since snapshot is running, it will skip to clean the
> HFiles this time
> {code}
> This is because the snapshotHandlers never be cleaned after call the
> snapshotManager#takeSnapshot. I think the asynSnapshot may has the same
> problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-16091) Canary takes lot more time when there are delete markers in the table
[ https://issues.apache.org/jira/browse/HBASE-16091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720741#comment-16720741 ] Hudson commented on HBASE-16091: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-16091 Canary takes lot more time when there are delete markers in (apurtell: rev dd12d8b6f55370cbac80d4ec59b5fbf7195911de) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java > Canary takes lot more time when there are delete markers in the table > - > > Key: HBASE-16091 > URL: https://issues.apache.org/jira/browse/HBASE-16091 > Project: HBase > Issue Type: Bug > Components: canary >Affects Versions: 2.0.0 >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Major > Fix For: 1.4.0, 0.98.21, 1.3.3, 2.0.0 > > Attachments: HBASE-16091.00.patch, HBASE-16091.01.patch, > HBASE-16091.02.patch > > > We have a table which has lot of delete markers and we running Canary test on > a regular interval sometimes tests are timing out because to reading first > row would skip all these delete markers. Since purpose of Canary is to find > health of the region, i think keeping raw=true would not defeat the purpose > but provide good perf improvement. > Following are the example of one such scan where > without changing code it took 62.3 sec for onre region scan > 2016-06-23 08:49:11,670 INFO [pool-2-thread-1] tool.Canary - read from > region . column family 0 in 62338ms > whereas after setting raw=true, it reduced to 58ms > 2016-06-23 08:45:20,259 INFO [pool-2-thread-1] tests.Canary - read from > region . column family 0 in 58ms > Taking this over multiple tables , with multiple region would be a good > performance gain. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20647) Backport HBASE-20616 "TruncateTableProcedure is stuck in retry loop in TRUNCATE_TABLE_CREATE_FS_LAYOUT state" to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720772#comment-16720772 ] Hudson commented on HBASE-20647: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20647 Backport HBASE-20616 "TruncateTableProcedure is stuck in (apurtell: rev 03885b434f0cfbaaa0ed367d5d30c062eb2a48b7) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestTruncateTableProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java > Backport HBASE-20616 "TruncateTableProcedure is stuck in retry loop in > TRUNCATE_TABLE_CREATE_FS_LAYOUT state" to branch-1 > - > > Key: HBASE-20647 > URL: https://issues.apache.org/jira/browse/HBASE-20647 > Project: HBase > Issue Type: Sub-task > Components: backport >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.5 > > Attachments: HBASE-20647.branch-1.001.patch > > > Backport parent issue to branch-1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18248) Warn if monitored RPC task has been tied up beyond a configurable threshold
[ https://issues.apache.org/jira/browse/HBASE-18248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720755#comment-16720755 ] Hudson commented on HBASE-18248: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-18248 Warn if monitored RPC task has been tied up beyond a (apurtell: rev ac241dcde6850df351d6dbe8b00a319488fb465a) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTaskImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredTask.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/TaskMonitor.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/monitoring/TestTaskMonitor.java > Warn if monitored RPC task has been tied up beyond a configurable threshold > --- > > Key: HBASE-18248 > URL: https://issues.apache.org/jira/browse/HBASE-18248 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.4.0, 1.3.3, 2.0.0 > > Attachments: HBASE-18248-branch-1.patch, HBASE-18248-branch-1.patch, > HBASE-18248.patch, HBASE-18248.patch > > > Warn if monitored task has been tied up beyond a configurable threshold. We > especially want to do this for RPC tasks. Use a separate threshold for > warning about stuck RPC tasks versus other types of tasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19905) ReplicationSyncUp tool will not exit if a peer replication is disabled
[ https://issues.apache.org/jira/browse/HBASE-19905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720762#comment-16720762 ] Hudson commented on HBASE-19905: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-19905 ReplicationSyncUp tool will not exit if a peer replication (apurtell: rev 8a4b2b54cdadfbe2ee730733f10bd3ec57b05340) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java > ReplicationSyncUp tool will not exit if a peer replication is disabled > -- > > Key: HBASE-19905 > URL: https://issues.apache.org/jira/browse/HBASE-19905 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.1 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Major > Fix For: 1.5.0, 2.0.0-beta-2, 1.3.3, 1.4.2, 2.0.0 > > Attachments: HBASE-19905.patch > > > In our test cluster we had two peer clusters, in which one peer cluster > replication was disabled. Now when used ReplicationSyncUp tool to replicate > the data to peer cluster, the tool replicated the data to the enabled peer > cluster but it was keep on retrying to replicate the data to disabled peer > cluster and hence it was not getting terminated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21546) ConnectException in TestThriftHttpServer
[ https://issues.apache.org/jira/browse/HBASE-21546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720795#comment-16720795 ] Hudson commented on HBASE-21546: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-21546 ConnectException in TestThriftHttpServer (apurtell: rev e063aa8db09d8e8224a923d01aa027742f253aff) * (edit) hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java > ConnectException in TestThriftHttpServer > > > Key: HBASE-21546 > URL: https://issues.apache.org/jira/browse/HBASE-21546 > Project: HBase > Issue Type: Bug > Components: test, Thrift >Affects Versions: 1.5.0, 1.4.9 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.9 > > Attachments: HBASE-21546.branch-1.01.patch > > > TestThriftHttpServer is the first on the flaky list for branch-1 and > branch-1.4 with approximately 60% failure rate. > Thrift server is not yet accepting request at the time the test starts. > java.net.ConnectException: Connection refused (Connection refused) at > org.apache.hadoop.hbase.thrift.TestThriftHttpServer.checkHttpMethods(TestThriftHttpServer.java:275) > at > org.apache.hadoop.hbase.thrift.TestThriftHttpServer.testThriftServerHttpOptionsForbiddenWhenOptionsDisabled(TestThriftHttpServer.java:176) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20857) JMX - add Balancer status = enabled / disabled
[ https://issues.apache.org/jira/browse/HBASE-20857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720793#comment-16720793 ] Hudson commented on HBASE-20857: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20857 balancer status tag in jmx metrics (apurtell: rev 10c4f59eca5c8b6e037e5551388e19928cd20fce) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockNoopMasterServices.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/balancer/MetricsBalancerSourceImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/MetricsStochasticBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/MetricsBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBalancerStatusTagInJMXMetrics.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/balancer/MetricsBalancerSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java > JMX - add Balancer status = enabled / disabled > -- > > Key: HBASE-20857 > URL: https://issues.apache.org/jira/browse/HBASE-20857 > Project: HBase > Issue Type: Improvement > Components: API, master, metrics, REST, tooling, Usability >Reporter: Hari Sekhon >Assignee: Kiran Kumar Maturi >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1 > > Attachments: HBASE-20857.branch-1.4.001.patch, > HBASE-20857.branch-1.4.002.patch > > > Add HBase Balancer enabled/disabled status to JMX API on HMaster. > Right now the HMaster will give a warning near the top of HMaster UI if > balancer is disabled, but scraping this is for monitoring integration is not > nice, it should be available in JMX API as there is already a > Master,sub=Balancer bean with metrics for the balancer ops etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18083) Make large/small file clean thread number configurable in HFileCleaner
[ https://issues.apache.org/jira/browse/HBASE-18083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720783#comment-16720783 ] Hudson commented on HBASE-18083: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20559 Backport HBASE-18083 (Make large/small file clean thread (apurtell: rev 2434162594f22df0bba94bc40186bd5628501c8e) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java > Make large/small file clean thread number configurable in HFileCleaner > -- > > Key: HBASE-18083 > URL: https://issues.apache.org/jira/browse/HBASE-18083 > Project: HBase > Issue Type: Improvement >Reporter: Yu Li >Assignee: Yu Li >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-18083.patch, HBASE-18083.v2.patch, > HBASE-18083.v3.patch > > > Currently we have only one thread for both large and small file cleaning, but > when write pressure is huge we might need more cleaner threads, so we need to > make the thread number configurable. > We observed more than 1.8PB data in archive directory online due to business > access rate change, and this proposal is one of the necessary changes > required. > Default value of the configurations would still be left to 1 to keep low > pressure to NN for normal case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20723) Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
[
https://issues.apache.org/jira/browse/HBASE-20723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720787#comment-16720787
]
Hudson commented on HBASE-20723:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20723 Custom hbase.wal.dir results in data loss because we write
(apurtell: rev 2702f23a4addaa459d6f221c0008be0c556f672d)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DefaultWALProvider.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFactory.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
> Custom hbase.wal.dir results in data loss because we write recovered edits
> into a different place than where the recovering region server looks for them
>
>
> Key: HBASE-20723
> URL: https://issues.apache.org/jira/browse/HBASE-20723
> Project: HBase
> Issue Type: Bug
> Components: Recovery, wal
>Affects Versions: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 2.0.0
>Reporter: Rohan Pednekar
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.1.0, 1.5.0, 1.3.3, 1.4.6
>
> Attachments: 20723.branch-1.txt, 20723.branch-2.txt, 20723.v1.txt,
> 20723.v10.txt, 20723.v2.txt, 20723.v3.txt, 20723.v4.txt, 20723.v5.txt,
> 20723.v5.txt, 20723.v6.txt, 20723.v7.txt, 20723.v8.txt, 20723.v9.txt, logs.zip
>
>
> Description:
> When custom hbase.wal.dir is configured the recovery system uses it in place
> of the HBase root dir and thus constructs an incorrect path for recovered
> edits when splitting WALs. This causes the recovery code in Region Servers to
> believe there are no recovered edits to replay, which causes a loss of writes
> that had not flushed prior to loss of a server.
>
> Reproduction:
> This is an Azure HDInsight HBase cluster with HDP 2.6. and HBase
> 1.1.2.2.6.3.2-14
> By default the underlying data is going to wasb://x@y/hbase
> I tried to move WAL folders to HDFS, which is the SSD mounted on each VM at
> /mnt.
> hbase.wal.dir= hdfs://mycluster/walontest
> hbase.wal.dir.perms=700
> hbase.rootdir.perms=700
> hbase.rootdir=
> wasb://XYZ[@hbaseperf.core.net|mailto:duohbase5ds...@duohbaseperf.blob.core.windows.net]/hbase
> Procedure to reproduce this issue:
> 1. create a table in hbase shell
> 2. insert a row in hbase shell
> 3. reboot the VM which hosts that region
> 4. scan the table in hbase shell and it is empty
> Looking at the region server logs:
> {code:java}
> 2018-06-12 22:08:40,455 INFO [RS_LOG_REPLAY_OPS-wn2-duohba:16020-0-Writer-1]
> wal.WALSplitter: This region's directory doesn't exist:
> hdfs://mycluster/walontest/data/default/tb1/b7fd7db5694eb71190955292b3ff7648.
> It is very likely that it was already split so it's safe to discard those
> edits.
> {code}
> The log split/replay ignored actual WAL due to WALSplitter is looking for the
> region directory in the hbase.wal.dir we specified rather than the
> hbase.rootdir.
> Looking at the source code,
>
> [https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java]
> it uses the rootDir, which is walDir, as the tableDir root path.
> So if we use HBASE-17437, waldir and hbase rootdir are in different path or
> even in different filesystem, then the #5 uses walDir as tableDir is
> apparently wrong.
> CC: [~zyork], [~yuzhih...@gmail.com] Attached the logs for quick review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18451) PeriodicMemstoreFlusher should inspect the queue before adding a delayed flush request
[
https://issues.apache.org/jira/browse/HBASE-18451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720792#comment-16720792
]
Hudson commented on HBASE-18451:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-18451 PeriodicMemstoreFlusher should inspect the queue before (apurtell:
rev 785e21fe545da33811a50e0718d7cfeb7dc74df7)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHeapMemoryManager.java
> PeriodicMemstoreFlusher should inspect the queue before adding a delayed
> flush request
> --
>
> Key: HBASE-18451
> URL: https://issues.apache.org/jira/browse/HBASE-18451
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0-alpha-1
>Reporter: Jean-Marc Spaggiari
>Assignee: Xu Cang
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-18451.branch-1.001.patch,
> HBASE-18451.branch-1.002.patch, HBASE-18451.branch-1.002.patch,
> HBASE-18451.master.002.patch, HBASE-18451.master.003.patch,
> HBASE-18451.master.004.patch, HBASE-18451.master.004.patch,
> HBASE-18451.master.patch
>
>
> If you run a big job every 4 hours, impacting many tables (they have 150
> regions per server), ad the end all the regions might have some data to be
> flushed, and we want, after one hour, trigger a periodic flush. That's
> totally fine.
> Now, to avoid a flush storm, when we detect a region to be flushed, we add a
> "randomDelay" to the delayed flush, that way we spread them away.
> RANGE_OF_DELAY is 5 minutes. So we spread the flush over the next 5 minutes,
> which is very good.
> However, because we don't check if there is already a request in the queue,
> 10 seconds after, we create a new request, with a new randomDelay.
> If you generate a randomDelay every 10 seconds, at some point, you will end
> up having a small one, and the flush will be triggered almost immediatly.
> As a result, instead of spreading all the flush within the next 5 minutes,
> you end-up getting them all way more quickly. Like within the first minute.
> Which not only feed the queue to to many flush requests, but also defeats the
> purpose of the randomDelay.
> {code}
> @Override
> protected void chore() {
> final StringBuffer whyFlush = new StringBuffer();
> for (Region r : this.server.onlineRegions.values()) {
> if (r == null) continue;
> if (((HRegion)r).shouldFlush(whyFlush)) {
> FlushRequester requester = server.getFlushRequester();
> if (requester != null) {
> long randomDelay = RandomUtils.nextInt(RANGE_OF_DELAY) +
> MIN_DELAY_TIME;
> LOG.info(getName() + " requesting flush of " +
> r.getRegionInfo().getRegionNameAsString() + " because " +
> whyFlush.toString() +
> " after random delay " + randomDelay + "ms");
> //Throttle the flushes by putting a delay. If we don't throttle,
> and there
> //is a balanced write-load on the regions in a table, we might
> end up
> //overwhelming the filesystem with too many flushes at once.
> requester.requestDelayedFlush(r, randomDelay, false);
> }
> }
> }
> }
> {code}
> {code}
> 2017-07-24 18:44:33,338 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after random delay 270785ms
> 2017-07-24 18:44:43,328 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after random delay 200143ms
> 2017-07-24 18:44:53,954 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after random delay 191082ms
> 2017-07-24 18:45:03,528 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
>
[jira] [Commented] (HBASE-21275) Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http server (branch 1 only)
[ https://issues.apache.org/jira/browse/HBASE-21275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720794#comment-16720794 ] Hudson commented on HBASE-21275: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-21275 - Disable TRACE HTTP method for thrift http server (branch 1 (apurtell: rev 82f187efba6e476cd1b88bc6ae8b238e4c670288) * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java * (edit) hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java > Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http > server (branch 1 only) > > > Key: HBASE-21275 > URL: https://issues.apache.org/jira/browse/HBASE-21275 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 1.5.0, 1.3.3, 1.4.9 > > Attachments: HBASE-21275-branch-1.001.patch, > HBASE-21275-branch-1.2.001.patch, HBASE-21275-branch-1.2.002.patch, > HBASE-21275-branch-1.2.003.patch, HBASE-21275-branch-1.2.003.patch, > HBASE-21275-branch-1.4.001.patch > > > There's been a reasonable number of users running thrift http server on hbase > 1.x suffering with security audit tests pointing thrift server allows TRACE > requests. > After doing some search, I can see HBASE-20406 added restrictions for > TRACE/OPTIONS method when Thrift is running over http, but it relies on many > other commits applied to thrift http server. This patch was later reverted > from master. Then again later, HBASE-20004 had made TRACE/OPTIONS > configurable via "*hbase.thrift.http.allow.options.method*" property, with > both methods being disabled by default. This also seems to rely on many > changes applied to thrift http server, and a branch 1 compatible patch does > not seem feasible. > A solution for branch 1 is pretty simple though, am proposing a patch that > simply uses *WebAppContext*, instead of *Context*, as the context for the > *HttpServer* instance. *WebAppContext* will already restrict TRACE methods by > default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21212) Wrong flush time when update flush metric
[ https://issues.apache.org/jira/browse/HBASE-21212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720791#comment-16720791 ] Hudson commented on HBASE-21212: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-21212 Wrong flush time when update flush metric (apurtell: rev a4baeebdfc16878011633aa548b90dd53afddc37) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Wrong flush time when update flush metric > - > > Key: HBASE-21212 > URL: https://issues.apache.org/jira/browse/HBASE-21212 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 2.0.2 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Minor > Fix For: 3.0.0, 1.3.3, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-21212.branch-2.0.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20695) Implement table level RegionServer replication metrics
[ https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720786#comment-16720786 ] Hudson commented on HBASE-20695: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20858 Port HBASE-20695 (Implement table level RegionServer (apurtell: rev 9e39a2009648ba21d55676ea90b328941bb9b7db) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java > Implement table level RegionServer replication metrics > --- > > Key: HBASE-20695 > URL: https://issues.apache.org/jira/browse/HBASE-20695 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Fix For: 2.1.0 > > Attachments: HBASE-20695.master.001.patch, > HBASE-20695.master.002.patch, HBASE-20695.master.003.patch, > HBASE-20695.master.004.patch, HBASE-20695.master.005.patch, > HBASE-20695.master.006.patch, HBASE-20695.master.007.patch, > HBASE-20695.master.008.patch, HBASE-20695.master.009.patch, > HBASE-20695.master.010.patch > > > Region server metrics now are mainly global metrics. It would be nice to have > table level metrics such as table level source.AgeOfLastShippedOp to indicate > operators which table's replication is lagging behind. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20554) "WALs outstanding" message from CleanerChore is noisy
[
https://issues.apache.org/jira/browse/HBASE-20554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720771#comment-16720771
]
Hudson commented on HBASE-20554:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20554 "WALs outstanding" message from CleanerChore is noisy (apurtell:
rev 511a4c49ceb223b289462121c788994fbfa2713b)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
> "WALs outstanding" message from CleanerChore is noisy
> -
>
> Key: HBASE-20554
> URL: https://issues.apache.org/jira/browse/HBASE-20554
> Project: HBase
> Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20554.patch
>
>
> WARN level "WALs outstanding" from CleanerChore should be DEBUG and are not
> always correct.
> I left a cluster configured for ITBLL (retaining all WALs for post hoc
> analysis) and in the morning found the master log full of "WALs outstanding"
> warnings from CleanerChore.
> Should this really be a warning?
> {quote}
> 2018-05-09 16:42:03,893 WARN
> [node-1.cluster,16000,1525851521469_ChoreService_2] cleaner.CleanerChore:
> WALs outstanding under hdfs://node-1.cluster/hbase/oldWALs
> {quote}
> If someone has configured really long WAL retention then having WALs in
> oldWALs will be normal.
> Also, it seems the warning is sometimes incorrect.
> {quote}
> 2018-05-09 16:42:24,751 WARN
> [node-1.cluster,16000,1525851521469_ChoreService_1] cleaner.CleanerChore:
> WALs outstanding under hdfs://node-1.cluster/hbase/archive
> {quote}
> There are no WALs under archive/.
> Even at DEBUG level, if it is not correct, then it can lead an operator to be
> concerned about nothing, so better to just remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed
[
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720779#comment-16720779
]
Hudson commented on HBASE-17215:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20557 Backport HBASE-17215 to branch-1 (apurtell: rev
30b1dc00b47a0289a721cd0f932d9cdd8422534e)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
> Separate small/large file delete threads in HFileCleaner to accelerate
> archived hfile cleanup speed
> ---
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
> Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-17215.patch, HBASE-17215.v2.patch,
> HBASE-17215.v3.patch
>
>
> When using PCIe-SSD the flush speed will be really quick, and although we
> have per CF flush, we still have the
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other
> mechanism to avoid data kept in memory for too long to flush small hfiles. In
> our online environment we found the single thread cleaner kept cleaning
> earlier flushed small files while large files got no chance, which caused
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the
> workload of namenode, so here we propose to separate large/small hfile
> cleaner threads just like we do for compaction, and it turned out to work
> well in our cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20858) port HBASE-20695 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720785#comment-16720785 ] Hudson commented on HBASE-20858: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20858 Port HBASE-20695 (Implement table level RegionServer (apurtell: rev 9e39a2009648ba21d55676ea90b328941bb9b7db) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java > port HBASE-20695 to branch-1 > > > Key: HBASE-20858 > URL: https://issues.apache.org/jira/browse/HBASE-20858 > Project: HBase > Issue Type: Improvement >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.3.3, 1.4.10 > > Attachments: HBASE-20858.branch-1.001.patch, > HBASE-20858.branch-1.002.patch > > > port HBASE-20695 to branch-1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18549) Unclaimed replication queues can go undetected
[ https://issues.apache.org/jira/browse/HBASE-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720789#comment-16720789 ] Hudson commented on HBASE-18549: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-18549 Add metrics for failed replication queue recovery (apurtell: rev 1ecfca3a5c7587a3a69f718ffbeec0d36cbcf4d1) * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java > Unclaimed replication queues can go undetected > -- > > Key: HBASE-18549 > URL: https://issues.apache.org/jira/browse/HBASE-18549 > Project: HBase > Issue Type: Bug > Components: Replication >Reporter: Ashu Pachauri >Assignee: Xu Cang >Priority: Critical > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1 > > Attachments: HBASE-18549-.master.001.patch, > HBASE-18549-.master.002.patch, HBASE-18549-.master.003.patch, > HBASE-18549-.master.004.patch, HBASE-18549.branch-1.001.patch, > HBASE-18549.branch-1.001.patch > > > We have come across this situation multiple times where a zookeeper issues > can cause NodeFailoverWorker to fail picking up replication queue for a dead > region server silently. One example is when the znode size for a particular > queue exceed jute.maxBuffer value. > There can be other situations that may lead to this and just go undetected. > We need to have a metric for number of unclaimed replication queues. This > will help in mitigating the problem through alerting on the metric and > identifying underlying issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20469) Directory used for sidelining old recovered edits files should be made configurable
[
https://issues.apache.org/jira/browse/HBASE-20469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720788#comment-16720788
]
Hudson commented on HBASE-20469:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20469 Directory used for sidelining old recovered edits files (apurtell:
rev 1d362f6de3e5c21c9f124656a928a555e4853a2b)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
> Directory used for sidelining old recovered edits files should be made
> configurable
> ---
>
> Key: HBASE-20469
> URL: https://issues.apache.org/jira/browse/HBASE-20469
> Project: HBase
> Issue Type: Improvement
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 2.1.1, 1.4.7
>
> Attachments: HBASE-20469.master.001.patch
>
>
> Currently the directory used for sidelining of old recovered edit files is
> hardcoded to be "/tmp"
> {code:java}
> Path tmp = new Path("/tmp");
> {code}
> [See L484
> WALSplittter.java|https://github.com/apache/hbase/blob/273d252838e96c4b4af2401743d84e482c4ec565/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java#L484]
> Instead, we can use some configurable directory in the following manner:
>
> {code:java}
> String tmpDirName = conf.get(HConstants.TEMPORARY_FS_DIRECTORY_KEY,
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY);
> .
> .
> Path tmp = new Path(tmpDirName);
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20141) Fix TooManyFiles exception when RefreshingChannels in FileIOEngine
[ https://issues.apache.org/jira/browse/HBASE-20141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720765#comment-16720765 ] Hudson commented on HBASE-20141: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20141 Fix TooManyFiles exception when RefreshingChannels (apurtell: rev ecfa9a8a7997a188c1f3ec2d7776245664dc7dbe) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestFileIOEngine.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java > Fix TooManyFiles exception when RefreshingChannels in FileIOEngine > -- > > Key: HBASE-20141 > URL: https://issues.apache.org/jira/browse/HBASE-20141 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 1.4.0, 2.0.0-beta-1, 1.4.2 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.3, 2.0.0 > > Attachments: HBASE-20141.master.001.patch, > HBASE-20141.master.002.patch, HBASE-20141.master.003.patch, > HBASE-20141.master.004.patch > > > HBASE-19435 implements a fix for reopening file channels when they are > unnexpected closed > to avoid disabling the BucketCache. However, it was missed that the the > channels might not > actually be completely closed (the write or read channel might still be open > (see > https://docs.oracle.com/javase/7/docs/api/java/nio/channels/ClosedChannelException.html) > This commit closes any open channels before creating a new channel. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do
[
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720774#comment-16720774
]
Hudson commented on HBASE-20770:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20770 WAL cleaner logs way too much; gets clogged when lots of (apurtell:
rev 7bfec2a8989083c70c7fcc1b8c7a1b7af4d65eb8)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslClientHandler.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
> Issue Type: Bug
> Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 1.5.0, 1.3.3, 1.4.6, 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster,
> Master log is in a continuous spew of logging output fililng disks. It is
> stuck making no progress but hard to tell because it is logging minutiae
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
> exit...
> 2018-06-21 01:19:12,823 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
> exit...
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta
> 2018-06-21 01:19:12,927 DEBUG
>
[jira] [Commented] (HBASE-20616) TruncateTableProcedure is stuck in retry loop in TRUNCATE_TABLE_CREATE_FS_LAYOUT state
[
https://issues.apache.org/jira/browse/HBASE-20616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720773#comment-16720773
]
Hudson commented on HBASE-20616:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20647 Backport HBASE-20616 "TruncateTableProcedure is stuck in (apurtell:
rev 03885b434f0cfbaaa0ed367d5d30c062eb2a48b7)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestTruncateTableProcedure.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java
> TruncateTableProcedure is stuck in retry loop in
> TRUNCATE_TABLE_CREATE_FS_LAYOUT state
> --
>
> Key: HBASE-20616
> URL: https://issues.apache.org/jira/browse/HBASE-20616
> Project: HBase
> Issue Type: Bug
> Components: amv2
> Environment: HDP-2.5.3
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: 20616.master.004.patch, HBASE-20616.master.001.patch,
> HBASE-20616.master.002.patch, HBASE-20616.master.003.patch,
> HBASE-20616.master.004.patch
>
>
> At first, TruncateTableProcedure failed to write some files to HDFS in
> TRUNCATE_TABLE_CREATE_FS_LAYOUT state for some reason.
> {code:java}
> 2018-05-15 08:00:25,346 WARN [ProcedureExecutorThread-8]
> procedure.TruncateTableProcedure: Retriable error trying to truncate
> table=: state=TRUNCATE_TABLE_CREATE_FS_LAYOUT
> java.io.IOException: java.util.concurrent.ExecutionException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /apps/hbase/data/.tmp/data.regioninfo could
> only be replicated to 0 nodes instead of minReplication (=1). There are number of DNs> datanode(s) running and no node(s) are excluded in this
> operation.
> ...
> {code}
> But at this time, seemed like writing some files to HDFS was successful.
> And then, TruncateTableProcedure was stuck in retry loop in
> TRUNCATE_TABLE_CREATE_FS_LAYOUT state. At this point, the following log
> messages were shown repeatedly in the master log:
> {code:java}
> 2018-05-15 08:00:25,463 WARN [ProcedureExecutorThread-8]
> procedure.TruncateTableProcedure: Retriable error trying to truncate
> table=: state=TRUNCATE_TABLE_CREATE_FS_LAYOUT
> java.io.IOException: java.util.concurrent.ExecutionException:
> java.io.IOException: The specified region already exists on disk:
> hdfs:///apps/hbase/data/.tmp/data///
> ...
> {code}
> It seems like this is because TruncateTableProcedure tried to write the files
> that were written successfully in the first try.
> I think we need to delete all the files and directories that are written
> successfully in the previous try before retrying the
> TRUNCATE_TABLE_CREATE_FS_LAYOUT state.
> Actually, this issue was observed in HDP-2.5.3, but I think the upstream has
> the same issue. Also, it looks to me that CreateTableProcedure has a similar
> issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18309) Support multi threads in CleanerChore
[ https://issues.apache.org/jira/browse/HBASE-18309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720768#comment-16720768 ] Hudson commented on HBASE-18309: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20352 [Chore] Backport HBASE-18309 (Support multi threads in (apurtell: rev 193d1dcb72c22252fc86ee8433c765c42349d3cc) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileLinkCleaner.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FileStatusFilter.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java > Support multi threads in CleanerChore > - > > Key: HBASE-18309 > URL: https://issues.apache.org/jira/browse/HBASE-18309 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: Reid Chan >Priority: Major > Fix For: 2.0.0-beta-1, 2.0.0 > > Attachments: HBASE-18309.addendum.patch, > HBASE-18309.branch-1.001.patch, HBASE-18309.branch-1.002.patch, > HBASE-18309.branch-1.003.patch, HBASE-18309.branch-1.004.patch, > HBASE-18309.branch-1.005.patch, HBASE-18309.branch-1.006.patch, > HBASE-18309.master.001.patch, HBASE-18309.master.002.patch, > HBASE-18309.master.004.patch, HBASE-18309.master.005.patch, > HBASE-18309.master.006.patch, HBASE-18309.master.007.patch, > HBASE-18309.master.008.patch, HBASE-18309.master.009.patch, > HBASE-18309.master.010.patch, HBASE-18309.master.011.patch, > HBASE-18309.master.012.patch, space_consumption_in_archive.png > > > There is only one thread in LogCleaner to clean oldWALs and in our big > cluster we find this is not enough. The number of files under oldWALs reach > the max-directory-items limit of HDFS and cause region server crash, so we > use multi threads for LogCleaner and the crash not happened any more. > What's more, currently there's only one thread iterating the archive > directory, and we could use multiple threads cleaning sub directories in > parallel to speed it up. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19358) Improve the stability of splitting log when do fail over
[ https://issues.apache.org/jira/browse/HBASE-19358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720759#comment-16720759 ] Hudson commented on HBASE-19358: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-19358 Improve the stability of splitting log when do fail over (apurtell: rev e4f46f53b737d0a5ea0c4c079b6182a5d918e91d) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplayBoundedLogWriterCreation.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitBoundedLogWriterCreation.java > Improve the stability of splitting log when do fail over > > > Key: HBASE-19358 > URL: https://issues.apache.org/jira/browse/HBASE-19358 > Project: HBase > Issue Type: Improvement > Components: MTTR >Affects Versions: 0.98.24 >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Fix For: 1.4.1, 1.5.0, 2.0.0-beta-1, 2.0.0 > > Attachments: HBASE-18619-branch-2-v2.patch, > HBASE-19358-branch-1-v2.patch, HBASE-19358-branch-1-v3.patch, > HBASE-19358-branch-1.patch, HBASE-19358-branch-2-v3.patch, > HBASE-19358-v1.patch, HBASE-19358-v4.patch, HBASE-19358-v5.patch, > HBASE-19358-v6.patch, HBASE-19358-v7.patch, HBASE-19358-v8.patch, > HBASE-19358.patch, split-1-log.png, split-logic-new.jpg, split-logic-old.jpg, > split-table.png, split_test_result.png > > > The way we splitting log now is like the following figure: > !https://issues.apache.org/jira/secure/attachment/12905027/split-logic-old.jpg! > The problem is the OutputSink will write the recovered edits during splitting > log, which means it will create one WriterAndPath for each region and retain > it until the end. If the cluster is small and the number of regions per rs is > large, it will create too many HDFS streams at the same time. Then it is > prone to failure since each datanode need to handle too many streams. > Thus I come up with a new way to split log. > !https://issues.apache.org/jira/secure/attachment/12905028/split-logic-new.jpg! > We try to cache all the recovered edits, but if it exceeds the MaxHeapUsage, > we will pick the largest EntryBuffer and write it to a file (close the writer > after finish). Then after we read all entries into memory, we will start a > writeAndCloseThreadPool, it starts a certain number of threads to write all > buffers to files. Thus it will not create HDFS streams more than > *_hbase.regionserver.hlog.splitlog.writer.threads_* we set. > The biggest benefit is we can control the number of streams we create during > splitting log, > it will not exceeds *_hbase.regionserver.wal.max.splitters * > hbase.regionserver.hlog.splitlog.writer.threads_*, but before it is > *_hbase.regionserver.wal.max.splitters * the number of region the hlog > contains_*. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19163) "Maximum lock count exceeded" from region server's batch processing
[
https://issues.apache.org/jira/browse/HBASE-19163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720760#comment-16720760
]
Hudson commented on HBASE-19163:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-19163 "Maximum lock count exceeded" from region server's batch (apurtell:
rev b3f911c1c6381c801d88b1d0fe8f4620860aada0)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> "Maximum lock count exceeded" from region server's batch processing
> ---
>
> Key: HBASE-19163
> URL: https://issues.apache.org/jira/browse/HBASE-19163
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 3.0.0, 2.0.0-alpha-3, 1.2.7
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Major
> Fix For: 1.4.1, 1.5.0, 1.3.3, 2.0.0
>
> Attachments: HBASE-19163-branch-1-v001.patch,
> HBASE-19163-branch-1-v001.patch, HBASE-19163-master-v001.patch,
> HBASE-19163.master.001.patch, HBASE-19163.master.002.patch,
> HBASE-19163.master.004.patch, HBASE-19163.master.005.patch,
> HBASE-19163.master.006.patch, HBASE-19163.master.007.patch,
> HBASE-19163.master.008.patch, HBASE-19163.master.009.patch,
> HBASE-19163.master.009.patch, HBASE-19163.master.010.patch, unittest-case.diff
>
>
> In one of use cases, we found the following exception and replication is
> stuck.
> {code}
> 2017-10-25 19:41:17,199 WARN [hconnection-0x28db294f-shared--pool4-t936]
> client.AsyncProcess: #3, table=foo, attempt=5/5 failed=262836ops, last
> exception: java.io.IOException: java.io.IOException: Maximum lock count
> exceeded
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2215)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
> Caused by: java.lang.Error: Maximum lock count exceeded
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:528)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:488)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1327)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getRowLock(HRegion.java:5163)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3018)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2877)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2819)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:753)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:715)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2148)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
> ... 3 more
> {code}
> While we are still examining the data pattern, it is sure that there are too
> many mutations in the batch against the same row, this exceeds the maximum
> 64k shared lock count and it throws an error and failed the whole batch.
> There are two approaches to solve this issue.
> 1). Let's say there are mutations against the same row in the batch, we just
> need to acquire the lock once for the same row vs to acquire the lock for
> each mutation.
> 2). We catch the error and start to process whatever it gets and loop back.
> With HBASE-17924, approach 1 seems easy to implement now.
> Create the jira and will post update/patch when investigation moving forward.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19364) Truncate_preserve fails with table when replica region > 1
[ https://issues.apache.org/jira/browse/HBASE-19364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720758#comment-16720758 ] Hudson commented on HBASE-19364: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-19364 Truncate_preserve fails with table when replica region > 1 (apurtell: rev cd1726f53181e655246bda16b63a7f1fb990955a) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/TruncateTableProcedure.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ProcedureSyncWait.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestTruncateTableProcedure.java > Truncate_preserve fails with table when replica region > 1 > -- > > Key: HBASE-19364 > URL: https://issues.apache.org/jira/browse/HBASE-19364 > Project: HBase > Issue Type: Bug > Components: master >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.3 > > Attachments: HBASE-19364-branch-1-v2.patch, > HBASE-19364-branch-1-v3.patch, HBASE-19364-branch-1.patch, > HBASE-19364-branch-1.patch > > > Root cause is same as HBASE-17319, here we need to exclude secondary regions > while reading meta. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20559) Backport HBASE-18083 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720782#comment-16720782 ] Hudson commented on HBASE-20559: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20559 Backport HBASE-18083 (Make large/small file clean thread (apurtell: rev 2434162594f22df0bba94bc40186bd5628501c8e) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java > Backport HBASE-18083 to branch-1 > > > Key: HBASE-20559 > URL: https://issues.apache.org/jira/browse/HBASE-20559 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.6 > > Attachments: HBASE-20559.branch-1.001.patch, > HBASE-20559.branch-1.002.patch > > > As part of HBASE-20555, HBASE-18083 is the fourth/last patch that is needed > for backporting. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21126) Add ability for HBase Canary to ignore a configurable number of ZooKeeper down nodes
[
https://issues.apache.org/jira/browse/HBASE-21126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720790#comment-16720790
]
Hudson commented on HBASE-21126:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-21126 Configurable number of allowed failures for ZooKeeper Canary
(apurtell: rev 09069df2f8a9cb19ce368a54770d333f0e36fe5d)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java
> Add ability for HBase Canary to ignore a configurable number of ZooKeeper
> down nodes
>
>
> Key: HBASE-21126
> URL: https://issues.apache.org/jira/browse/HBASE-21126
> Project: HBase
> Issue Type: Improvement
> Components: canary, Zookeeper
>Affects Versions: 1.0.0, 3.0.0, 2.0.0
>Reporter: David Manning
>Assignee: David Manning
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-21126.branch-1.001.patch,
> HBASE-21126.master.001.patch, HBASE-21126.master.002.patch,
> HBASE-21126.master.003.patch, zookeeperCanaryLocalTestValidation.txt
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When running org.apache.hadoop.hbase.tool.Canary with args -zookeeper
> -treatFailureAsError, the Canary will try to get a znode from each ZooKeeper
> server in the ensemble. If any server is unavailable or unresponsive, the
> canary will exit with a failure code.
> If we use the Canary to gauge server health, and alert accordingly, this can
> be too strict. For example, in a 5-node ZooKeeper cluster, having one node
> down is safe and expected in rolling upgrades/patches.
> This is a request to allow the Canary to take another parameter
> {code:java}
> -permittedZookeeperFailures {code}
> If N=1, in the 5-node ZooKeeper ensemble example, then the Canary will still
> pass if 4 ZooKeeper nodes are reachable, but fail if 3 or fewer are reachable.
> (This is my first Jira posting... sorry if I messed anything up.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20047) AuthenticationTokenIdentifier should provide a toString
[
https://issues.apache.org/jira/browse/HBASE-20047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720766#comment-16720766
]
Hudson commented on HBASE-20047:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20047 AuthenticationTokenIdentifier should provide a toString (apurtell:
rev 35e94c98785095f83c7dd0c7b97b04678fb9c875)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
> AuthenticationTokenIdentifier should provide a toString
> ---
>
> Key: HBASE-20047
> URL: https://issues.apache.org/jira/browse/HBASE-20047
> Project: HBase
> Issue Type: Improvement
> Components: Usability
>Reporter: Sean Busbey
>Assignee: maoling
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.10
>
> Attachments: HBASE-20047.master.v0.patch, HBASE-20047.master.v1.patch
>
>
> It'd be easier to debug things like MapReduce and Spark jobs if our
> AuthenticationTokenIdentifier provided a toString method.
> For comparison, here's an example of a MapReduce job that has both an HDFS
> delegation token and our delegation token:
> {code:java}
> 18/02/21 20:40:06 INFO mapreduce.JobSubmitter: Kind: HBASE_AUTH_TOKEN,
> Service: 92a63bd8-9e00-4c04-ab61-da8e606068e1, Ident:
> (org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier@17)
> 18/02/21 20:40:06 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN,
> Service: 172.31.118.118:8020, Ident: (token for some_user:
> HDFS_DELEGATION_TOKEN owner=some_u...@example.com, renewer=yarn, realUser=,
> issueDate=1519274405003, maxDate=1519879205003, sequenceNumber=23,
> masterKeyId=9)
> {code}
> Stuff in TokenIdentifier is supposed to be public, so we should be fine to
> dump everything, similar to Hadoop's AbstractDelegationTokenIdentifier.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20558) Backport HBASE-17854 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720780#comment-16720780 ] Hudson commented on HBASE-20558: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20558 Port HBASE-17854 (Use StealJobQueue in HFileCleaner after (apurtell: rev 976f07e87cbeb35b80a7ca8eea46f973e932bf83) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/StealJobQueue.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestStealJobQueue.java > Backport HBASE-17854 to branch-1 > > > Key: HBASE-20558 > URL: https://issues.apache.org/jira/browse/HBASE-20558 > Project: HBase > Issue Type: Sub-task > Components: HFile >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.6 > > Attachments: HBASE-20558.branch-1.001.patch, report.html > > > As part of HBASE-20555, HBASE-17854 is the third patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20404) Ugly cleanerchore complaint that dir is not empty
[
https://issues.apache.org/jira/browse/HBASE-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720769#comment-16720769
]
Hudson commented on HBASE-20404:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20404 Fixes to CleanChore correctness and operability. (apurtell: rev
be4915eb4f6604edd570b2bb937800e663f76043)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java
> Ugly cleanerchore complaint that dir is not empty
> -
>
> Key: HBASE-20404
> URL: https://issues.apache.org/jira/browse/HBASE-20404
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.4.4, 2.0.0
>Reporter: stack
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.4, 2.0.1
>
> Attachments: HBASE-20404.0.patch, HBASE-20404.1.patch,
> HBASE-20404.2.patch
>
>
> I see these big dirty exceptions in my master log during a long-run Lets
> clean them up (Are they exceptions I as an operator can actually do something
> about? Are they 'problems'? Should they be LOG.warn?)
> {code}
> 2018-04-12 16:02:09,911 WARN [ForkJoinPool-1-worker-15]
> cleaner.CleanerChore: Could not delete dir under
> hdfs://ve0524.halxg.cloudera.com:8020/hbase/archive/data/default/IntegrationTestBigLinkedList/1e24549061df3adc4858fbcaf1929553/meta;
> {}
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException):
>
> `/hbase/archive/data/default/IntegrationTestBigLinkedList/1e24549061df3adc4858fbcaf1929553/meta
> is non empty': Directory is not empty
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(FSDirDeleteOp.java:115)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:2848)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:1048)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:641)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489)
> at org.apache.hadoop.ipc.Client.call(Client.java:1435)
> at org.apache.hadoop.ipc.Client.call(Client.java:1345)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy26.delete(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:568)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
> at com.sun.proxy.$Proxy27.delete(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> ...
> {code}
> Looks like log format is off too...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19391) Calling HRegion#initializeRegionInternals from a region replica can still re-create a region directory
[ https://issues.apache.org/jira/browse/HBASE-19391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720763#comment-16720763 ] Hudson commented on HBASE-19391: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-19391 Calling HRegion#initializeRegionInternals from a region (apurtell: rev 453935f31d3cd56e0db54fce34fdd3078d0b7981) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java > Calling HRegion#initializeRegionInternals from a region replica can still > re-create a region directory > -- > > Key: HBASE-19391 > URL: https://issues.apache.org/jira/browse/HBASE-19391 > Project: HBase > Issue Type: Bug >Affects Versions: 2.0.0-alpha-4, 1.4.2 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Major > Fix For: 1.5.0, 2.0.0-beta-2, 1.3.3, 1.4.3, 2.0.0 > > Attachments: > 0001-HBASE-19391-Calling-HRegion-initializeRegionInternal.patch, > HBASE-19391.master.v0.patch > > > This is a follow up from HBASE-18024. There stills a chance that attempting > to open a region that is not the default region replica can still create a > GC'd region directory by the CatalogJanitor causing inconsistencies with hbck. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20672) New metrics ReadRequestRate and WriteRequestRate
[ https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720775#comment-16720775 ] Hudson commented on HBASE-20672: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20672 New metrics ReadRequestRate and WriteRequestRate (apurtell: rev 08246957e7b37f71ed3874e153ec79e69a87fc5a) * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java > New metrics ReadRequestRate and WriteRequestRate > > > Key: HBASE-20672 > URL: https://issues.apache.org/jira/browse/HBASE-20672 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Ankit Jain >Assignee: Ankit Jain >Priority: Minor > Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.10 > > Attachments: HBASE-20672.branch-1.001.patch, > HBASE-20672.branch-1.002.patch, HBASE-20672.branch-2.001.patch, > HBASE-20672.master.001.patch, HBASE-20672.master.002.patch, > HBASE-20672.master.003.patch, hits1vs2.4.40.400.png > > > Hbase currently provides counter read/write requests (ReadRequestCount, > WriteRequestCount). That said it is not easy to use counter that reset only > after a restart of the service, we would like to expose 2 new metrics in > HBase to provide ReadRequestRate and WriteRequestRate at region server level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20523) PE tool should support configuring client side buffering sizes
[ https://issues.apache.org/jira/browse/HBASE-20523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720770#comment-16720770 ] Hudson commented on HBASE-20523: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20523 PE tool should support configuring client side buffering (apurtell: rev 017cb75bd306226dbae7904b6181b99c7576be22) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/TestPerformanceEvaluation.java > PE tool should support configuring client side buffering sizes > -- > > Key: HBASE-20523 > URL: https://issues.apache.org/jira/browse/HBASE-20523 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.1.0, 1.5.0, 2.0.1, 1.4.5, 1.2.7 > > Attachments: HBASE-20523.patch, HBASE-20523_1.patch, > HBASE-20523_branch-1.2.patch, HBASE-20523_branch-1.4.patch, > HBASE-20523_branch-1.patch > > > The client side buffering size impacts the write load and the write > performance. Hence for testing purpose it is better we allow client side > buffering to be configurable in PE. Already YCSB has such facility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20352) [Chore] Backport HBASE-18309 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720767#comment-16720767 ] Hudson commented on HBASE-20352: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20352 [Chore] Backport HBASE-18309 (Support multi threads in (apurtell: rev 193d1dcb72c22252fc86ee8433c765c42349d3cc) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileLinkCleaner.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FileStatusFilter.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java > [Chore] Backport HBASE-18309 to branch-1 > > > Key: HBASE-20352 > URL: https://issues.apache.org/jira/browse/HBASE-20352 > Project: HBase > Issue Type: Improvement >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.4 > > Attachments: HBASE-20352.branch-1.001.patch > > > Using multiple threads to scan directory and to clean old WALs -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20557) Backport HBASE-17215 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-20557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720778#comment-16720778 ] Hudson commented on HBASE-20557: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-20557 Backport HBASE-17215 to branch-1 (apurtell: rev 30b1dc00b47a0289a721cd0f932d9cdd8422534e) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java > Backport HBASE-17215 to branch-1 > > > Key: HBASE-20557 > URL: https://issues.apache.org/jira/browse/HBASE-20557 > Project: HBase > Issue Type: Sub-task > Components: HFile, master >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.6 > > Attachments: HBASE-20557.branch-1.001.patch, > HBASE-20557.branch-1.002.patch, HBASE-20557.branch-1.003.patch > > > As part of HBASE-20555, HBASE-17215 is the second patch that is needed for > backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20808) Wrong shutdown order between Chores and ChoreService
[
https://issues.apache.org/jira/browse/HBASE-20808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720777#comment-16720777
]
Hudson commented on HBASE-20808:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20808 Wrong shutdown order between Chores and ChoreService (apurtell: rev
8e36761e89d0ecdd6a476f06293c5d1c8295eb06)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> Wrong shutdown order between Chores and ChoreService
>
>
> Key: HBASE-20808
> URL: https://issues.apache.org/jira/browse/HBASE-20808
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Nihal Jain
>Priority: Minor
> Labels: beginner
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20808.branch-1.001.patch,
> HBASE-20808.branch-1.002.patch, HBASE-20808.master.001.patch,
> HBASE-20808.master.002.patch, HBASE-20808.master.003.patch,
> HBASE-20808.master.004.patch, HBASE-20808.master.addendum.patch
>
>
> When stopping master, {{ChoreService}}, which serves all the chores, is
> stopped before canceling all running chores.
> It should cancel all running chores, then shutdown {{ChoreService}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17854) Use StealJobQueue in HFileCleaner after HBASE-17215
[
https://issues.apache.org/jira/browse/HBASE-17854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720781#comment-16720781
]
Hudson commented on HBASE-17854:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20558 Port HBASE-17854 (Use StealJobQueue in HFileCleaner after
(apurtell: rev 976f07e87cbeb35b80a7ca8eea46f973e932bf83)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/util/StealJobQueue.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestStealJobQueue.java
> Use StealJobQueue in HFileCleaner after HBASE-17215
> ---
>
> Key: HBASE-17854
> URL: https://issues.apache.org/jira/browse/HBASE-17854
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-17854.patch, HBASE-17854.v2.patch,
> HBASE-17854.v3.patch, HBASE-17854.v4.patch, HBASE-17854.v5.patch,
> HBASE-17854.v5.patch
>
>
> In HBASE-17215 we use specific threads for deleting large/small (archived)
> hfiles, and will improve it from below aspects in this JIRA:
> 1. Using {{StealJobQueue}} to allow large file deletion thread to steal jobs
> from small queue, based on the experience that in real world there'll be much
> more small hfiles
> 2. {{StealJobQueue}} is a kind of {{PriorityQueue}}, so we could also delete
> from the larger file in the queues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20732) Shutdown scan pool when master is stopped.
[
https://issues.apache.org/jira/browse/HBASE-20732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720776#comment-16720776
]
Hudson commented on HBASE-20732:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20732 Shutdown scan pool when master is stopped (apurtell: rev
955264ed437165fbb26f04172f617fb44835ff6b)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestCleanerChore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> Shutdown scan pool when master is stopped.
> --
>
> Key: HBASE-20732
> URL: https://issues.apache.org/jira/browse/HBASE-20732
> Project: HBase
> Issue Type: Bug
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20732.branch-1.001.patch,
> HBASE-20732.master.001.patch, HBASE-20732.master.002.patch,
> HBASE-20732.master.003.patch, HBASE-20732.master.004.patch,
> HBASE-20732.master.005.patch, HBASE-20732.master.006.patch,
> HBASE-20732.master.007.patch, HBASE-20732.master.008.patch,
> HBASE-20732.master.009.patch
>
>
> If master is stopped, {{DirScanPool}} is kept open. This is found by
> [~chia7712] when reviewing HBASE-20352.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-16399) Provide an API to get list of failed regions and servername in Canary
[ https://issues.apache.org/jira/browse/HBASE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720743#comment-16720743 ] Hudson commented on HBASE-16399: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-16399 Provide an API to get list of failed regions and servername (apurtell: rev 46f265a73b2b7485b29d2f919460ce72bfea3769) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java > Provide an API to get list of failed regions and servername in Canary > - > > Key: HBASE-16399 > URL: https://issues.apache.org/jira/browse/HBASE-16399 > Project: HBase > Issue Type: Improvement > Components: canary >Affects Versions: 1.3.1, 0.98.21 >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Major > Fix For: 1.4.0, 0.98.22, 1.3.3, 2.0.0 > > Attachments: HBASE-16399.0.98.00.patch, HBASE-16399.0.98.01.patch, > HBASE-16399.0.98.02.patch, HBASE-16399.00.patch, HBASE-16399.01.patch, > HBASE-16399.02.patch, HBASE-16399.03.patch, HBASE-16399.branch-1.00.patch, > HBASE-16399.branch-1.01.patch, HBASE-16399.branch-1.02.patch, > HBASE-16399.branch-1.03.patch > > > At present HBase Canary tool only prints the failures as part of logs. It > does not provide an API to get the list or summarizes it so caller can take > action on the failed host. This Jira would additional API so caller can get > read or write canary failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17118) StoreScanner leaked in KeyValueHeap
[
https://issues.apache.org/jira/browse/HBASE-17118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720751#comment-16720751
]
Hudson commented on HBASE-17118:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-17118 StoreScanner leaked in KeyValueHeap (binlijin) (apurtell: rev
25135dd0a13538e4dd4ffdd34dcb3120a010be3b)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
> StoreScanner leaked in KeyValueHeap
> ---
>
> Key: HBASE-17118
> URL: https://issues.apache.org/jira/browse/HBASE-17118
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.1.7, 1.2.4
>Reporter: binlijin
>Assignee: binlijin
>Priority: Major
> Fix For: 1.4.0, 1.2.5, 1.1.8, 1.3.3, 2.0.0
>
> Attachments: HBASE-17118-master_v1.patch,
> HBASE-17118-master_v2.patch, HBASE-17118-master_v3.patch,
> HBASE-17118-master_v4.patch, HBASE-17118-master_v5.patch,
> HBASE-17118.branch-1.1.v1.patch, HBASE-17118.branch-1.2.v1.patch,
> HBASE-17118.branch-1.addnumv1.patch, HBASE-17118.branch-1.v1.patch,
> StoreScanner.png, StoreScannerLeakHeap.png
>
>
> KeyValueHeap#generalizedSeek
> KeyValueScanner scanner = current;
> while (scanner != null) {
> Cell topKey = scanner.peek();
> ..
> boolean seekResult;
> if (isLazy && heap.size() > 0) {
> // If there is only one scanner left, we don't do lazy seek.
> seekResult = scanner.requestSeek(seekKey, forward, useBloom);
> } else {
> seekResult = NonLazyKeyValueScanner.doRealSeek(scanner, seekKey,
> forward);
> }
> ..
> scanner = heap.poll();
> }
> (1) scanner = heap.poll(); Retrieves and removes the head of this queue
> (2) scanner.requestSeek(seekKey, forward, useBloom); or
> NonLazyKeyValueScanner.doRealSeek(scanner, seekKey, forward);
> throw exception, and scanner will have no chance to close, so will cause the
> scanner leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19435) Reopen Files for ClosedChannelException in BucketCache
[ https://issues.apache.org/jira/browse/HBASE-19435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720757#comment-16720757 ] Hudson commented on HBASE-19435: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-19435 Reopen Files for ClosedChannelException in BucketCache (apurtell: rev 4b6e589bbbe600ad68c205c758ea7f90ff3c67c7) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestFileIOEngine.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java > Reopen Files for ClosedChannelException in BucketCache > -- > > Key: HBASE-19435 > URL: https://issues.apache.org/jira/browse/HBASE-19435 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 1.3.1, 2.0.0 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Fix For: 1.4.0, 2.0.0-beta-1, 1.3.3, 2.0.0 > > Attachments: HBASE-19435.branch-1.001.patch, > HBASE-19435.master.001.patch, HBASE-19435.master.002.patch, > HBASE-19435.master.003.patch, HBASE-19435.master.004.patch, > HBASE-19435.master.005.patch, HBASE-19435.master.006.patch, > HBASE-19435.master.007.patch, HBASE-19435.master.007.patch > > > When using the FileIOEngine for BucketCache, the cache will be disabled if > the connection is interrupted or closed. HBase will then get > ClosedChannelExceptions trying to access the file. After 60s, the RS will > disable the cache. This causes severe read performance degradation for > workloads that rely on this cache. FileIOEngine never tries to reopen the > connection. This JIRA is to reopen files when the BucketCache encounters a > ClosedChannelException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18830) TestCanaryTool does not check Canary monitor's error code
[ https://issues.apache.org/jira/browse/HBASE-18830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720747#comment-16720747 ] Hudson commented on HBASE-18830: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) Amend HBASE-18830 TestCanaryTool does not check Canary monitor's error (apurtell: rev 5d0d3aa2b962a27a7eb33b20ef91476893380b89) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java > TestCanaryTool does not check Canary monitor's error code > - > > Key: HBASE-18830 > URL: https://issues.apache.org/jira/browse/HBASE-18830 > Project: HBase > Issue Type: Bug >Reporter: Chinmay Kulkarni >Assignee: Chinmay Kulkarni >Priority: Major > Fix For: 1.4.0, 1.3.3, 2.0.0 > > Attachments: HBASE-18830.001.patch > > > None of the tests inside TestCanaryTool check Canary monitor's error code. > Thus, it is possible that the monitor has registered an error and yet the > tests pass. We should check the value returned by the _ToolRunner.run()_ > method inside each unit test. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18058) Zookeeper retry sleep time should have an upper limit
[
https://issues.apache.org/jira/browse/HBASE-18058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720753#comment-16720753
]
Hudson commented on HBASE-18058:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-18058 Zookeeper retry sleep time should have an upper limit (Allan
(apurtell: rev ed4f7d1b1b4497caf859da6119c2940fdfaba9a9)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
* (edit) hbase-common/src/main/resources/hbase-default.xml
> Zookeeper retry sleep time should have an upper limit
> -
>
> Key: HBASE-18058
> URL: https://issues.apache.org/jira/browse/HBASE-18058
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0, 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 1.4.0, 1.3.3, 2.0.0
>
> Attachments: HBASE-18058-branch-1.patch,
> HBASE-18058-branch-1.v2.patch, HBASE-18058-branch-1.v3.patch,
> HBASE-18058.patch, HBASE-18058.v2.patch
>
>
> Now, in {{RecoverableZooKeeper}}, the retry backoff sleep time grow
> exponentially, but it doesn't have any up limit. It directly lead to a long
> long recovery time after Zookeeper going down for some while and come back.
> A case of damage done by high sleep time:
> If the server hosting zookeeper is disk full, the zookeeper quorum won't
> really went down but reject all write request. So at HBase side, new zk write
> request will suffers from exception and retry. But connection remains so the
> session won't timeout. When disk full situation have been resolved, the
> zookeeper quorum can work normally again. But the very high sleep time cause
> some module of RegionServer/HMaster will still sleep for a long time(for
> example, the balancer) before working.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20052) TestRegionOpen#testNonExistentRegionReplica fails due to NPE
[
https://issues.apache.org/jira/browse/HBASE-20052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720764#comment-16720764
]
Hudson commented on HBASE-20052:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-20052 TestRegionOpen#testNonExistentRegionReplica fails due to NPE
(apurtell: rev 8bc84fdc1b67ce2b28a2a8774defe669dfca1501)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
> TestRegionOpen#testNonExistentRegionReplica fails due to NPE
>
>
> Key: HBASE-20052
> URL: https://issues.apache.org/jira/browse/HBASE-20052
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.3, 2.0.0
>
> Attachments: 20052.v1.txt, 20052.v2.txt
>
>
> After HBASE-19391 was integrated, the following test failure can be observed:
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.TestRegionOpen.testNonExistentRegionReplica(TestRegionOpen.java:122)
> {code}
> This was due null being returned from
> HRegionFileSystem#createRegionOnFileSystem().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17731) Fractional latency reporting in MultiThreadedAction
[ https://issues.apache.org/jira/browse/HBASE-17731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720750#comment-16720750 ] Hudson commented on HBASE-17731: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-17731 Fractional latency reporting in MultiThreadedAction (apurtell: rev 286ade8155e5c198feef83bbd5fcc4f00a3d7796) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/util/MultiThreadedAction.java > Fractional latency reporting in MultiThreadedAction > --- > > Key: HBASE-17731 > URL: https://issues.apache.org/jira/browse/HBASE-17731 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Trivial > Fix For: 1.4.0, 1.3.3, 2.0.0 > > Attachments: HBASE-17731.patch > > > When average latency is less than one millisecond the LoadTestTool tool > reports a latency of 0. Better to report a fraction out to a couple of > decimal points. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17912) Avoid major compactions on region server startup
[ https://issues.apache.org/jira/browse/HBASE-17912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720740#comment-16720740 ] Hudson commented on HBASE-17912: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-17912 Avoid major compactions on region server startup (apurtell: rev 35907e401e7dac0ef9c23899d63393577e57dcec) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Avoid major compactions on region server startup > > > Key: HBASE-17912 > URL: https://issues.apache.org/jira/browse/HBASE-17912 > Project: HBase > Issue Type: Improvement > Components: Compaction >Affects Versions: 1.3.1, 0.98.24, 2.0.0 >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 1.4.0, 1.3.3, 2.0.0 > > Attachments: HBASE-17912.patch > > > The HRegionServer.CompactionChecker chore wakes up every 10s and for each > store in each region mods against a chore frequency (by default slightly > under 3 hours) to see if it's time to check if a major compaction is > necessary for that store. > The check to see if it needs to check for major compactions is calculated by > if (iteration % multiplier != 0) continue; > where iteration is the number of times the chore has woken up. > Because 0 % anything is 0, this will always check for necessary major > compactions on each store when this chore is first run after the region > server starts up. This can result in compaction storms when doing a rolling > restart, because, for example, the new instance of the region server might > get a lower jitter value than the old one had. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17930) Avoid using Canary.sniff in HBaseTestingUtility
[ https://issues.apache.org/jira/browse/HBASE-17930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720744#comment-16720744 ] Hudson commented on HBASE-17930: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-17930 Avoid using Canary.sniff in HBaseTestingUtility (apurtell: rev 59d36e5d64676409a2d9769a9f28c71a7681ac0a) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java > Avoid using Canary.sniff in HBaseTestingUtility > --- > > Key: HBASE-17930 > URL: https://issues.apache.org/jira/browse/HBASE-17930 > Project: HBase > Issue Type: Bug > Components: canary, test >Affects Versions: 1.4.0, 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: trivial > Fix For: 1.4.0, 1.3.3, 2.0.0 > > Attachments: HBASE-17930.patch > > > It will make it easier to rewrite Canary with async client in the future. > And it is also very easy to write a simple sniff method for > HBaseTestingUtility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19553) Old replica regions should be cleared from AM memory after primary region split or merge
[ https://issues.apache.org/jira/browse/HBASE-19553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720761#comment-16720761 ] Hudson commented on HBASE-19553: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-19553 Old replica regions should be cleared from AM memory after (apurtell: rev 5c799c18f684121352b1f071676a442a8efdc8a1) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin1.java > Old replica regions should be cleared from AM memory after primary region > split or merge > > > Key: HBASE-19553 > URL: https://issues.apache.org/jira/browse/HBASE-19553 > Project: HBase > Issue Type: Bug > Components: master >Reporter: huaxiang sun >Assignee: Pankaj Kumar >Priority: Minor > Fix For: 1.5.0, 1.3.3, 1.4.3 > > Attachments: HBASE-19553-branch-1-v2.patch, > HBASE-19553-branch-1-v3.patch, HBASE-19553-branch-1-v4.patch, > HBASE-19553-branch-1-v4.patch, HBASE-19553-branch-1.patch > > > Similar to HBASE-18025, the replica parent's info is not removed from master. > Actually I think it can be removed after replica region is split or merged, I > will check the logic and apply one patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17991) Add more details about compaction queue on /dump
[
https://issues.apache.org/jira/browse/HBASE-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720754#comment-16720754
]
Hudson commented on HBASE-17991:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-17991 Add more details about compaction queue on /dump (Guangxu
(apurtell: rev 3e0d693935e9a03cf4b9a078b32f2b7cc4e0c95f)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java
> Add more details about compaction queue on /dump
>
>
> Key: HBASE-17991
> URL: https://issues.apache.org/jira/browse/HBASE-17991
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Minor
> Fix For: 1.4.0, 1.3.3, 2.0.0
>
> Attachments: HBASE-17991-master-v1.patch,
> HBASE-17991-master-v2.patch, HBASE-17991-master-v3.patch
>
>
> RS dump information as follows
> {code}
> RS Queue:
> ===
> Compaction/Split Queue summary: compaction_queue=(20:0), split_queue=0,
> merge_queue=0
> Compaction/Split Queue dump:
> LargeCompation Queue:
> Request =
> regionName=usertable,user4180497275766179957,1491904095205.1d4085e4438752f3611f66e7b043fe44.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920409804647
> Request =
> regionName=usertable,user4568009753557153251,1491904099262.95bf004e3c9b35a58c60ca5d5b11d190.,
> storeName=0, fileCount=1, fileSize=1.9 G (1.9 G), priority=1,
> time=21697920413223800
> SmallCompation Queue:
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 108
> Store = b, pri = 109
> {code}
> Compaction queue information will be displayed on page /dump.
> If compation has selected the file, it will print the details information of
> the compation, otherwise only print storename and priority(Store = b, pri =
> 108) which is useless for us.
> So, we should also print more detailed information, such as regionName,
> starttime etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18786) FileNotFoundException should not be silently handled for primary region replicas
[ https://issues.apache.org/jira/browse/HBASE-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720756#comment-16720756 ] Hudson commented on HBASE-18786: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-18786 FileNotFoundException should not be silently handled for (apurtell: rev b442a7fc92763fb63ed1c01ef4072c9823836d79) * (delete) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCorruptedRegionStoreFile.java * (delete) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAbort.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > FileNotFoundException should not be silently handled for primary region > replicas > > > Key: HBASE-18786 > URL: https://issues.apache.org/jira/browse/HBASE-18786 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Ashu Pachauri >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.4.0, 1.3.3, 2.0.0 > > Attachments: HBASE-18786-branch-1.3.patch, > HBASE-18786-branch-1.patch, HBASE-18786-branch-1.patch, HBASE-18786.patch, > HBASE-18786.patch > > > This is a follow up for HBASE-18186. > FileNotFoundException while scanning from a primary region replica can be > indicative of a more severe problem. Handling them silently can cause many > underlying issues go undetected. We should either > 1. Hard fail the regionserver if there is a FNFE on a primary region replica, > OR > 2. Report these exceptions as some region / server level metric so that these > can be proactively investigated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-17924) Consider sorting the row order when processing multi() ops before taking rowlocks
[
https://issues.apache.org/jira/browse/HBASE-17924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720752#comment-16720752
]
Hudson commented on HBASE-17924:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See
[https://builds.apache.org/job/HBase-1.3-IT/509/])
HBASE-17924 Consider sorting the row order when processing multi() ops
(apurtell: rev 4e4756e68887e3a1d77e4483f52f431b441611a5)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
> Consider sorting the row order when processing multi() ops before taking
> rowlocks
> -
>
> Key: HBASE-17924
> URL: https://issues.apache.org/jira/browse/HBASE-17924
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.1.8, 2.0.0
>Reporter: Andrew Purtell
>Assignee: Allan Yang
>Priority: Major
> Fix For: 1.4.0, 1.3.3, 2.0.0
>
> Attachments: HBASE-17924.patch, HBASE-17924.v0.patch,
> HBASE-17924.v2.patch, HBASE-17924.v3.patch, HBASE-17924.v4.patch,
> HBASE-17924.v5.patch
>
>
> When processing a batch mutation, we take row locks in whatever order the
> mutations were added to the multi op by the client.
>
> {noformat}
> RSRpcServices#multi -> RSRpcServices#mutateRows -> HRegion#mutateRow ->
> HRegion#mutateRowsWithLocks -> HRegion#processRowsWithLocks
> {noformat}
> Or
> {noformat}
> RSRpcServices#multi -> RSRpcServices#doNonAtomicRegionMutation ->
> HRegion#get
> | HRegion#append
> | HRegion#increment
> | HRegionServer#doBatchOp -> HRegion#batchMutate ->
> HRegion#doMiniBatchMutation
> {noformat}
>
> multi() is fed by client APIs that accept a RowMutations object containing
> actions for multiple rows. The container for ops inside RowMutations is an
> ArrayList, which doesn't change the ordering of objects added to it. The
> protobuf implementation of the messages for multi ops do not reorder the list
> of actions. When processing multi ops we iterate over the actions in the
> order rehydrated from protobuf.
> We should discuss sorting the order of ops by row key when processing multi()
> ops before taking row locks. Does this make lock ordering more predictable
> for server side operations? Yes, but potentially surprising for the client,
> right? Is there any legitimate reason we should take locks out of row key
> sorted order because the client has structured the request as such?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18762) Canary sink type cast error
[ https://issues.apache.org/jira/browse/HBASE-18762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720746#comment-16720746 ] Hudson commented on HBASE-18762: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-18762 Canary sink type cast error (apurtell: rev e72ed4f1914edf4752c3ff7630e51a1265a50b61) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java > Canary sink type cast error > --- > > Key: HBASE-18762 > URL: https://issues.apache.org/jira/browse/HBASE-18762 > Project: HBase > Issue Type: Bug >Reporter: Chinmay Kulkarni >Assignee: Chinmay Kulkarni >Priority: Major > Fix For: 1.4.0, 1.3.3, 2.0.0 > > Attachments: HBASE-18762.001.patch, HBASE-18830-addendum.patch > > > When running the main method of Canary.java, we see the following error: > Exception in thread "main" java.lang.ClassCastException: > org.apache.hadoop.hbase.tool.Canary$RegionServerStdOutSink cannot be cast to > org.apache.hadoop.hbase.tool.Canary$RegionStdOutSink > at org.apache.hadoop.hbase.tool.Canary.newMonitor(Canary.java:911) > at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:796) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1571) > This happens because we typecast the sink depending on the mode (zookeeper > mode/region server mode) that Canary is configured in. In case no mode is > specified, we typecast the sink into _RegionStdOutSink_. In general, it is > possible to provide inconsistent mode and sink types while running Canary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15727) Canary Tool for Zookeeper
[ https://issues.apache.org/jira/browse/HBASE-15727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16720742#comment-16720742 ] Hudson commented on HBASE-15727: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #509 (See [https://builds.apache.org/job/HBase-1.3-IT/509/]) HBASE-15727 Canary Tool for Zookeeper (churro morales) (apurtell: rev cd9037d49538a17e549de5963edab7c144b11563) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java > Canary Tool for Zookeeper > - > > Key: HBASE-15727 > URL: https://issues.apache.org/jira/browse/HBASE-15727 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: churro morales >Assignee: churro morales >Priority: Major > Fix For: 1.4.0, 0.98.20, 1.3.3, 2.0.0 > > Attachments: HBASE-15727-v1.patch, HBASE-15727-v2.patch, > HBASE-15727-v3.patch, HBASE-15727.patch, HBASE-15727.v4.patch, > HBASE-15727.v5.patch > > > It would be nice to have the canary tool also monitor zookeeper. Something > simple like doing a getData() call on zookeeper.znode.parent > It would be nice to create clients for every instance in the quorum such that > you could monitor overloaded or poor behaving instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
