[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498233#comment-15498233
 ] 

Ashish Singhi commented on HBASE-16257:
---

You got it right [~jerryhe].

bq. the staging dir will not be there under the 1.x root, which will then case 
the HFileReplicator to fail.
Sorry I did not get this why HFileReplicator will fail here ?

> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, 
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then 
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
>   + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir.  We are bringing 
> secure bulkload to the core. It makes sense to bring it under core control as 
> well, instead of an optional property.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16647) hbck should do offline reference repair before online repair

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498211#comment-15498211
 ] 

Jerry He commented on HBASE-16647:
--

The error state gets cleared in later stages after the move of 
offlineReferenceFileRepair to the beginning.
Will fix the test.
Thanks, Ted.

> hbck should do offline reference repair before online repair
> 
>
> Key: HBASE-16647
> URL: https://issues.apache.org/jira/browse/HBASE-16647
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
> Attachments: HBASE-16647.patch
>
>
> {noformat}
> hbck
> -fixReferenceFiles  Try to offline lingering reference store files
> Metadata Repair shortcuts
> -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles 
> -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps 
> -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes
> {noformat}
> Bad reference files prevent the region from coming online.
> If used in the shortcut combination, the reference files should be fixed 
> before other online fix.
> I have seen repeated '-repair' did not work because bad reference files 
> failed regions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498196#comment-15498196
 ] 

Ashish Singhi commented on HBASE-16598:
---

+1

> Enable zookeeper useMulti always and clean up in HBase code
> ---
>
> Key: HBASE-16598
> URL: https://issues.apache.org/jira/browse/HBASE-16598
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, 
> HBASE-16598-v4.patch, HBASE-16598.patch
>
>
> We have zookeeper useMulti default true since HBase 1.0.0
> And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0"
> The benefit of useMulti is obvious. 
> Let's clean up the code to remove useMulti false case so that we don't have 
> to continue with the hassle of maintaining two version of the code (e.g. in 
> replication) .  No go back to pre 3.4.x Zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498188#comment-15498188
 ] 

Hudson commented on HBASE-16544:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1619 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1619/])
HBASE-16544 Remove or Clarify Using Amazon S3 Storage section in the (jerryjch: 
rev bb3d9ccd489fb64e3cb2020583935a393382a678)
* (edit) src/main/asciidoc/_chapters/ops_mgt.adoc


> Remove or Clarify  'Using Amazon S3 Storage' section in the reference guide
> ---
>
> Key: HBASE-16544
> URL: https://issues.apache.org/jira/browse/HBASE-16544
> Project: HBase
>  Issue Type: Bug
>  Components: documentation, Filesystem Integration
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-16544-V1.patch, HBASE-16544-addendum.patch
>
>
> reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration
> (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I 
> think this title means that we can use S3 storage to replace HDFS, I really 
> tried this :(,   always give me errors and hbase even can not start, see 
> error mentioned in jira HBASE-11045.
> (2) And the details in this section are more about deploying HBase on Amazon 
> EC2 cluster, which has nothing to do 'using Amazon S3 storage'
> (3) In all, I think we need to remove this section, or at least clarify this 
> section if someone fully test HBase on S3.  see HBASE-15646 for more details 
> about this doc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16647) hbck should do offline reference repair before online repair

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498159#comment-15498159
 ] 

Ted Yu commented on HBASE-16647:


Looks like testLingeringReferenceFile needs to be adjusted for the change.

> hbck should do offline reference repair before online repair
> 
>
> Key: HBASE-16647
> URL: https://issues.apache.org/jira/browse/HBASE-16647
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
> Attachments: HBASE-16647.patch
>
>
> {noformat}
> hbck
> -fixReferenceFiles  Try to offline lingering reference store files
> Metadata Repair shortcuts
> -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles 
> -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps 
> -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes
> {noformat}
> Bad reference files prevent the region from coming online.
> If used in the shortcut combination, the reference files should be fixed 
> before other online fix.
> I have seen repeated '-repair' did not work because bad reference files 
> failed regions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498121#comment-15498121
 ] 

Ted Yu commented on HBASE-16646:


Patch v2 fixes javadoc and findbugs warnings.

> Enhance LoadIncrementalHFiles API to accept store file paths as input
> -
>
> Key: HBASE-16646
> URL: https://issues.apache.org/jira/browse/HBASE-16646
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16646.v0.txt, 16646.v1.txt, 16646.v2.txt
>
>
> Currently LoadIncrementalHFiles takes the directory (output path) as input 
> parameter.
> In some scenarios (incremental restore of bulk loaded hfiles), the List of 
> paths to hfiles is known.
> LoadIncrementalHFiles can take the List as input parameter and proceed with 
> loading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-16646:
---
Attachment: 16646.v2.txt

> Enhance LoadIncrementalHFiles API to accept store file paths as input
> -
>
> Key: HBASE-16646
> URL: https://issues.apache.org/jira/browse/HBASE-16646
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16646.v0.txt, 16646.v1.txt, 16646.v2.txt
>
>
> Currently LoadIncrementalHFiles takes the directory (output path) as input 
> parameter.
> In some scenarios (incremental restore of bulk loaded hfiles), the List of 
> paths to hfiles is known.
> LoadIncrementalHFiles can take the List as input parameter and proceed with 
> loading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498088#comment-15498088
 ] 

Hadoop QA commented on HBASE-16646:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 5s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 46s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s 
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 
total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 29s 
{color} | {color:red} hbase-server generated 1 new + 1 unchanged - 0 fixed = 2 
total (was 1) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 21s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 139m 18s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
|  |  Nullcheck of queue at line 478 of value previously dereferenced in 
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.performBulkLoad(Admin, 
Table, RegionLocator, Deque)  At LoadIncrementalHFiles.java:478 of value 
previously dereferenced in 
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.performBulkLoad(Admin, 
Table, RegionLocator, Deque)  At LoadIncrementalHFiles.java:[line 427] |
| Failed junit tests | 
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
|   | hadoop.hbase.regionserver.TestHRegion |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
|   | org.apache.hadoop.hbase.wal.TestWALFiltering |
|   | org.apache.hadoop.hbase.wal.TestWALSplitCompressed |
|   | org.apache.hadoop.hbase.TestZooKeeper |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828925/16646.v1.txt |
| JIRA Issue | HBASE-16646 |

[jira] [Commented] (HBASE-16534) Procedure v2 - Perf Tool for Scheduler

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498073#comment-15498073
 ] 

Hadoop QA commented on HBASE-16534:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
24m 42s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s 
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 48s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
25s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 117m 11s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure |
|   | org.apache.hadoop.hbase.replication.TestReplicationStateHBaseImpl |
|   | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures 
|
|   | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
|   | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828941/HBASE-16534.master.002.patch
 |
| JIRA Issue | HBASE-16534 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 04d88e67435b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 1a1003a |
| Default Java | 1.8.0_101 

[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498070#comment-15498070
 ] 

Jerry He commented on HBASE-16257:
--

Hi, [~enis]
Thinking a bit more. Correct me if I am wrong.
HFileReplicator operates on the sink side. It tries to copy from the sourceFs 
to local sink staging dir.  If we have 2.x -> 1.x, there is no rootDir/staging 
on the sink. But it is ok. Sink knows how to handle its own staging dir, new 
way or old way.  Source will not try to access sink staging dir and sink will 
not try to access source staging dir.

> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, 
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then 
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
>   + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir.  We are bringing 
> secure bulkload to the core. It makes sense to bring it under core control as 
> well, instead of an optional property.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497934#comment-15497934
 ] 

Jerry He commented on HBASE-16544:
--

Committed the addendum.  Thanks.

> Remove or Clarify  'Using Amazon S3 Storage' section in the reference guide
> ---
>
> Key: HBASE-16544
> URL: https://issues.apache.org/jira/browse/HBASE-16544
> Project: HBase
>  Issue Type: Bug
>  Components: documentation, Filesystem Integration
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-16544-V1.patch, HBASE-16544-addendum.patch
>
>
> reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration
> (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I 
> think this title means that we can use S3 storage to replace HDFS, I really 
> tried this :(,   always give me errors and hbase even can not start, see 
> error mentioned in jira HBASE-11045.
> (2) And the details in this section are more about deploying HBase on Amazon 
> EC2 cluster, which has nothing to do 'using Amazon S3 storage'
> (3) In all, I think we need to remove this section, or at least clarify this 
> section if someone fully test HBase on S3.  see HBASE-15646 for more details 
> about this doc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16647) hbck should do offline reference repair before online repair

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497932#comment-15497932
 ] 

Hadoop QA commented on HBASE-16647:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 27s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 53s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 23s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.util.TestHBaseFsckTwoRS |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
|   | org.apache.hadoop.hbase.client.TestEnableTable |
|   | 
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
|   | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
|   | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient |
|   | org.apache.hadoop.hbase.client.TestIncrementsFromClientSide |
|   | org.apache.hadoop.hbase.client.TestResultSizeEstimation |
|   | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828927/HBASE-16647.patch |
| JIRA Issue | HBASE-16647 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux a336abed431e 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 1a1003a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 

[jira] [Updated] (HBASE-16534) Procedure v2 - Perf Tool for Scheduler

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-16534:

Fix Version/s: (was: 1.4.0)

> Procedure v2 - Perf Tool for Scheduler
> --
>
> Key: HBASE-16534
> URL: https://issues.apache.org/jira/browse/HBASE-16534
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, tooling
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-16534.master.001.patch, 
> HBASE-16534.master.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16534) Procedure v2 - Perf Tool for Scheduler

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-16534:

Status: Patch Available  (was: Open)

> Procedure v2 - Perf Tool for Scheduler
> --
>
> Key: HBASE-16534
> URL: https://issues.apache.org/jira/browse/HBASE-16534
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, tooling
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16534.master.001.patch, 
> HBASE-16534.master.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles API to accept store file paths as input

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-16646:
---
Status: Patch Available  (was: Open)

> Enhance LoadIncrementalHFiles API to accept store file paths as input
> -
>
> Key: HBASE-16646
> URL: https://issues.apache.org/jira/browse/HBASE-16646
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16646.v0.txt, 16646.v1.txt
>
>
> Currently LoadIncrementalHFiles takes the directory (output path) as input 
> parameter.
> In some scenarios (incremental restore of bulk loaded hfiles), the List of 
> paths to hfiles is known.
> LoadIncrementalHFiles can take the List as input parameter and proceed with 
> loading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497701#comment-15497701
 ] 

Jerry He commented on HBASE-16257:
--

Good point.Let me think thru this case.
FYI Ashish Singhi

> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, 
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then 
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
>   + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir.  We are bringing 
> secure bulkload to the core. It makes sense to bring it under core control as 
> well, instead of an optional property.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16647) hbck should do offline reference repair before online repair

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-16647:
-
Status: Patch Available  (was: Open)

> hbck should do offline reference repair before online repair
> 
>
> Key: HBASE-16647
> URL: https://issues.apache.org/jira/browse/HBASE-16647
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
> Attachments: HBASE-16647.patch
>
>
> {noformat}
> hbck
> -fixReferenceFiles  Try to offline lingering reference store files
> Metadata Repair shortcuts
> -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles 
> -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps 
> -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes
> {noformat}
> Bad reference files prevent the region from coming online.
> If used in the shortcut combination, the reference files should be fixed 
> before other online fix.
> I have seen repeated '-repair' did not work because bad reference files 
> failed regions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16647) hbck should do offline reference repair before online repair

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-16647:
-
Attachment: HBASE-16647.patch

> hbck should do offline reference repair before online repair
> 
>
> Key: HBASE-16647
> URL: https://issues.apache.org/jira/browse/HBASE-16647
> Project: HBase
>  Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
> Attachments: HBASE-16647.patch
>
>
> {noformat}
> hbck
> -fixReferenceFiles  Try to offline lingering reference store files
> Metadata Repair shortcuts
> -repairShortcut for -fixAssignments -fixMeta -fixHdfsHoles 
> -fixHdfsOrphans -fixHdfsOverlaps -fixVersionFile -sidelineBigOverlaps 
> -fixReferenceFiles -fixTableLocks -fixOrphanedTableZnodes
> {noformat}
> Bad reference files prevent the region from coming online.
> If used in the shortcut combination, the reference files should be fixed 
> before other online fix.
> I have seen repeated '-repair' did not work because bad reference files 
> failed regions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497687#comment-15497687
 ] 

Ted Yu commented on HBASE-16646:


In the map:
bq. Map map

key is family.
value is List of Paths each of which contains family.

> Enhance LoadIncrementalHFiles to accept store file paths as input
> -
>
> Key: HBASE-16646
> URL: https://issues.apache.org/jira/browse/HBASE-16646
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16646.v0.txt, 16646.v1.txt
>
>
> Currently LoadIncrementalHFiles takes the directory (output path) as input 
> parameter.
> In some scenarios (incremental restore of bulk loaded hfiles), the List of 
> paths to hfiles is known.
> LoadIncrementalHFiles can take the List as input parameter and proceed with 
> loading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16345) RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497689#comment-15497689
 ] 

huaxiang sun commented on HBASE-16345:
--

Hi [~enis], any comments for v5 patch? Thanks.

> RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer 
> Exceptions
> --
>
> Key: HBASE-16345
> URL: https://issues.apache.org/jira/browse/HBASE-16345
> Project: HBase
>  Issue Type: Bug
>  Components: Client
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-16345-v001.patch, HBASE-16345.master.001.patch, 
> HBASE-16345.master.002.patch, HBASE-16345.master.003.patch, 
> HBASE-16345.master.004.patch, HBASE-16345.master.005.patch, 
> HBASE-16345.master.005.patch
>
>
> Update for the description. Debugged more at this front based on the comments 
> from Enis. 
> The cause is that for the primary replica, if its retry is exhausted too 
> fast, f.get() [1] returns ExecutionException. This Exception needs to be 
> ignored and continue with the replicas.
> The other issue is that after adding calls for the replicas, if the first 
> completed task gets ExecutionException (due to the retry exhausted), it 
> throws the exception to the client[2].
> In this case, it needs to loop through these tasks, waiting for the success 
> one. If no one succeeds, throw exception.
> Similar for the scan as well
> [1] 
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L197
> [2] 
> https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L219



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16578) Mob data loss after mob compaction and normal compcation

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497684#comment-15497684
 ] 

huaxiang sun commented on HBASE-16578:
--

Sorry [~jingcheng...@intel.com]. I do not have bandwidth to work on this issue 
recently. Could you go ahead to post the patch? Thanks.

> Mob data loss after mob compaction and normal compcation
> 
>
> Key: HBASE-16578
> URL: https://issues.apache.org/jira/browse/HBASE-16578
> Project: HBase
>  Issue Type: Bug
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: TestMobCompaction.java, TestMobCompaction.java
>
>
> I have a unittest case, which could explore the mob data loss issue. The root 
> cause is that with the following line:
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java#L625
> It will make the old mob reference cell win during compaction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16349) TestClusterId may hang during cluster shutdown

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-16349:
---
   Resolution: Fixed
 Assignee: Ted Yu
 Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
   2.0.0
   Status: Resolved  (was: Patch Available)

The test passed in recent builds.

Will reopen if it fails in shutdown.

Thanks for the review, Heng.

> TestClusterId may hang during cluster shutdown
> --
>
> Key: HBASE-16349
> URL: https://issues.apache.org/jira/browse/HBASE-16349
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16349.branch-1.v1.txt, 16349.v1.txt
>
>
> I was running TestClusterId on branch-1 where I observed the test hang during 
> test tearDown().
> {code}
> 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(1415): Closing hbase:meta,,1.1588230740: disabling 
> compactions & flushes
> 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(1442): Updates disabled for region 
> hbase:meta,,1.1588230740
> 2016-08-03 11:36:39,600 INFO  [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(2253): Flushing 1/1 column families, memstore=232 B
> 2016-08-03 11:36:39,601 WARN  [RS_OPEN_META-cn012:49371-0.append-pool17-t1] 
> wal.FSHLog$RingBufferEventHandler(1900): Append sequenceId=8, requesting roll 
> of WAL
> java.io.IOException: All datanodes 
> DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK]
>  are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
> 2016-08-03 11:36:39,602 FATAL [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegionServer(2085): ABORTING region server 
> cn012.l42scl.hortonworks.com,49371,1470249187586: Unrecoverable   
> exception while closing region hbase:meta,,1.1588230740, still finishing close
> org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append 
> sequenceId=8, requesting roll of WAL
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1902)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1754)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1676)
>   at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: All datanodes 
> DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK]
>  are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
> 2016-08-03 11:36:39,603 FATAL [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegionServer(2093): RegionServer abort: loaded coprocessors 
> are: [org.apache.hadoop.hbase.coprocessor.   MultiRowMutationEndpoint]
> {code}
> This led to rst.join() hanging:
> {code}
> "RS:0;cn012:49371" #648 prio=5 os_prio=0 tid=0x7fdab24b5000 nid=0x621a 
> waiting on condition [0x7fd785fe]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
>   at java.lang.Thread.sleep(Native Method)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1326)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1312)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1082)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16349) TestClusterId may hang during cluster shutdown

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-16349:
---
Issue Type: Test  (was: Bug)

> TestClusterId may hang during cluster shutdown
> --
>
> Key: HBASE-16349
> URL: https://issues.apache.org/jira/browse/HBASE-16349
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16349.branch-1.v1.txt, 16349.v1.txt
>
>
> I was running TestClusterId on branch-1 where I observed the test hang during 
> test tearDown().
> {code}
> 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(1415): Closing hbase:meta,,1.1588230740: disabling 
> compactions & flushes
> 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(1442): Updates disabled for region 
> hbase:meta,,1.1588230740
> 2016-08-03 11:36:39,600 INFO  [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(2253): Flushing 1/1 column families, memstore=232 B
> 2016-08-03 11:36:39,601 WARN  [RS_OPEN_META-cn012:49371-0.append-pool17-t1] 
> wal.FSHLog$RingBufferEventHandler(1900): Append sequenceId=8, requesting roll 
> of WAL
> java.io.IOException: All datanodes 
> DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK]
>  are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
> 2016-08-03 11:36:39,602 FATAL [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegionServer(2085): ABORTING region server 
> cn012.l42scl.hortonworks.com,49371,1470249187586: Unrecoverable   
> exception while closing region hbase:meta,,1.1588230740, still finishing close
> org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append 
> sequenceId=8, requesting roll of WAL
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1902)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1754)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1676)
>   at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: All datanodes 
> DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK]
>  are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
> 2016-08-03 11:36:39,603 FATAL [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegionServer(2093): RegionServer abort: loaded coprocessors 
> are: [org.apache.hadoop.hbase.coprocessor.   MultiRowMutationEndpoint]
> {code}
> This led to rst.join() hanging:
> {code}
> "RS:0;cn012:49371" #648 prio=5 os_prio=0 tid=0x7fdab24b5000 nid=0x621a 
> waiting on condition [0x7fd785fe]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
>   at java.lang.Thread.sleep(Native Method)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1326)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1312)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1082)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497446#comment-15497446
 ] 

Hadoop QA commented on HBASE-16598:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 5s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 18m 
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 
33s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 37s 
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 45s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 16s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 16s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 16s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 18m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 1s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s 
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 39s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 6s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s 
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 57s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 57s {color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 
23s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | 

[jira] [Updated] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-16646:
---
Attachment: 16646.v0.txt

Patch v0 illustrates the tentative change.

List of hfiles may be retrieved from BulkLoadDescriptor.

> Enhance LoadIncrementalHFiles to accept store file paths as input
> -
>
> Key: HBASE-16646
> URL: https://issues.apache.org/jira/browse/HBASE-16646
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16646.v0.txt
>
>
> Currently LoadIncrementalHFiles takes the directory (output path) as input 
> parameter.
> In some scenarios (incremental restore of bulk loaded hfiles), the List of 
> paths to hfiles is known.
> LoadIncrementalHFiles can take the List as input parameter and proceed with 
> loading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16646) Enhance LoadIncrementalHFiles to accept store file paths as input

[Ted Yu (JIRA)]

Ted Yu created HBASE-16646:
--

 Summary: Enhance LoadIncrementalHFiles to accept store file paths 
as input
 Key: HBASE-16646
 URL: https://issues.apache.org/jira/browse/HBASE-16646
 Project: HBase
  Issue Type: Improvement
Reporter: Ted Yu
Assignee: Ted Yu


Currently LoadIncrementalHFiles takes the directory (output path) as input 
parameter.

In some scenarios (incremental restore of bulk loaded hfiles), the List of 
paths to hfiles is known.

LoadIncrementalHFiles can take the List as input parameter and proceed with 
loading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497422#comment-15497422
 ] 

Enis Soztutar commented on HBASE-16257:
---

Thanks Jerry for the latest patch. Sorry, but one last item may need to be 
addressed. Otherwise, everything looks good. The HFileReplicator now uses the 
bulk load staging dir as a path under root. This will become a problem if the 
source cluster is upgraded first before the sink cluster. In case of cyclic 
replication, it will always be the case that you can have 2.x -> 1.x 
replication. If that is the case, the staging dir will not be there under the 
1.x root, which will then case the HFileReplicator to fail. Maybe we can do a 
simple exists check on the remote FS to see whether we can use the new 
hbase.root.dir/staging or we should fall back to using the value from 
configuration. 

> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: HBASE-16257-v1.patch, HBASE-16257-v2.patch, 
> HBASE-16257-v3.patch, HBASE-16257-v4.patch
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then 
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
>   + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir.  We are bringing 
> secure bulkload to the core. It makes sense to bring it under core control as 
> well, instead of an optional property.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16587) Procedure v2 - Cleanup suspended proc execution

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497334#comment-15497334
 ] 

Hadoop QA commented on HBASE-16587:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
24m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 53s 
{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 29s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
27s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 116m 1s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.master.TestMasterFailover |
|   | org.apache.hadoop.hbase.master.TestTableLockManager |
|   | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
|   | org.apache.hadoop.hbase.io.hfile.TestScannerFromBucketCache |
|   | org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite |
|   | org.apache.hadoop.hbase.master.TestMetaShutdownHandler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828883/HBASE-16587-v1.patch |
| JIRA Issue | HBASE-16587 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  xml  |
| uname | Linux b26bf3a4ced4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497321#comment-15497321
 ] 

Enis Soztutar commented on HBASE-16598:
---

Great cleanup. +1. 

> Enable zookeeper useMulti always and clean up in HBase code
> ---
>
> Key: HBASE-16598
> URL: https://issues.apache.org/jira/browse/HBASE-16598
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, 
> HBASE-16598-v4.patch, HBASE-16598.patch
>
>
> We have zookeeper useMulti default true since HBase 1.0.0
> And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0"
> The benefit of useMulti is obvious. 
> Let's clean up the code to remove useMulti false case so that we don't have 
> to continue with the hassle of maintaining two version of the code (e.g. in 
> replication) .  No go back to pre 3.4.x Zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497219#comment-15497219
 ] 

Hadoop QA commented on HBASE-16645:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
32m 15s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 116m 54s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
23s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 167m 6s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| Timed out junit tests | 
org.apache.hadoop.hbase.security.access.TestCellACLWithMultipleVersions |
|   | org.apache.hadoop.hbase.security.token.TestDelegationTokenWithEncryption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.1 Server=1.12.1 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828855/HBASE-16645.v0.patch |
| JIRA Issue | HBASE-16645 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 5ea5176f7bf8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 2cf8907 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/3576/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/3576/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/3576/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/3576/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was 

[jira] [Commented] (HBASE-16586) Procedure v2 - Cleanup sched wait/lock semantic

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497206#comment-15497206
 ] 

Hudson commented on HBASE-16586:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1614 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1614/])
HBASE-16586 Procedure v2 - Cleanup sched wait/lock semantic (matteo.bertozzi: 
rev b6b72361b68634f15f8cf83738d89147633ac378)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestMasterProcedureScheduler.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureScheduler.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DispatchMergingRegionsProcedure.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureEnv.java


> Procedure v2 - Cleanup sched wait/lock semantic
> ---
>
> Key: HBASE-16586
> URL: https://issues.apache.org/jira/browse/HBASE-16586
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-16586-v0.patch, HBASE-16586-v1.patch, 
> HBASE-16586-v2.patch, HBASE-16586-v3.patch
>
>
> For some reason waitEvent() and waitRegion() had a mismatching return value. 
> unity the wait semantic in being: return true we wait, false we don't wait.
> procedures using hasLock = waitRegion() should change to hasLock = 
> !waitRegion(). at the moment we have only DispatchMergingRegionsProcedure 
> using it (in master).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497205#comment-15497205
 ] 

Hudson commented on HBASE-16631:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1614 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1614/])
HBASE-16631 Extract AsyncRequestFuture related code from AsyncProcess 
(chenheng: rev 2cf8907db53b84a0118acc1edd1dfb9b37abe8b7)
* (add) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/BatchErrors.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java
* (add) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFuture.java
* (add) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRequestFutureImpl.java
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestReplicasClient.java
* (edit) 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java


> Extract AsyncRequestFuture related code from AsyncProcess
> -
>
> Key: HBASE-16631
> URL: https://issues.apache.org/jira/browse/HBASE-16631
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, 
> HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch
>
>
> Now, AsyncProcess class is too large (over 2000+ lines),  and there are so 
> many sub classes in it.  
> AsyncRequestFutureImpl is the biggest subclass in AP,  we could extract it 
> out from AP to reduce the AP size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16447) Replication by namespaces config in peer

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497126#comment-15497126
 ] 

Enis Soztutar commented on HBASE-16447:
---

Thanks Guanghao Zhang for the updated patch. I've committed this to master. 
branch-1 has a lot of conflicts, do you mind attaching a patch for branch-1 as 
well. 

> Replication by namespaces config in peer
> 
>
> Key: HBASE-16447
> URL: https://issues.apache.org/jira/browse/HBASE-16447
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16447-v1.patch, HBASE-16447-v2.patch, 
> HBASE-16447-v3.patch, HBASE-16447-v4.patch, HBASE-16447-v5.patch, 
> HBASE-16447-v5.patch
>
>
> Now we only config table cfs in peer. But in our production cluster, there 
> are a dozen of namespace and every namespace has dozens of tables. It was 
> complicated to config all table cfs in peer. For some namespace, it need 
> replication all tables to other slave cluster. It will be easy to config if 
> we support replication by namespace. Suggestions and discussions are welcomed.
> Review board: https://reviews.apache.org/r/51521/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16447) Replication by namespaces config in peer

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-16447:
--
Fix Version/s: 1.4.0
   2.0.0

> Replication by namespaces config in peer
> 
>
> Key: HBASE-16447
> URL: https://issues.apache.org/jira/browse/HBASE-16447
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16447-v1.patch, HBASE-16447-v2.patch, 
> HBASE-16447-v3.patch, HBASE-16447-v4.patch, HBASE-16447-v5.patch, 
> HBASE-16447-v5.patch
>
>
> Now we only config table cfs in peer. But in our production cluster, there 
> are a dozen of namespace and every namespace has dozens of tables. It was 
> complicated to config all table cfs in peer. For some namespace, it need 
> replication all tables to other slave cluster. It will be easy to config if 
> we support replication by namespace. Suggestions and discussions are welcomed.
> Review board: https://reviews.apache.org/r/51521/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497085#comment-15497085
 ] 

Ted Yu commented on HBASE-16645:


[~anastas]:
Can you take a look ?

> Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
> --
>
> Key: HBASE-16645
> URL: https://issues.apache.org/jira/browse/HBASE-16645
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16645.v0.patch
>
>
> Two reasons are shown below:
> 1) CellFlatMap#find doesn’t consider desc order array
> 2) CellFlatMap#getValidIndex return the wrong upper bound



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16587) Procedure v2 - Cleanup suspended proc execution

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-16587:

Status: Patch Available  (was: Open)

> Procedure v2 - Cleanup suspended proc execution
> ---
>
> Key: HBASE-16587
> URL: https://issues.apache.org/jira/browse/HBASE-16587
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-16587-v0.patch, HBASE-16587-v1.patch
>
>
> for procedures like the assignment or the lock one we need to be able to hold 
> on locks while suspended. At the moment the way to do that is up to the proc 
> implementation. This patch moves the logic to the base Procedure and 
> ProcedureExecutor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16587) Procedure v2 - Cleanup suspended proc execution

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-16587:

Attachment: HBASE-16587-v1.patch

> Procedure v2 - Cleanup suspended proc execution
> ---
>
> Key: HBASE-16587
> URL: https://issues.apache.org/jira/browse/HBASE-16587
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-16587-v0.patch, HBASE-16587-v1.patch
>
>
> for procedures like the assignment or the lock one we need to be able to hold 
> on locks while suspended. At the moment the way to do that is up to the proc 
> implementation. This patch moves the logic to the base Procedure and 
> ProcedureExecutor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497039#comment-15497039
 ] 

stack commented on HBASE-16644:
---

bq. Regardless of how such a file was written, still if we can read some file 
on 1.2 and we can't on 1.3 that is a critical regression on the read path IMO.

Agree.

> Errors when reading legit HFile' Trailer on branch 1.3
> --
>
> Key: HBASE-16644
> URL: https://issues.apache.org/jira/browse/HBASE-16644
> Project: HBase
>  Issue Type: Bug
>  Components: HFile
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: HBASE-16644.branch-1.3.patch
>
>
> There seems to be a regression in branch 1.3 where we can't read HFile 
> trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on 
> some HFiles that could be successfully read on 1.2.
> I've seen this error manifesting in two ways so far.
> {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: 
> Problem reading HFile Trailer from file  
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>   ... 6 more
> Caused by: java.io.IOException: Invalid HFile block magic: 
> \x00\x00\x04\x00\x00\x00\x00\x00
>   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
>   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
> {code}
> and second
> {code}
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
> reading HFile Trailer from file 
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>   ... 6 more
> Caused by: java.io.IOException: Premature EOF from inputStream (read returned 
> -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully 
> read 1072
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712)
>   at 
> 

[jira] [Commented] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3

[Mikhail Antonov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497029#comment-15497029
 ] 

Mikhail Antonov commented on HBASE-16644:
-

[~stack] thanks for quick reply! So far my thoughts on it:

 - that seems to only happen under specific conditions, since I've only seen a 
handful of such cases. Still looking at what's special about those files. So 
far it *seems" that it only happens on specific blocks, the code path in 
HFileReaderV2 constructor when we read meta index (data index was read fine).
 - Debugging and reproducing is kind of painful as I've only seen that on real 
files and don't yet have cooked-up test file that can demonstrate this effect 
on.
 - Regardless of how such a file was written, still if we can read some file on 
1.2 and we can't on 1.3 that is a critical regression on the read path IMO.

> Errors when reading legit HFile' Trailer on branch 1.3
> --
>
> Key: HBASE-16644
> URL: https://issues.apache.org/jira/browse/HBASE-16644
> Project: HBase
>  Issue Type: Bug
>  Components: HFile
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: HBASE-16644.branch-1.3.patch
>
>
> There seems to be a regression in branch 1.3 where we can't read HFile 
> trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on 
> some HFiles that could be successfully read on 1.2.
> I've seen this error manifesting in two ways so far.
> {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: 
> Problem reading HFile Trailer from file  
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>   ... 6 more
> Caused by: java.io.IOException: Invalid HFile block magic: 
> \x00\x00\x04\x00\x00\x00\x00\x00
>   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
>   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
> {code}
> and second
> {code}
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
> reading HFile Trailer from file 
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at 

[jira] [Commented] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497015#comment-15497015
 ] 

Matteo Bertozzi commented on HBASE-16544:
-

+1

> Remove or Clarify  'Using Amazon S3 Storage' section in the reference guide
> ---
>
> Key: HBASE-16544
> URL: https://issues.apache.org/jira/browse/HBASE-16544
> Project: HBase
>  Issue Type: Bug
>  Components: documentation, Filesystem Integration
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-16544-V1.patch, HBASE-16544-addendum.patch
>
>
> reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration
> (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I 
> think this title means that we can use S3 storage to replace HDFS, I really 
> tried this :(,   always give me errors and hbase even can not start, see 
> error mentioned in jira HBASE-11045.
> (2) And the details in this section are more about deploying HBase on Amazon 
> EC2 cluster, which has nothing to do 'using Amazon S3 storage'
> (3) In all, I think we need to remove this section, or at least clarify this 
> section if someone fully test HBase on S3.  see HBASE-15646 for more details 
> about this doc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16544) Remove or Clarify 'Using Amazon S3 Storage' section in the reference guide

[Yi Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liang updated HBASE-16544:
-
Attachment: HBASE-16544-addendum.patch

forget to remove links to the removed section 'using amazon s3'. Provide 
addendum to remove it.

> Remove or Clarify  'Using Amazon S3 Storage' section in the reference guide
> ---
>
> Key: HBASE-16544
> URL: https://issues.apache.org/jira/browse/HBASE-16544
> Project: HBase
>  Issue Type: Bug
>  Components: documentation, Filesystem Integration
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
> Fix For: 2.0.0
>
> Attachments: HBASE-16544-V1.patch, HBASE-16544-addendum.patch
>
>
> reference guide at https://hbase.apache.org/book.html#amazon_s3_configuration
> (1) the title 'Using Amazon S3 Storage' is confusing.From my point of view, I 
> think this title means that we can use S3 storage to replace HDFS, I really 
> tried this :(,   always give me errors and hbase even can not start, see 
> error mentioned in jira HBASE-11045.
> (2) And the details in this section are more about deploying HBase on Amazon 
> EC2 cluster, which has nothing to do 'using Amazon S3 storage'
> (3) In all, I think we need to remove this section, or at least clarify this 
> section if someone fully test HBase on S3.  see HBASE-15646 for more details 
> about this doc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496953#comment-15496953
 ] 

Matteo Bertozzi commented on HBASE-16598:
-

looks good to me, +1

> Enable zookeeper useMulti always and clean up in HBase code
> ---
>
> Key: HBASE-16598
> URL: https://issues.apache.org/jira/browse/HBASE-16598
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, 
> HBASE-16598-v4.patch, HBASE-16598.patch
>
>
> We have zookeeper useMulti default true since HBase 1.0.0
> And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0"
> The benefit of useMulti is obvious. 
> Let's clean up the code to remove useMulti false case so that we don't have 
> to continue with the hassle of maintaining two version of the code (e.g. in 
> replication) .  No go back to pre 3.4.x Zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496934#comment-15496934
 ] 

Jerry He commented on HBASE-16598:
--

[~mbertozzi] [~ashish singhi]  Are you good with the patch v4?

> Enable zookeeper useMulti always and clean up in HBase code
> ---
>
> Key: HBASE-16598
> URL: https://issues.apache.org/jira/browse/HBASE-16598
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, 
> HBASE-16598-v4.patch, HBASE-16598.patch
>
>
> We have zookeeper useMulti default true since HBase 1.0.0
> And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0"
> The benefit of useMulti is obvious. 
> Let's clean up the code to remove useMulti false case so that we don't have 
> to continue with the hassle of maintaining two version of the code (e.g. in 
> replication) .  No go back to pre 3.4.x Zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496931#comment-15496931
 ] 

Jerry He commented on HBASE-16598:
--

All the failed/timeout tests passed locally (multiple times)
{noformat}
Running org.apache.hadoop.hbase.TestMultiVersions
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.537 sec - in 
org.apache.hadoop.hbase.TestMultiVersions
Running 
org.apache.hadoop.hbase.security.visibility.TestWithDisabledAuthorization
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.159 sec - in 
org.apache.hadoop.hbase.security.visibility.TestWithDisabledAuthorization
Running org.apache.hadoop.hbase.security.access.TestAccessController
Tests run: 66, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.392 sec - 
in org.apache.hadoop.hbase.security.access.TestAccessController
Running org.apache.hadoop.hbase.security.access.TestWithDisabledAuthorization
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.62 sec - in 
org.apache.hadoop.hbase.security.access.TestWithDisabledAuthorization
Running org.apache.hadoop.hbase.security.access.TestNamespaceCommands
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.432 sec - in 
org.apache.hadoop.hbase.security.access.TestNamespaceCommands
Running org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 272.686 sec - 
in org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot
Running org.apache.hadoop.hbase.snapshot.TestExportSnapshot
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 273.527 sec - 
in org.apache.hadoop.hbase.snapshot.TestExportSnapshot

Results :

Tests run: 102, Failures: 0, Errors: 0, Skipped: 0
{noformat}

> Enable zookeeper useMulti always and clean up in HBase code
> ---
>
> Key: HBASE-16598
> URL: https://issues.apache.org/jira/browse/HBASE-16598
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, 
> HBASE-16598-v4.patch, HBASE-16598.patch
>
>
> We have zookeeper useMulti default true since HBase 1.0.0
> And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0"
> The benefit of useMulti is obvious. 
> Let's clean up the code to remove useMulti false case so that we don't have 
> to continue with the hassle of maintaining two version of the code (e.g. in 
> replication) .  No go back to pre 3.4.x Zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16349) TestClusterId may hang during cluster shutdown

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496909#comment-15496909
 ] 

Hudson commented on HBASE-16349:


FAILURE: Integrated in Jenkins build HBase-1.4 #418 (See 
[https://builds.apache.org/job/HBase-1.4/418/])
HBASE-16349 TestClusterId may hang during cluster shutdown (tedyu: rev 
591cc4cfb8c908556eb6e64811aa7b46fb8bbb7a)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestClusterId.java


> TestClusterId may hang during cluster shutdown
> --
>
> Key: HBASE-16349
> URL: https://issues.apache.org/jira/browse/HBASE-16349
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16349.branch-1.v1.txt, 16349.v1.txt
>
>
> I was running TestClusterId on branch-1 where I observed the test hang during 
> test tearDown().
> {code}
> 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(1415): Closing hbase:meta,,1.1588230740: disabling 
> compactions & flushes
> 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(1442): Updates disabled for region 
> hbase:meta,,1.1588230740
> 2016-08-03 11:36:39,600 INFO  [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(2253): Flushing 1/1 column families, memstore=232 B
> 2016-08-03 11:36:39,601 WARN  [RS_OPEN_META-cn012:49371-0.append-pool17-t1] 
> wal.FSHLog$RingBufferEventHandler(1900): Append sequenceId=8, requesting roll 
> of WAL
> java.io.IOException: All datanodes 
> DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK]
>  are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
> 2016-08-03 11:36:39,602 FATAL [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegionServer(2085): ABORTING region server 
> cn012.l42scl.hortonworks.com,49371,1470249187586: Unrecoverable   
> exception while closing region hbase:meta,,1.1588230740, still finishing close
> org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append 
> sequenceId=8, requesting roll of WAL
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1902)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1754)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1676)
>   at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: All datanodes 
> DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK]
>  are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
> 2016-08-03 11:36:39,603 FATAL [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegionServer(2093): RegionServer abort: loaded coprocessors 
> are: [org.apache.hadoop.hbase.coprocessor.   MultiRowMutationEndpoint]
> {code}
> This led to rst.join() hanging:
> {code}
> "RS:0;cn012:49371" #648 prio=5 os_prio=0 tid=0x7fdab24b5000 nid=0x621a 
> waiting on condition [0x7fd785fe]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
>   at java.lang.Thread.sleep(Native Method)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1326)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1312)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1082)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496911#comment-15496911
 ] 

Jerry He commented on HBASE-15523:
--

ok. It is hard to make all happy :-(  One wants to control it via hbase-env, 
and others wants to control via a visible command.

> enhance hbase-daemon.sh to enable autorestart.
> --
>
> Key: HBASE-15523
> URL: https://issues.apache.org/jira/browse/HBASE-15523
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Attachments: HBASE-15523.patch
>
>
> enhance hbase-daemon.sh to enable autorestart.
> component(like master, region server) will auto-start when terminated/killed 
> abnormally if 
>(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. 
>  export HBASE_AUTORESTART=true
>   (b) Then add the following 3 simple lines(59-61) to  
> /bin/hbase-daemon.sh
>  
>  51 # get arguments
>  52 startStop=$1
>  53 shift
>  54
>  55 command=$1
>  56 shift
>  57
>  58 #make sure the auto-restart are default settings
>  59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; 
> then 
>  60   startStop="autorestart" 
>  61 fi



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496891#comment-15496891
 ] 

Nick Dimiduk commented on HBASE-15523:
--

IMHO adding an optional param makes more sense than "hiding" the behavior 
behind an ENV variable. Would be much easier to diagnose/debug when shit hits 
the fan in the middle of the night when the feature is exposed right there in 
the launch command.

> enhance hbase-daemon.sh to enable autorestart.
> --
>
> Key: HBASE-15523
> URL: https://issues.apache.org/jira/browse/HBASE-15523
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Attachments: HBASE-15523.patch
>
>
> enhance hbase-daemon.sh to enable autorestart.
> component(like master, region server) will auto-start when terminated/killed 
> abnormally if 
>(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. 
>  export HBASE_AUTORESTART=true
>   (b) Then add the following 3 simple lines(59-61) to  
> /bin/hbase-daemon.sh
>  
>  51 # get arguments
>  52 startStop=$1
>  53 shift
>  54
>  55 command=$1
>  56 shift
>  57
>  58 #make sure the auto-restart are default settings
>  59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; 
> then 
>  60   startStop="autorestart" 
>  61 fi



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496861#comment-15496861
 ] 

Yi Liang commented on HBASE-15523:
--

I closed this one on Aug by accident, I reopen it to see if anyone have any 
other suggestion or idea of this proposal. From my point of view,  the 
autostart command is also a kind of extension for start command, we need 
somehow extend start command to support this autostart, not have a new command 
for autostart. 

> enhance hbase-daemon.sh to enable autorestart.
> --
>
> Key: HBASE-15523
> URL: https://issues.apache.org/jira/browse/HBASE-15523
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Attachments: HBASE-15523.patch
>
>
> enhance hbase-daemon.sh to enable autorestart.
> component(like master, region server) will auto-start when terminated/killed 
> abnormally if 
>(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. 
>  export HBASE_AUTORESTART=true
>   (b) Then add the following 3 simple lines(59-61) to  
> /bin/hbase-daemon.sh
>  
>  51 # get arguments
>  52 startStop=$1
>  53 shift
>  54
>  55 command=$1
>  56 shift
>  57
>  58 #make sure the auto-restart are default settings
>  59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; 
> then 
>  60   startStop="autorestart" 
>  61 fi



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-16645:
--
Description: 
Two reasons are shown below:
1) CellFlatMap#find doesn’t consider desc order array
2) CellFlatMap#getValidIndex return the wrong upper bound

> Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
> --
>
> Key: HBASE-16645
> URL: https://issues.apache.org/jira/browse/HBASE-16645
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16645.v0.patch
>
>
> Two reasons are shown below:
> 1) CellFlatMap#find doesn’t consider desc order array
> 2) CellFlatMap#getValidIndex return the wrong upper bound



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-16645:
--
Status: Patch Available  (was: Open)

> Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
> --
>
> Key: HBASE-16645
> URL: https://issues.apache.org/jira/browse/HBASE-16645
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16645.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-16645:
--
Attachment: HBASE-16645.v0.patch

> Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap
> --
>
> Key: HBASE-16645
> URL: https://issues.apache.org/jira/browse/HBASE-16645
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16645.v0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16645) Wrong range of Cells is caused by CellFlatMap#tailMap, headMap, and SubMap

[ChiaPing Tsai (JIRA)]

ChiaPing Tsai created HBASE-16645:
-

 Summary: Wrong range of Cells is caused by CellFlatMap#tailMap, 
headMap, and SubMap
 Key: HBASE-16645
 URL: https://issues.apache.org/jira/browse/HBASE-16645
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.0.0
Reporter: ChiaPing Tsai
Priority: Minor
 Fix For: 2.0.0






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.

[Dima Spivak (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496793#comment-15496793
 ] 

Dima Spivak commented on HBASE-15523:
-

Moving this to a variable (that would ostensibly need to be set on every 
machine in a cluster) moves the onus from the script to the user and also means 
you couldn't grep your cluster's restart behavior via looking at the process 
tree. For those reasons, I'm not a fan of this proposal (and I'm still not 
clear on why it was reopened after being closed for six months).

> enhance hbase-daemon.sh to enable autorestart.
> --
>
> Key: HBASE-15523
> URL: https://issues.apache.org/jira/browse/HBASE-15523
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Attachments: HBASE-15523.patch
>
>
> enhance hbase-daemon.sh to enable autorestart.
> component(like master, region server) will auto-start when terminated/killed 
> abnormally if 
>(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. 
>  export HBASE_AUTORESTART=true
>   (b) Then add the following 3 simple lines(59-61) to  
> /bin/hbase-daemon.sh
>  
>  51 # get arguments
>  52 startStop=$1
>  53 shift
>  54
>  55 command=$1
>  56 shift
>  57
>  58 #make sure the auto-restart are default settings
>  59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; 
> then 
>  60   startStop="autorestart" 
>  61 fi



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-16631:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

> Extract AsyncRequestFuture related code from AsyncProcess
> -
>
> Key: HBASE-16631
> URL: https://issues.apache.org/jira/browse/HBASE-16631
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, 
> HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch
>
>
> Now, AsyncProcess class is too large (over 2000+ lines),  and there are so 
> many sub classes in it.  
> AsyncRequestFutureImpl is the biggest subclass in AP,  we could extract it 
> out from AP to reduce the AP size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496770#comment-15496770
 ] 

Heng Chen commented on HBASE-16631:
---

Timeout test case could pass locally,  relates testcase such as 
TestAsyncProcess,  TestFromClientSideXXX has passed.
will commit it to master

> Extract AsyncRequestFuture related code from AsyncProcess
> -
>
> Key: HBASE-16631
> URL: https://issues.apache.org/jira/browse/HBASE-16631
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, 
> HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch
>
>
> Now, AsyncProcess class is too large (over 2000+ lines),  and there are so 
> many sub classes in it.  
> AsyncRequestFutureImpl is the biggest subclass in AP,  we could extract it 
> out from AP to reduce the AP size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496759#comment-15496759
 ] 

Jerry He commented on HBASE-15523:
--

Yes. autorestart is supported in hbase-daemon.sh.
hbase-daemon.sh  start|stop|restart|autorestart

I think the argument is that 'autorestart' is behavior option or extension for 
the operation 'start'.  It is better and more flexible to control this behavior 
option via other means, instead of having a separate command.
In the Ambari case, even it were going to be built into Ambari, for a user to 
be able to control and switch dynamically on the Ambari console, it probably 
still need a dynamic switch on the console, and either 'start' or 'autorestart' 
is invoked based on the switch.
Then why not have this dynamic switch in HBase?
We can introduce this env property, and deprecate the ''autorestart' command.

My two cents.

> enhance hbase-daemon.sh to enable autorestart.
> --
>
> Key: HBASE-15523
> URL: https://issues.apache.org/jira/browse/HBASE-15523
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Attachments: HBASE-15523.patch
>
>
> enhance hbase-daemon.sh to enable autorestart.
> component(like master, region server) will auto-start when terminated/killed 
> abnormally if 
>(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. 
>  export HBASE_AUTORESTART=true
>   (b) Then add the following 3 simple lines(59-61) to  
> /bin/hbase-daemon.sh
>  
>  51 # get arguments
>  52 startStop=$1
>  53 shift
>  54
>  55 command=$1
>  56 shift
>  57
>  58 #make sure the auto-restart are default settings
>  59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; 
> then 
>  60   startStop="autorestart" 
>  61 fi



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-16598:
-
Attachment: HBASE-16598-v4.patch

> Enable zookeeper useMulti always and clean up in HBase code
> ---
>
> Key: HBASE-16598
> URL: https://issues.apache.org/jira/browse/HBASE-16598
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, 
> HBASE-16598-v4.patch, HBASE-16598.patch
>
>
> We have zookeeper useMulti default true since HBase 1.0.0
> And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0"
> The benefit of useMulti is obvious. 
> Let's clean up the code to remove useMulti false case so that we don't have 
> to continue with the hassle of maintaining two version of the code (e.g. in 
> replication) .  No go back to pre 3.4.x Zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16598) Enable zookeeper useMulti always and clean up in HBase code

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-16598:
-
Attachment: (was: HBASE-16598-v4.patch)

> Enable zookeeper useMulti always and clean up in HBase code
> ---
>
> Key: HBASE-16598
> URL: https://issues.apache.org/jira/browse/HBASE-16598
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-16598-v2.patch, HBASE-16598-v3.patch, 
> HBASE-16598.patch
>
>
> We have zookeeper useMulti default true since HBase 1.0.0
> And in our Doc, "ZooKeeper 3.4.x is required as of HBase 1.0.0"
> The benefit of useMulti is obvious. 
> Let's clean up the code to remove useMulti false case so that we don't have 
> to continue with the hassle of maintaining two version of the code (e.g. in 
> replication) .  No go back to pre 3.4.x Zookeeper.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16636) Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting Regions Stuck in Transition

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496683#comment-15496683
 ] 

huaxiang sun commented on HBASE-16636:
--

[~harisekhon], it seems to be fixed by HBASE-14644, I did a similar test with 
the fix before. 

> Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting 
> Regions Stuck in Transition
> -
>
> Key: HBASE-16636
> URL: https://issues.apache.org/jira/browse/HBASE-16636
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 1.1.2
> Environment: HDP 2.3.2
>Reporter: Hari Sekhon
>Priority: Minor
> Attachments: Regions_in_Transition_UI.png, ritCountOverThreshold.png
>
>
> I've discovered that the Region in Transition counts are wrong in the HMaster 
> UI /jmx page.
> The /master-status page clearly shows 3 regions stuck in transition but the 
> /jmx page I was monitoring reported 0 for ritCountOverThreshold.
> {code}
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger",
> "modelerType" : "Master,sub=AssignmentManger",
> "tag.Context" : "master",
> ...
> "ritOldestAge" : 0,
> "ritCountOverThreshold" : 0,
> ...
> "ritCount" : 0,
> {code}
> I have a nagios plugin I wrote which was checking this which I've since had 
> to rewrite to parse the /master-status page instead (the code is in 
> check_hbase_regions_stuck_in_transition.py at 
> https://github.com/harisekhon/nagios-plugins).
> I'm attaching screenshots of both /master-status and /jmx to show the 
> difference in the 2 pages on the HMaster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16586) Procedure v2 - Cleanup sched wait/lock semantic

[Matteo Bertozzi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-16586:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Procedure v2 - Cleanup sched wait/lock semantic
> ---
>
> Key: HBASE-16586
> URL: https://issues.apache.org/jira/browse/HBASE-16586
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0
>
> Attachments: HBASE-16586-v0.patch, HBASE-16586-v1.patch, 
> HBASE-16586-v2.patch, HBASE-16586-v3.patch
>
>
> For some reason waitEvent() and waitRegion() had a mismatching return value. 
> unity the wait semantic in being: return true we wait, false we don't wait.
> procedures using hasLock = waitRegion() should change to hasLock = 
> !waitRegion(). at the moment we have only DispatchMergingRegionsProcedure 
> using it (in master).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16349) TestClusterId may hang during cluster shutdown

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496619#comment-15496619
 ] 

Hudson commented on HBASE-16349:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1613 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1613/])
HBASE-16349 TestClusterId may hang during cluster shutdown (tedyu: rev 
2597217ae5aa057e1931c772139ce8cc7a2b3efb)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestClusterId.java


> TestClusterId may hang during cluster shutdown
> --
>
> Key: HBASE-16349
> URL: https://issues.apache.org/jira/browse/HBASE-16349
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Attachments: 16349.branch-1.v1.txt, 16349.v1.txt
>
>
> I was running TestClusterId on branch-1 where I observed the test hang during 
> test tearDown().
> {code}
> 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(1415): Closing hbase:meta,,1.1588230740: disabling 
> compactions & flushes
> 2016-08-03 11:36:39,600 DEBUG [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(1442): Updates disabled for region 
> hbase:meta,,1.1588230740
> 2016-08-03 11:36:39,600 INFO  [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegion(2253): Flushing 1/1 column families, memstore=232 B
> 2016-08-03 11:36:39,601 WARN  [RS_OPEN_META-cn012:49371-0.append-pool17-t1] 
> wal.FSHLog$RingBufferEventHandler(1900): Append sequenceId=8, requesting roll 
> of WAL
> java.io.IOException: All datanodes 
> DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK]
>  are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
> 2016-08-03 11:36:39,602 FATAL [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegionServer(2085): ABORTING region server 
> cn012.l42scl.hortonworks.com,49371,1470249187586: Unrecoverable   
> exception while closing region hbase:meta,,1.1588230740, still finishing close
> org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append 
> sequenceId=8, requesting roll of WAL
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1902)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1754)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1676)
>   at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: All datanodes 
> DatanodeInfoWithStorage[127.0.0.1:37765,DS-9870993e-fb98-45fc-b151-708f72aa02d2,DISK]
>  are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1113)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:876)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:402)
> 2016-08-03 11:36:39,603 FATAL [RS_CLOSE_META-cn012:49371-0] 
> regionserver.HRegionServer(2093): RegionServer abort: loaded coprocessors 
> are: [org.apache.hadoop.hbase.coprocessor.   MultiRowMutationEndpoint]
> {code}
> This led to rst.join() hanging:
> {code}
> "RS:0;cn012:49371" #648 prio=5 os_prio=0 tid=0x7fdab24b5000 nid=0x621a 
> waiting on condition [0x7fd785fe]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
>   at java.lang.Thread.sleep(Native Method)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.sleep(HRegionServer.java:1326)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1312)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1082)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.

[Dima Spivak (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496591#comment-15496591
 ] 

Dima Spivak commented on HBASE-15523:
-

Not sure why this was reopened. Autorestart works and it shouldn't be up to 
HBase to fix a problem in Ambari. 

> enhance hbase-daemon.sh to enable autorestart.
> --
>
> Key: HBASE-15523
> URL: https://issues.apache.org/jira/browse/HBASE-15523
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Attachments: HBASE-15523.patch
>
>
> enhance hbase-daemon.sh to enable autorestart.
> component(like master, region server) will auto-start when terminated/killed 
> abnormally if 
>(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. 
>  export HBASE_AUTORESTART=true
>   (b) Then add the following 3 simple lines(59-61) to  
> /bin/hbase-daemon.sh
>  
>  51 # get arguments
>  52 startStop=$1
>  53 shift
>  54
>  55 command=$1
>  56 shift
>  57
>  58 #make sure the auto-restart are default settings
>  59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; 
> then 
>  60   startStop="autorestart" 
>  61 fi



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15523) enhance hbase-daemon.sh to enable autorestart.

[Nick Dimiduk (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496580#comment-15496580
 ] 

Nick Dimiduk commented on HBASE-15523:
--

This sounds like the realm of system management tools -- write an init or 
upstart or whatever script for it, right? I have one in 
https://github.com/ndimiduk/cesnet-hbase/blob/hdp/templates/services/hbase-regionserver.conf.erb

Maybe whatever tools builds the RPM should also include system service scripts 
for its target platform. See also 
http://mail-archives.apache.org/mod_mbox/hbase-dev/201606.mbox/%3CCANZa=GsMsAz0R=BKzGO7ThZNHbh=HS5L=evofnb-a7_3bds...@mail.gmail.com%3E

> enhance hbase-daemon.sh to enable autorestart.
> --
>
> Key: HBASE-15523
> URL: https://issues.apache.org/jira/browse/HBASE-15523
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Attachments: HBASE-15523.patch
>
>
> enhance hbase-daemon.sh to enable autorestart.
> component(like master, region server) will auto-start when terminated/killed 
> abnormally if 
>(a) Add a new env variable $HBASE_AUTORESTART to hbase-env.sh i.e. 
>  export HBASE_AUTORESTART=true
>   (b) Then add the following 3 simple lines(59-61) to  
> /bin/hbase-daemon.sh
>  
>  51 # get arguments
>  52 startStop=$1
>  53 shift
>  54
>  55 command=$1
>  56 shift
>  57
>  58 #make sure the auto-restart are default settings
>  59 if [ "$HBASE_AUTORESTART" == "true" ] && [ "$startStop" == "start" ]; 
> then 
>  60   startStop="autorestart" 
>  61 fi



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15134) Add visibility into Flush and Compaction queues

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496565#comment-15496565
 ] 

Hadoop QA commented on HBASE-15134:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
33s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 21s 
{color} | {color:red} hbase-hadoop2-compat in master has 1 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 40s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s 
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s 
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 31s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
35s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 29s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
|   | org.apache.hadoop.hbase.client.TestScannerTimeout |
|   | 
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
|   | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
|   | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828829/HBASE-15134.master.002.patch
 |
| JIRA Issue | HBASE-15134 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 7535a27e61f8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 

[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496544#comment-15496544
 ] 

Hadoop QA commented on HBASE-16631:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
24m 41s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 0s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 26s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
24s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 119m 48s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
|   | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures 
|
|   | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828828/HBASE-16631.v2.patch |
| JIRA Issue | HBASE-16631 |
| Optional Tests |  asflicense  javac  javadoc  unit  xml  compile  findbugs  
hadoopcheck  hbaseanti  checkstyle  |
| uname | Linux db1a18b996d5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 2597217 |
| Default Java | 1.8.0_101 |
| 

[jira] [Commented] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496288#comment-15496288
 ] 

stack commented on HBASE-16631:
---

+1 on v1 (minus the pom change). Yeah, good idea making a new issue.  I started 
another run just so can see if failures are related at all (they should not be 
given you made no change).

> Extract AsyncRequestFuture related code from AsyncProcess
> -
>
> Key: HBASE-16631
> URL: https://issues.apache.org/jira/browse/HBASE-16631
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, 
> HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch
>
>
> Now, AsyncProcess class is too large (over 2000+ lines),  and there are so 
> many sub classes in it.  
> AsyncRequestFutureImpl is the biggest subclass in AP,  we could extract it 
> out from AP to reduce the AP size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15134) Add visibility into Flush and Compaction queues

[Abhishek Singh Chouhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Singh Chouhan updated HBASE-15134:
---
Attachment: HBASE-15134.master.002.patch

Fixing whitespace errors and getting another QA

> Add visibility into Flush and Compaction queues
> ---
>
> Key: HBASE-15134
> URL: https://issues.apache.org/jira/browse/HBASE-15134
> Project: HBase
>  Issue Type: New Feature
>  Components: Compaction, metrics, regionserver
>Reporter: Elliott Clark
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-15134.master.001.patch, 
> HBASE-15134.master.002.patch, HBASE-15134.patch, HBASE-15134.patch
>
>
> On busy spurts we can see regionservers start to see large queues for 
> compaction. It's really hard to tell if the server is queueing a lot of 
> compactions for the same region, lots of compactions for lots of regions, or 
> just falling behind.
> For flushes much the same. There can be flushes in queue that aren't being 
> run because of delayed flushes. There's no way to know from the metrics how 
> many flushes are for each region, how many are delayed. Etc.
> We should add either more metrics around this ( num per region, max per 
> region, min per region ) or add on a UI page that has the list of compactions 
> and flushes.
> Or both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16631) Extract AsyncRequestFuture related code from AsyncProcess

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-16631:
--
Attachment: HBASE-16631.v2.patch

Get another run in to see if the failures are related at all.

> Extract AsyncRequestFuture related code from AsyncProcess
> -
>
> Key: HBASE-16631
> URL: https://issues.apache.org/jira/browse/HBASE-16631
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Heng Chen
>Assignee: Heng Chen
> Attachments: HBASE-16631.v1.patch, HBASE-16631.v1.patch, 
> HBASE-16631.v2.patch, HBASE-16631.v2.patch, HBASE-16631.wip.patch
>
>
> Now, AsyncProcess class is too large (over 2000+ lines),  and there are so 
> many sub classes in it.  
> AsyncRequestFutureImpl is the biggest subclass in AP,  we could extract it 
> out from AP to reduce the AP size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16408) Handle On heap BucketCache size when HeapMemoryManager tunes memory

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496270#comment-15496270
 ] 

stack commented on HBASE-16408:
---

Will an on-heap bucketcache that does not make copies be faster than our 
lrublockcache? (Seems like it would be?)  I'd say we can forbid it though to 
keep things simple (and given we don't have much practice doing onheap bucket 
cache). mmap'd is considered file-based? Its not looked on as onheap?

> Handle On heap BucketCache size when HeapMemoryManager tunes memory
> ---
>
> Key: HBASE-16408
> URL: https://issues.apache.org/jira/browse/HBASE-16408
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496255#comment-15496255
 ] 

stack commented on HBASE-16644:
---

Ouch. That looks like an ugly one to debug [~mantonov]. Sorry. Yeah, made a 
mistake presuming we always checksum (how did you write files w/o a checksum? A 
mistake?). Patch is great but needs a unit test, don't you think?

> Errors when reading legit HFile' Trailer on branch 1.3
> --
>
> Key: HBASE-16644
> URL: https://issues.apache.org/jira/browse/HBASE-16644
> Project: HBase
>  Issue Type: Bug
>  Components: HFile
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: HBASE-16644.branch-1.3.patch
>
>
> There seems to be a regression in branch 1.3 where we can't read HFile 
> trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on 
> some HFiles that could be successfully read on 1.2.
> I've seen this error manifesting in two ways so far.
> {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: 
> Problem reading HFile Trailer from file  
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>   ... 6 more
> Caused by: java.io.IOException: Invalid HFile block magic: 
> \x00\x00\x04\x00\x00\x00\x00\x00
>   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
>   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
> {code}
> and second
> {code}
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
> reading HFile Trailer from file 
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>   ... 6 more
> Caused by: java.io.IOException: Premature EOF from inputStream (read returned 
> -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully 
> read 1072
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459)
>   at 
> 

[jira] [Commented] (HBASE-15134) Add visibility into Flush and Compaction queues

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496256#comment-15496256
 ] 

Hadoop QA commented on HBASE-15134:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
29s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 19s 
{color} | {color:red} hbase-hadoop2-compat in master has 1 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 58s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s 
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s 
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 7s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
35s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 123m 18s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | 
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
 |
|   | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures 
|
|   | org.apache.hadoop.hbase.master.TestTableLockManager |
|   | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828822/HBASE-15134.master.001.patch
 |
| JIRA Issue | HBASE-15134 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 543c7d6b9061 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (HBASE-15134) Add visibility into Flush and Compaction queues

[Abhishek Singh Chouhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496009#comment-15496009
 ] 

Abhishek Singh Chouhan commented on HBASE-15134:


Thanks [~busbey] will use the guidelines going forward :)
Have added maxCompactionQueueSize and corresponding flush metric, which will 
get updated every 45 seconds as per the value at that point in time. Let me 
know if this looks fine.

> Add visibility into Flush and Compaction queues
> ---
>
> Key: HBASE-15134
> URL: https://issues.apache.org/jira/browse/HBASE-15134
> Project: HBase
>  Issue Type: New Feature
>  Components: Compaction, metrics, regionserver
>Reporter: Elliott Clark
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-15134.master.001.patch, HBASE-15134.patch, 
> HBASE-15134.patch
>
>
> On busy spurts we can see regionservers start to see large queues for 
> compaction. It's really hard to tell if the server is queueing a lot of 
> compactions for the same region, lots of compactions for lots of regions, or 
> just falling behind.
> For flushes much the same. There can be flushes in queue that aren't being 
> run because of delayed flushes. There's no way to know from the metrics how 
> many flushes are for each region, how many are delayed. Etc.
> We should add either more metrics around this ( num per region, max per 
> region, min per region ) or add on a UI page that has the list of compactions 
> and flushes.
> Or both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15134) Add visibility into Flush and Compaction queues

[Abhishek Singh Chouhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhishek Singh Chouhan updated HBASE-15134:
---
Attachment: HBASE-15134.master.001.patch

> Add visibility into Flush and Compaction queues
> ---
>
> Key: HBASE-15134
> URL: https://issues.apache.org/jira/browse/HBASE-15134
> Project: HBase
>  Issue Type: New Feature
>  Components: Compaction, metrics, regionserver
>Reporter: Elliott Clark
>Assignee: Abhishek Singh Chouhan
> Attachments: HBASE-15134.master.001.patch, HBASE-15134.patch, 
> HBASE-15134.patch
>
>
> On busy spurts we can see regionservers start to see large queues for 
> compaction. It's really hard to tell if the server is queueing a lot of 
> compactions for the same region, lots of compactions for lots of regions, or 
> just falling behind.
> For flushes much the same. There can be flushes in queue that aren't being 
> run because of delayed flushes. There's no way to know from the metrics how 
> many flushes are for each region, how many are delayed. Etc.
> We should add either more metrics around this ( num per region, max per 
> region, min per region ) or add on a UI page that has the list of compactions 
> and flushes.
> Or both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3

[Mikhail Antonov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-16644:

Attachment: HBASE-16644.branch-1.3.patch

> Errors when reading legit HFile' Trailer on branch 1.3
> --
>
> Key: HBASE-16644
> URL: https://issues.apache.org/jira/browse/HBASE-16644
> Project: HBase
>  Issue Type: Bug
>  Components: HFile
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Critical
> Fix For: 1.3.0
>
> Attachments: HBASE-16644.branch-1.3.patch
>
>
> There seems to be a regression in branch 1.3 where we can't read HFile 
> trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on 
> some HFiles that could be successfully read on 1.2.
> I've seen this error manifesting in two ways so far.
> {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: 
> Problem reading HFile Trailer from file  
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>   ... 6 more
> Caused by: java.io.IOException: Invalid HFile block magic: 
> \x00\x00\x04\x00\x00\x00\x00\x00
>   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
>   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
> {code}
> and second
> {code}
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
> reading HFile Trailer from file 
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>   ... 6 more
> Caused by: java.io.IOException: Premature EOF from inputStream (read returned 
> -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully 
> read 1072
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
>   at 
> 

[jira] [Updated] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3

[Mikhail Antonov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov updated HBASE-16644:

Priority: Critical  (was: Major)

> Errors when reading legit HFile' Trailer on branch 1.3
> --
>
> Key: HBASE-16644
> URL: https://issues.apache.org/jira/browse/HBASE-16644
> Project: HBase
>  Issue Type: Bug
>  Components: HFile
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Critical
> Fix For: 1.3.0
>
>
> There seems to be a regression in branch 1.3 where we can't read HFile 
> trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on 
> some HFiles that could be successfully read on 1.2.
> I've seen this error manifesting in two ways so far.
> {code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: 
> Problem reading HFile Trailer from file  
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>   ... 6 more
> Caused by: java.io.IOException: Invalid HFile block magic: 
> \x00\x00\x04\x00\x00\x00\x00\x00
>   at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
>   at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
> {code}
> and second
> {code}
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
> reading HFile Trailer from file 
>   at 
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
>   at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
>   at 
> org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
>   at 
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
>   at 
> org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
>   at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
>   ... 6 more
> Caused by: java.io.IOException: Premature EOF from inputStream (read returned 
> -1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully 
> read 1072
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
>   at 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
>   

[jira] [Created] (HBASE-16644) Errors when reading legit HFile' Trailer on branch 1.3

[Mikhail Antonov (JIRA)]

Mikhail Antonov created HBASE-16644:
---

 Summary: Errors when reading legit HFile' Trailer on branch 1.3
 Key: HBASE-16644
 URL: https://issues.apache.org/jira/browse/HBASE-16644
 Project: HBase
  Issue Type: Bug
  Components: HFile
Affects Versions: 1.3.0, 1.4.0
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
 Fix For: 1.3.0


There seems to be a regression in branch 1.3 where we can't read HFile 
trailer(getting "CorruptHFileException: Problem reading HFile Trailer") on some 
HFiles that could be successfully read on 1.2.

I've seen this error manifesting in two ways so far.

{code}Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: 
Problem reading HFile Trailer from file  
at 
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
at 
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
at 
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
... 6 more
Caused by: java.io.IOException: Invalid HFile block magic: 
\x00\x00\x04\x00\x00\x00\x00\x00
at org.apache.hadoop.hbase.io.hfile.BlockType.parse(BlockType.java:155)
at org.apache.hadoop.hbase.io.hfile.BlockType.read(BlockType.java:167)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock.(HFileBlock.java:344)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1735)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156)
at 
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
{code}

and second

{code}
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
reading HFile Trailer from file 
at 
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:497)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525)
at 
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1164)
at 
org.apache.hadoop.hbase.io.HalfStoreFileReader.(HalfStoreFileReader.java:104)
at 
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:256)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528)
at 
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518)
at 
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:652)
at 
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:117)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
... 6 more
Caused by: java.io.IOException: Premature EOF from inputStream (read returned 
-1, was trying to read 10083 necessary bytes and 24 extra bytes, successfully 
read 1072
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:737)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1459)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1712)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1558)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1397)
at 
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1405)
at 
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.(HFileReaderV2.java:156)
at 
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:485)
{code}

In my case this problem was reproducible 

[jira] [Commented] (HBASE-16586) Procedure v2 - Cleanup sched wait/lock semantic

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495781#comment-15495781
 ] 

Hadoop QA commented on HBASE-16586:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 47s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 4s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
13s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 123m 4s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.token.TestGenerateDelegationToken |
| Timed out junit tests | org.apache.hadoop.hbase.constraint.TestConstraint |
|   | org.apache.hadoop.hbase.TestNamespace |
|   | 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes |
|   | 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelReplicationWithExpAsString
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828799/HBASE-16586-v3.patch |
| JIRA Issue | HBASE-16586 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 56d173564743 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / e19632a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/3571/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/3571/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/3571/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 

[jira] [Commented] (HBASE-16635) RpcClient under heavy load leaks some netty bytebuf

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495734#comment-15495734
 ] 

ramkrishna.s.vasudevan commented on HBASE-16635:


Thanks [~Apache9]
Yes in the above condition it is the admin connection that is actually closed() 
but the Bytebuf created for that connection is not released. And reading the 
code, you are right we cannot release the Bytebuf in shutdown() because there 
the connection is put back to the pool for reusing it. 

> RpcClient under heavy load leaks some netty bytebuf
> ---
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
>
> Yet to analyse the actual root cause. 
> But the case is that when we run a PE tool with 50 threads under heavy load 
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure 
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16] 
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's 
> garbage-collected. Enable advanced leak reporting to find out where the leak 
> occurred. To enable advanced leak reporting, specify the JVM option 
> '-Dio.netty.leakDetection.level=advanced' or call 
> ResourceLeakDetector.setLevel() See 
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released 
> properly by the client and hence it gets GCed automatically. Netty provides 
> tips and tricks to find the root cause. Will get back here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to inproper ref counting of segments

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-16643:
---
Description: 
In the reverse scanner case,
While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the 
backward heap, we do a 
{code}
if ((backwardHeap == null) && (forwardHeap != null)) {
forwardHeap.close();
forwardHeap = null;
// before building the heap seek for the relevant key on the scanners,
// for the heap to be built from the scanners correctly
for (KeyValueScanner scan : scanners) {
  if (toLast) {
res |= scan.seekToLastRow();
  } else {
res |= scan.backwardSeek(cell);
  }
}
{code}
forwardHeap.close(). This would internally decrement the MSLAB ref counter for 
the current active segment and snapshot segment.
When the scan is actually closed again we do close() and that will again 
decrement the count. Here chances are there that the count would go negative 
and hence the actual MSLAB closure that checks for refCount==0 will fail. Apart 
from this, when the refCount becomes 0 after the firstClose if any other thread 
requests to close the segment, then we will end up in corrupted segment because 
the segment could be put back to the MSLAB pool. 

  was:
In the reverse scanner case,
While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the 
backward heap, we do a 
{code}
if ((backwardHeap == null) && (forwardHeap != null)) {
forwardHeap.close();
forwardHeap = null;
// before building the heap seek for the relevant key on the scanners,
// for the heap to be built from the scanners correctly
for (KeyValueScanner scan : scanners) {
  if (toLast) {
res |= scan.seekToLastRow();
  } else {
res |= scan.backwardSeek(cell);
  }
}
{code}
forwardHeap.close(). This would internally decrement the MSLAB ref counter for 
the current active segment and snapshot segment.
When the scan is actually closed again we do close() and that will again 
decrement the count. Here chances are there that the count would go negative 
and hence the actual MSLAB closure that checks for refCount==0 will fail. Apart 
from this, when the refCount becomes 0 after the firstClose if any other thread 
requests to close the segment, then we will end up in corrupted segment.


> Reverse scanner heap creation may not allow MSLAB closure due to inproper ref 
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
>  Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the 
> backward heap, we do a 
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
>   if (toLast) {
> res |= scan.seekToLastRow();
>   } else {
> res |= scan.backwardSeek(cell);
>   }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter 
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again 
> decrement the count. Here chances are there that the count would go negative 
> and hence the actual MSLAB closure that checks for refCount==0 will fail. 
> Apart from this, when the refCount becomes 0 after the firstClose if any 
> other thread requests to close the segment, then we will end up in corrupted 
> segment because the segment could be put back to the MSLAB pool. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495687#comment-15495687
 ] 

ramkrishna.s.vasudevan commented on HBASE-15871:


I think for this JIRA we can commit it as is and then later in 
https://issues.apache.org/jira/browse/HBASE-16643 we can handle the close() 
properly. Without which adding a 'closed' flag is going to be logically wrong 
because once we set closed flag it cannot be again used. So we need to clearly 
check and fix that part.

> Memstore flush doesn't finish because of backwardseek() in memstore scanner.
> 
>
> Key: HBASE-15871
> URL: https://issues.apache.org/jira/browse/HBASE-15871
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 1.1.2
>Reporter: Jeongdae Kim
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15871-branch-1.patch, 
> HBASE-15871.branch-1.1.001.patch, HBASE-15871.branch-1.1.002.patch, 
> HBASE-15871.branch-1.1.003.patch, HBASE-15871.patch, HBASE-15871_1.patch, 
> HBASE-15871_1.patch, HBASE-15871_2.patch, HBASE-15871_3.patch, 
> HBASE-15871_4.patch, memstore_backwardSeek().PNG
>
>
> Sometimes in our production hbase cluster, it takes a long time to finish 
> memstore flush.( for about more than 30 minutes)
> the reason is that a memstore flusher thread calls 
> StoreScanner.updateReaders(), waits for acquiring a lock that store scanner 
> holds in StoreScanner.next() and backwardseek() in memstore scanner runs for 
> a long time.
> I think that this condition could occur in reverse scan by the following 
> process.
> 1) create a reversed store scanner by requesting a reverse scan.
> 2) flush a memstore in the same HStore.
> 3) puts a lot of cells in memstore and memstore is almost full.
> 4) call the reverse scanner.next() and re-create all scanners in this store 
> because all scanners was already closed by 2)'s flush() and backwardseek() 
> with store's lastTop for all new scanners.
> 5) in this status, memstore is almost full by 2) and all cells in memstore 
> have sequenceID greater than this scanner's readPoint because of 2)'s 
> flush(). this condition causes searching all cells in memstore, and 
> seekToPreviousRow() repeatly seach cells that are already searched if a row 
> has one column. (described this in more detail in a attached file.)
> 6) flush a memstore again in the same HStore, and wait until 4-5) process 
> finished, to update store files in the same HStore after flusing.
> I searched HBase jira. and found a similar issue. (HBASE-14497) but, 
> HBASE-14497's fix can't solve this issue because that fix just changed 
> recursive call to loop.(and already applied to our HBase version)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to inproper ref counting of segments

[ramkrishna.s.vasudevan (JIRA)]

ramkrishna.s.vasudevan created HBASE-16643:
--

 Summary: Reverse scanner heap creation may not allow MSLAB closure 
due to inproper ref counting of segments
 Key: HBASE-16643
 URL: https://issues.apache.org/jira/browse/HBASE-16643
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Critical


In the reverse scanner case,
While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the 
backward heap, we do a 
{code}
if ((backwardHeap == null) && (forwardHeap != null)) {
forwardHeap.close();
forwardHeap = null;
// before building the heap seek for the relevant key on the scanners,
// for the heap to be built from the scanners correctly
for (KeyValueScanner scan : scanners) {
  if (toLast) {
res |= scan.seekToLastRow();
  } else {
res |= scan.backwardSeek(cell);
  }
}
{code}
forwardHeap.close(). This would internally decrement the MSLAB ref counter for 
the current active segment and snapshot segment.
When the scan is actually closed again we do close() and that will again 
decrement the count. Here chances are there that the count would go negative 
and hence the actual MSLAB closure that checks for refCount==0 will fail. Apart 
from this, when the refCount becomes 0 after the firstClose if any other thread 
requests to close the segment, then we will end up in corrupted segment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495675#comment-15495675
 ] 

ramkrishna.s.vasudevan commented on HBASE-15871:


I can change it. Thanks Ted for the review.
Tried adding a 'closed' flag in SegmentScanner and to see how things work and 
found an interesting bug. So if we need to go with 'closed' flag in 
SegmentScanner and check for the flag on every operation then we need to solve 
the new bug. It is quite critical too. 

> Memstore flush doesn't finish because of backwardseek() in memstore scanner.
> 
>
> Key: HBASE-15871
> URL: https://issues.apache.org/jira/browse/HBASE-15871
> Project: HBase
>  Issue Type: Bug
>  Components: Scanners
>Affects Versions: 1.1.2
>Reporter: Jeongdae Kim
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-15871-branch-1.patch, 
> HBASE-15871.branch-1.1.001.patch, HBASE-15871.branch-1.1.002.patch, 
> HBASE-15871.branch-1.1.003.patch, HBASE-15871.patch, HBASE-15871_1.patch, 
> HBASE-15871_1.patch, HBASE-15871_2.patch, HBASE-15871_3.patch, 
> HBASE-15871_4.patch, memstore_backwardSeek().PNG
>
>
> Sometimes in our production hbase cluster, it takes a long time to finish 
> memstore flush.( for about more than 30 minutes)
> the reason is that a memstore flusher thread calls 
> StoreScanner.updateReaders(), waits for acquiring a lock that store scanner 
> holds in StoreScanner.next() and backwardseek() in memstore scanner runs for 
> a long time.
> I think that this condition could occur in reverse scan by the following 
> process.
> 1) create a reversed store scanner by requesting a reverse scan.
> 2) flush a memstore in the same HStore.
> 3) puts a lot of cells in memstore and memstore is almost full.
> 4) call the reverse scanner.next() and re-create all scanners in this store 
> because all scanners was already closed by 2)'s flush() and backwardseek() 
> with store's lastTop for all new scanners.
> 5) in this status, memstore is almost full by 2) and all cells in memstore 
> have sequenceID greater than this scanner's readPoint because of 2)'s 
> flush(). this condition causes searching all cells in memstore, and 
> seekToPreviousRow() repeatly seach cells that are already searched if a row 
> has one column. (described this in more detail in a attached file.)
> 6) flush a memstore again in the same HStore, and wait until 4-5) process 
> finished, to update store files in the same HStore after flusing.
> I searched HBase jira. and found a similar issue. (HBASE-14497) but, 
> HBASE-14497's fix can't solve this issue because that fix just changed 
> recursive call to loop.(and already applied to our HBase version)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16636) Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting Regions Stuck in Transition

[Hari Sekhon (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495649#comment-15495649
 ] 

Hari Sekhon commented on HBASE-16636:
-

Hi [~huaxiang], it is related although I've noticed that both the regions in 
transition as well as the regions in transition over threshold counts are both 
zero, are both fixed by that other issue?

> Regions in Transition counts wrong (zero) in HMaster /jmx, prevents detecting 
> Regions Stuck in Transition
> -
>
> Key: HBASE-16636
> URL: https://issues.apache.org/jira/browse/HBASE-16636
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>Affects Versions: 1.1.2
> Environment: HDP 2.3.2
>Reporter: Hari Sekhon
>Priority: Minor
> Attachments: Regions_in_Transition_UI.png, ritCountOverThreshold.png
>
>
> I've discovered that the Region in Transition counts are wrong in the HMaster 
> UI /jmx page.
> The /master-status page clearly shows 3 regions stuck in transition but the 
> /jmx page I was monitoring reported 0 for ritCountOverThreshold.
> {code}
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManger",
> "modelerType" : "Master,sub=AssignmentManger",
> "tag.Context" : "master",
> ...
> "ritOldestAge" : 0,
> "ritCountOverThreshold" : 0,
> ...
> "ritCount" : 0,
> {code}
> I have a nagios plugin I wrote which was checking this which I've since had 
> to rewrite to parse the /master-status page instead (the code is in 
> check_hbase_regions_stuck_in_transition.py at 
> https://github.com/harisekhon/nagios-plugins).
> I'm attaching screenshots of both /master-status and /jmx to show the 
> difference in the 2 pages on the HMaster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16408) Handle On heap BucketCache size when HeapMemoryManager tunes memory

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495603#comment-15495603
 ] 

Anoop Sam John commented on HBASE-16408:


As of now there is no problem (I was wrong).. Even if BucketCache is onheap, we 
dont consider that size at all in size calc for HeapMemory tuner.Now we 
have optimized off heap read so am wondering why we need on heap version of 
bucket cache.   File and off heap mode is enough?  Do some one really using 
BucketCache on heap?  Off heap will perform equally or even better now.  
[~stack] how abt removing on heap Bucket cache mode?

> Handle On heap BucketCache size when HeapMemoryManager tunes memory
> ---
>
> Key: HBASE-16408
> URL: https://issues.apache.org/jira/browse/HBASE-16408
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16408) Handle On heap BucketCache size when HeapMemoryManager tunes memory

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422734#comment-15422734
 ] 

Anoop Sam John edited comment on HBASE-16408 at 9/16/16 6:50 AM:
-

Ya. The HeapMemoryManager handle L1 cache only as of now. When we have L2 cache 
as onheap, we might need to change size as per size change from tuner.


was (Author: anoop.hbase):
Ya. The HeapMemoryManager handle L1 cache only as of now. When we e have L2 
cache as off heap, we might need to change size as per size change from tuner.

> Handle On heap BucketCache size when HeapMemoryManager tunes memory
> ---
>
> Key: HBASE-16408
> URL: https://issues.apache.org/jira/browse/HBASE-16408
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495546#comment-15495546
 ] 

Anoop Sam John edited comment on HBASE-16626 at 9/16/16 6:36 AM:
-

It was because of some old force push.. Now I changed my git local commit 
repo.. Hope it wont come with later commits from me.  In commit history diff it 
shows as trivial merge only.. No extra unwanted change any way.
Sorry for this


was (Author: anoop.hbase):
It was because of some old force push.. Now I changed my git local commit 
repo.. Hope it wont come with later commits from me.  Sorry for this

> User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap 
> part
> -
>
> Key: HBASE-16626
> URL: https://issues.apache.org/jira/browse/HBASE-16626
> Project: HBase
>  Issue Type: Sub-task
>  Components: Offheaping
>Affects Versions: 2.0.0
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Fix For: 2.0.0
>
> Attachments: HBASE-16626-v1.patch
>
>
> Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a 
> new interface Shipper which contains a  "shipped()" method. 
> In our case, some application user defined customized scanner that implements 
> Interface RegionScanner in 1.X . 
> After we back ported the Off Heap feature of 2.0.0, 
> RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when 
> executing scanner.shipped(). It is because the customized scanner didn't 
> override the shipped method yet.
> Instead of forcing every user to add a empty implementation(if they don't 
> really need to scan the file or the RS don't use L2 cache, they don't need to 
> do anything in shipped method) , adding a default method of shipped in 
> Interface  RegionScanner might be a better way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16626) User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap part

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495546#comment-15495546
 ] 

Anoop Sam John commented on HBASE-16626:


It was because of some old force push.. Now I changed my git local commit 
repo.. Hope it wont come with later commits from me.  Sorry for this

> User customized RegionScanner from 1.X is incompatible with 2.0.0's off-heap 
> part
> -
>
> Key: HBASE-16626
> URL: https://issues.apache.org/jira/browse/HBASE-16626
> Project: HBase
>  Issue Type: Sub-task
>  Components: Offheaping
>Affects Versions: 2.0.0
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Fix For: 2.0.0
>
> Attachments: HBASE-16626-v1.patch
>
>
> Introduced by 2.0.0's off-heap feature, the interface RegionScanner extends a 
> new interface Shipper which contains a  "shipped()" method. 
> In our case, some application user defined customized scanner that implements 
> Interface RegionScanner in 1.X . 
> After we back ported the Off Heap feature of 2.0.0, 
> RegionScannerShippedCallBack throws a "java.lang.AbstractMethodError" when 
> executing scanner.shipped(). It is because the customized scanner didn't 
> override the shipped method yet.
> Instead of forcing every user to add a empty implementation(if they don't 
> really need to scan the file or the RS don't use L2 cache, they don't need to 
> do anything in shipped method) , adding a default method of shipped in 
> Interface  RegionScanner might be a better way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16349) TestClusterId may hang during cluster shutdown

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495534#comment-15495534
 ] 

Hadoop QA commented on HBASE-16349:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
30m 29s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 26s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 14s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
|   | org.apache.hadoop.hbase.client.TestScannerTimeout |
|   | 
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
|   | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
|   | org.apache.hadoop.hbase.client.TestMobSnapshotCloneIndependence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12828793/16349.v1.txt |
| JIRA Issue | HBASE-16349 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux be07317fd0ef 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / e19632a |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/3570/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  

[jira] [Created] (HBASE-16642) Use DelayQueue instead of TimeoutBlockingQueue

[Hiroshi Ikeda (JIRA)]

Hiroshi Ikeda created HBASE-16642:
-

 Summary: Use DelayQueue instead of TimeoutBlockingQueue
 Key: HBASE-16642
 URL: https://issues.apache.org/jira/browse/HBASE-16642
 Project: HBase
  Issue Type: Improvement
Reporter: Hiroshi Ikeda
Priority: Minor


Enqueue poisons in order to wake up and end the internal threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16640) TimeoutBlockingQueue#remove() should return whether the entry is removed

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495492#comment-15495492
 ] 

Hudson commented on HBASE-16640:


FAILURE: Integrated in Jenkins build HBase-1.4 #417 (See 
[https://builds.apache.org/job/HBase-1.4/417/])
HBASE-16640 TimeoutBlockingQueue#remove() should return whether the (tedyu: rev 
76a07602691fd5b63390b9dbde0a84672602c0be)
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/util/TimeoutBlockingQueue.java
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
* (edit) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/util/TestTimeoutBlockingQueue.java


> TimeoutBlockingQueue#remove() should return whether the entry is removed
> 
>
> Key: HBASE-16640
> URL: https://issues.apache.org/jira/browse/HBASE-16640
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16640.v1.txt, 16640.v2.txt
>
>
> Currently return type of TimeoutBlockingQueue#remove() is void.
> The caller has no way of knowing whether the entry is really removed.
> This issue changes the return type to boolean. True is returned when entry is 
> removed, false is returned otherwise.
> The change was proposed first in HBASE-16639.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16639) TestProcedureInMemoryChore#testChoreAddAndRemove occasionally fails

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495491#comment-15495491
 ] 

Hudson commented on HBASE-16639:


FAILURE: Integrated in Jenkins build HBase-1.4 #417 (See 
[https://builds.apache.org/job/HBase-1.4/417/])
HBASE-16639 TestProcedureInMemoryChore#testChoreAddAndRemove (matteo.bertozzi: 
rev 08e44919cc2994760cea26f76ad9ecbcba9fb804)
* (edit) 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureInMemoryChore.java
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
* (edit) 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java


> TestProcedureInMemoryChore#testChoreAddAndRemove occasionally fails
> ---
>
> Key: HBASE-16639
> URL: https://issues.apache.org/jira/browse/HBASE-16639
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 16639.v1.txt, HBASE-16639-v0.patch
>
>
> See 
> https://builds.apache.org/job/HBase-TRUNK_matrix/1608/jdk=JDK_1_8,label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.procedure2/TestProcedureInMemoryChore/testChoreAddAndRemove/
>  :
> {code}
> java.lang.AssertionError: null
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> org.apache.hadoop.hbase.procedure2.TestProcedureInMemoryChore.testChoreAddAndRemove(TestProcedureInMemoryChore.java:87)
> ...
> 2016-09-15 21:41:25,210 INFO  [main] 
> procedure2.TestProcedureInMemoryChore(86): chore latch count=1
> {code}
> When I run the test on laptop, I saw:
> {code}
> 2016-09-15 14:56:29,857 INFO  [main] 
> procedure2.TestProcedureInMemoryChore(86): chore latch count=5
> {code}
> This is a small test whose failure would prevent medium / large tests from 
> running.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)