[jira] [Updated] (HBASE-14919) Infrastructure refactoring for In-Memory Flush

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14919:
--
Attachment: HBASE-14919-V06.patch

Rerun

The hadoop failures are because of a bad download stuck a while on our build 
machines... not your fault.

> Infrastructure refactoring for In-Memory Flush
> --
>
> Key: HBASE-14919
> URL: https://issues.apache.org/jira/browse/HBASE-14919
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14919-V01.patch, HBASE-14919-V01.patch, 
> HBASE-14919-V02.patch, HBASE-14919-V03.patch, HBASE-14919-V04.patch, 
> HBASE-14919-V04.patch, HBASE-14919-V05.patch, HBASE-14919-V06.patch, 
> HBASE-14919-V06.patch
>
>
> Refactoring the MemStore hierarchy, introducing segment (StoreSegment) as 
> first-class citizen and decoupling memstore scanner from the memstore 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112972#comment-15112972
 ] 

Hadoop QA commented on HBASE-15128:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 51s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 28s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 28s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 25s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 25s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 3s 
{color} | {color:red} Patch generated 6 new checkstyle issues in hbase-client 
(total was 473, now 478). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 33s 
{color} | {color:red} Patch generated 7 new checkstyle issues in hbase-server 
(total was 339, now 345). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
54s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m 14s 
{color} | {color:red} The applied patch generated 45 new rubocop issues (total 
was 828, now 870). {color} |
| {color:red}-1{color} | {color:red} ruby-lint {color} | {color:red} 0m 5s 
{color} | {color:red} The applied patch generated 64 new ruby-lint issues 
(total was 530, now 594). {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s 
{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
24m 15s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 
2s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 26s 
{color} | 

[jira] [Updated] (HBASE-14963) Remove Guava dependency from HBase client code

[Devaraj Das (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj Das updated HBASE-14963:

Resolution: Fixed
Status: Resolved  (was: Patch Available)

Committed this.

> Remove Guava dependency from HBase client code
> --
>
> Key: HBASE-14963
> URL: https://issues.apache.org/jira/browse/HBASE-14963
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Fix For: 2.0.0
>
> Attachments: no-stopwatch.txt
>
>
> We ran into an issue where an application bundled its own Guava (and that 
> happened to be in the classpath first) and HBase's MetaTableLocator threw an 
> exception due to the fact that Stopwatch's constructor wasn't compatible... 
> Might be better to not depend on Stopwatch at all in MetaTableLocator since 
> the functionality is easily doable without.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15158) Change order in which we do write pipeline operations; do all under row locks!

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113172#comment-15113172
 ] 

Elliott Clark commented on HBASE-15158:
---

I am so excited for this patch! Region replay being broken really shouldn't 
stop this from going in. The replica ones though seem more related.

> Change order in which we do write pipeline operations; do all under row locks!
> --
>
> Key: HBASE-15158
> URL: https://issues.apache.org/jira/browse/HBASE-15158
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 15158.patch
>
>
> Change how we do our write pipeline. I want to do all write pipeline ops 
> under row lock so I lean on this fact fixing performance regression in 
> check-and-set type operations like increment, append, and checkAnd* (see 
> sibling issue HBASE-15082).
> To be specific, we write like this now:
> {code}
> # take rowlock
> # start mvcc
> # append to WAL
> # add to memstore
> # let go of rowlock
> # sync WAL
> # in case of error: rollback memstore
> {code}
> Instead, write like this:
> {code}
> # take rowlock
> # start mvcc
> # append to WAL
> # sync WAL
> # add to memstore
> # let go of rowlock
> ... no need to do rollback.
> {code}
> The old ordering was put in place because it got better performance in a time 
> when WAL was different and before row locks were read/write (HBASE-12751).
> Testing in branch-1 shows that a reordering and skipping mvcc waits gets us 
> back to the performance we had before we unified mvcc and sequenceid 
> (HBASE-8763). Tests in HBASE-15046 show that at the macro level using our 
> usual perf tools, reordering pipeline seems to cause no slowdown (see 
> HBASE-15046). A rough compare of increments with reordered write pipeline 
> seems to have us getting back a bunch of our performance (see tail of 
> https://issues.apache.org/jira/browse/HBASE-15082?focusedCommentId=15111703=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15111703
>  and subsequent comment).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14963) Remove Guava dependency from HBase client code

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113183#comment-15113183
 ] 

Sean Busbey commented on HBASE-14963:
-

we don't remove dependencies in patch releases, so branch-1.1 is out.

> Remove Guava dependency from HBase client code
> --
>
> Key: HBASE-14963
> URL: https://issues.apache.org/jira/browse/HBASE-14963
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Fix For: 2.0.0
>
> Attachments: no-stopwatch.txt
>
>
> We ran into an issue where an application bundled its own Guava (and that 
> happened to be in the classpath first) and HBase's MetaTableLocator threw an 
> exception due to the fact that Stopwatch's constructor wasn't compatible... 
> Might be better to not depend on Stopwatch at all in MetaTableLocator since 
> the functionality is easily doable without.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113131#comment-15113131
 ] 

Hadoop QA commented on HBASE-9393:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 53s 
{color} | {color:red} Patch causes 16 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 43s 
{color} | {color:red} Patch causes 16 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 34s 
{color} | {color:red} Patch causes 16 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 24s 
{color} | {color:red} Patch causes 16 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 15s 
{color} | {color:red} Patch causes 16 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 5s 
{color} | {color:red} Patch causes 16 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 56s 
{color} | {color:red} Patch causes 16 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 47s 
{color} | {color:red} Patch causes 16 errors with Hadoop v2.6.3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 

[jira] [Assigned] (HBASE-15135) Add metrics for storefile age

[Mikhail Antonov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Antonov reassigned HBASE-15135:
---

Assignee: Mikhail Antonov

> Add metrics for storefile age
> -
>
> Key: HBASE-15135
> URL: https://issues.apache.org/jira/browse/HBASE-15135
> Project: HBase
>  Issue Type: New Feature
>Reporter: Elliott Clark
>Assignee: Mikhail Antonov
>
> In order to make sure that compactions are fully up to date it would be nice 
> to have metrics on:
> * Max storefile age
> * Min storefile age
> * Average storefile age
> * Number of reference files
> If we could have those metrics per region and per regionserver it would be 
> awesome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14963) Remove Guava dependency from HBase client code

[Stephen Yuan Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113165#comment-15113165
 ] 

Stephen Yuan Jiang commented on HBASE-14963:


[~devaraj], are you planning to commit this patch to branch-1/branch-1.1 (or 
branch-1.2 once it is unlocked from release)?  I check the code in branch-1, 
the same problem exists. 

> Remove Guava dependency from HBase client code
> --
>
> Key: HBASE-14963
> URL: https://issues.apache.org/jira/browse/HBASE-14963
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Fix For: 2.0.0
>
> Attachments: no-stopwatch.txt
>
>
> We ran into an issue where an application bundled its own Guava (and that 
> happened to be in the classpath first) and HBase's MetaTableLocator threw an 
> exception due to the fact that Stopwatch's constructor wasn't compatible... 
> Might be better to not depend on Stopwatch at all in MetaTableLocator since 
> the functionality is easily doable without.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113175#comment-15113175
 ] 

Elliott Clark commented on HBASE-15146:
---

Working on a new version of the patch to address the issues that Gary was 
talking about.

> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, 
> HBASE-15146.5.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of 
> different queues. Those queues have different capacity limits. Currently the 
> dispatch method can block trying to add the the blocking queue in any of the 
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14919) Infrastructure refactoring for In-Memory Flush

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112968#comment-15112968
 ] 

stack commented on HBASE-14919:
---

The 82 extant findbugs are not your issue (And they have been cleaned up 
since). The hadoop failures are not yours either. They should be good now too. 
Let me rerun your patch

> Infrastructure refactoring for In-Memory Flush
> --
>
> Key: HBASE-14919
> URL: https://issues.apache.org/jira/browse/HBASE-14919
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14919-V01.patch, HBASE-14919-V01.patch, 
> HBASE-14919-V02.patch, HBASE-14919-V03.patch, HBASE-14919-V04.patch, 
> HBASE-14919-V04.patch, HBASE-14919-V05.patch, HBASE-14919-V06.patch
>
>
> Refactoring the MemStore hierarchy, introducing segment (StoreSegment) as 
> first-class citizen and decoupling memstore scanner from the memstore 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15158) Change order in which we do write pipeline operations; do all under row locks!

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113053#comment-15113053
 ] 

Hadoop QA commented on HBASE-15158:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 20 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 30s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m 
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
5s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 0s 
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 30s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 17s 
{color} | {color:red} hbase-examples in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 12s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 12s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 36s 
{color} | {color:red} Patch generated 19 new checkstyle issues in hbase-server 
(total was 863, now 777). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 40s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m 29s 
{color} | {color:red} hbase-server-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 
new issues (was 1, now 2). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 15s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s 
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s 
{color} | {color:green} hbase-examples in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 147m 49s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} unit 

[jira] [Updated] (HBASE-15132) Master region merge RPC should authorize user request

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15132:
---
Attachment: HBASE-15132.v7.patch

Patch v7 wraps call to master.cpHost.postDispatchMerge() in doAs()

> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132-branch-1.v6.patch, HBASE-15132.v1.patch, 
> HBASE-15132.v2.patch, HBASE-15132.v4.patch, HBASE-15132.v5.patch, 
> HBASE-15132.v6.patch, HBASE-15132.v7.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112933#comment-15112933
 ] 

Hadoop QA commented on HBASE-15132:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
8s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s 
{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m 
49s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 108m 57s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 16s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
19s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 234m 0s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hbase.master.procedure.TestModifyNamespaceProcedure |
| JDK v1.7.0_91 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-22 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783843/HBASE-15132-branch-1.v6.patch
 |
| JIRA Issue | HBASE-15132 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle 

[jira] [Reopened] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-13082:
---

Reopening. This commit introduces a test that fails reliably. Git bisect 
fingers this commit:

58521869b06a63894e422e9c9403e48b4b12f388 is the first bad commit
commit 58521869b06a63894e422e9c9403e48b4b12f388
Author: ramkrishna 
Date:   Thu Jan 21 21:22:40 2016 +0530

HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (Ram)

:04 04 ac9ba8c501616a32632f7b46adfe0afb7073458b 
140cac3904b52a7d599e20f21c8d55961847ca77 M  hbase-client
:04 04 f3f53edf84a0f7885dcc577159ba5bbc1a8c6922 
7c023622983f3bc4d9c1a43b2dd52c7d2d181b51 M  hbase-server


Here is my little test:

mvn clean install -DskipTests && mvn test -Dtest=TestHFileOutputFormat

Let me revert for now since its your w/e [~ram_krish]

> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, 
> HBASE-13082_12.patch, HBASE-13082_13.patch, HBASE-13082_14.patch, 
> HBASE-13082_15.patch, HBASE-13082_16.patch, HBASE-13082_17.patch, 
> HBASE-13082_18.patch, HBASE-13082_19.patch, HBASE-13082_1_WIP.patch, 
> HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, 
> HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, 
> HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, 
> LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking 
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and 
> required in the documentation (not picking on Phoenix, this one is my fault 
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. 
> RegionScanner operations would keep getting the locks and the 
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack reopened HBASE-14970:
---

Reopening. This commit introduces a test that fails reliably. Git bisect 
fingers this commit:
58521869b06a63894e422e9c9403e48b4b12f388 is the first bad commit
commit 58521869b06a63894e422e9c9403e48b4b12f388
Author: ramkrishna 
Date: Thu Jan 21 21:22:40 2016 +0530
HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (Ram)
:04 04 ac9ba8c501616a32632f7b46adfe0afb7073458b 
140cac3904b52a7d599e20f21c8d55961847ca77 M  hbase-client
:04 04 f3f53edf84a0f7885dcc577159ba5bbc1a8c6922 
7c023622983f3bc4d9c1a43b2dd52c7d2d181b51 M  hbase-server
Here is my little test:
mvn clean install -DskipTests && mvn test -Dtest=TestHFileOutputFormat

I confirmed this the culprit.

Let me revert for now since its your w/e ramkrishna.s.vasudevan

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112957#comment-15112957
 ] 

Ashish Singhi commented on HBASE-9393:
--

Ran some tests,
0. randomWrite
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=134 
--rows=10 randomWrite 20
{noformat}

1. randomRead
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=1000 
randomRead 20
{noformat}

Without patch
{noformat}
2016-01-22 23:49:52,305 INFO  [main] hbase.PerformanceEvaluation: 
[RandomReadTest] Summary of timings (ms): [7223, 7034, 6950, 6945, 6882, 6830, 
7107, 7097, 7122, 6675, 7072, 6636, 7080, 6533, 6987, 6305, 7227, 7172, 6608, 
6589]
2016-01-22 23:49:52,306 INFO  [main] hbase.PerformanceEvaluation: 
[RandomReadTest]  Min: 6305ms Max: 7227ms Avg: 6903ms
{noformat}

With patch
{noformat}
2016-01-22 23:43:08,695 INFO  [main] hbase.PerformanceEvaluation: 
[RandomReadTest] Summary of timings (ms): [6406, 6648, 7623, 6678, 7163, 6673, 
7150, 6712, 6412, 7169, 6364, 6214, 7293, 7484, 7633, 7212, 7350, 6447, 7101, 
6499]
2016-01-22 23:43:08,696 INFO  [main] hbase.PerformanceEvaluation: 
[RandomReadTest]  Min: 6214ms Max: 7633ms Avg: 6911ms
{noformat}

2. sequentialRead
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=1000 
sequentialRead 20
{noformat}

Without patch
{noformat}
2016-01-22 23:54:56,024 INFO  [main] hbase.PerformanceEvaluation: 
[SequentialReadTest] Summary of timings (ms): [6476, 6150, 6291, 6381, 6284, 
6182, 6069, 6350, 6394, 6200, 6260, 6349, 6240, 5974, 6014, 5965, 6483, 6025, 
6098, 6389]
2016-01-22 23:54:56,025 INFO  [main] hbase.PerformanceEvaluation: 
[SequentialReadTest]  Min: 5965ms Max: 6483ms Avg: 6228ms
{noformat}

With patch
{noformat}
2016-01-22 23:58:40,519 INFO  [main] hbase.PerformanceEvaluation: 
[RandomReadTest] Summary of timings (ms): [6985, 6720, 6970, 6756, 6468, 6890, 
6719, 7003, 6348, 6803, 6584, 6846, 6793, 6496, 6490, 6879, 6450, 6663, 6921, 
6896]
2016-01-22 23:58:40,520 INFO  [main] hbase.PerformanceEvaluation: 
[RandomReadTest]  Min: 6348ms Max: 7003ms Avg: 6734ms
{noformat}

3. randomSeekScan

{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=1000 
randomSeekScan 20
{noformat}

Without patch
{noformat}
2016-01-23 00:12:01,954 INFO  [main] hbase.PerformanceEvaluation: 
[RandomSeekScanTest] Summary of timings (ms): [105473, 107416, 88796, 91563, 
104032, 101899, 99557, 103790, 107929, 94213, 103251, 101177, 106168, 106903, 
106086, 101905, 97543, 68672, 91004, 105064]
2016-01-23 00:12:01,954 INFO  [main] hbase.PerformanceEvaluation: 
[RandomSeekScanTest]  Min: 68672msMax: 107929ms   Avg: 99622ms
{noformat}

With patch
{noformat}
2016-01-23 00:05:07,185 INFO  [main] hbase.PerformanceEvaluation: 
[RandomSeekScanTest] Summary of timings (ms): [78781, 82973, 76085, 81127, 
74558, 74974, 60761, 77760, 80286, 70820, 71463, 74105, 70433, 64313, 80937, 
82408, 81356, 83155, 65988, 82360]
2016-01-23 00:05:07,186 INFO  [main] hbase.PerformanceEvaluation: 
[RandomSeekScanTest]  Min: 60761msMax: 83155msAvg: 75732ms
{noformat}

4. filterScan
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --rows=100 
filterScan 3
{noformat}

Without patch
{noformat}
2016-01-23 01:01:10,168 INFO  [main] hbase.PerformanceEvaluation: 
[FilteredScanTest] Summary of timings (ms): [417507, 425604, 420263]
2016-01-23 01:01:10,169 INFO  [main] hbase.PerformanceEvaluation: 
[FilteredScanTest]Min: 417507ms   Max: 425604ms   Avg: 421124ms
{noformat}

With patch
{noformat}
2016-01-23 01:17:28,614 INFO  [main] hbase.PerformanceEvaluation: 
[FilteredScanTest] Summary of timings (ms): [359967, 358833, 359256]
2016-01-23 01:17:28,615 INFO  [main] hbase.PerformanceEvaluation: 
[FilteredScanTest]Min: 358833ms   Max: 359967ms   Avg: 359352ms
{noformat}

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, 
> HBASE-9393.v2.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> 

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112793#comment-15112793
 ] 

Ted Yu commented on HBASE-9393:
---

w.r.t. test failure:
{code}
java.lang.NullPointerException: null
at 
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:525)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:571)
at 
org.apache.hadoop.hbase.io.hfile.TestHFile.testCorrupt0LengthHFile(TestHFile.java:116)
{code}
For corrupt HFile, stream is null in the following call:
{code}
  if (stream.getWrappedStream() instanceof DFSInputStream) {
{code}
Please add null check.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112849#comment-15112849
 ] 

stack commented on HBASE-14970:
---

Reverted.

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14841) Allow Dictionary to work with BytebufferedCells

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112893#comment-15112893
 ] 

stack commented on HBASE-14841:
---

Looks like a few things to fix (a findbugs, whitespace).

On the patch, looks good.

Does this test for a BBCell have to out here in this BufferedDataBlockEncoder 
class?

1007  if (cell instanceof ByteBufferedCell) {
1008tagCompressionContext.compressTags(out, ((ByteBufferedCell) 
cell).getTagsByteBuffer(),
1009  ((ByteBufferedCell) cell).getTagsPosition(), tagsLength);
1010  } else {
1011tagCompressionContext.compressTags(out, 
cell.getTagsArray(), cell.getTagsOffset(),
1012  tagsLength);
1013  }

You fellows have been doing good job of containing the test of Cell type inside 
stuff like CellUtil... is this a violation of your rule?

Checkstyle will flag no brackets here:

592 for (int i = offset; i < offset + length; i++)
593   hash = (31 * hash) + (int) toByte(buf, i);

Patch LGTM otherwise. Get an Anoop +1 I'd say.

> Allow Dictionary to work with BytebufferedCells
> ---
>
> Key: HBASE-14841
> URL: https://issues.apache.org/jira/browse/HBASE-14841
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-14841.patch, HBASE-14841_1.patch, 
> HBASE-14841_2.patch, HBASE-14841_3.patch
>
>
> This is part of HBASE-14832 where we need to ensure that while BBCells are 
> getting compacted the TagCompression part should be working with BBCells.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15147) Shell should use Admin.listTableNames() instead of Admin.listTables()

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112814#comment-15112814
 ] 

Andrew Purtell commented on HBASE-15147:


bq. However, if there are use cases where Table descriptor might contain 
sensitive info,

This answer is yes, because HBase encryption can put key material in CF 
descriptors, and there can be arbitrary user supplied attributes on CF and 
table descriptors. The table and CF names, however, are not expected to be 
sensitive, since it's not possible to hide them for a number of reasons.

> Shell should use Admin.listTableNames() instead of Admin.listTables() 
> --
>
> Key: HBASE-15147
> URL: https://issues.apache.org/jira/browse/HBASE-15147
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: hbase-15147_v1.patch
>
>
> It seems that getTableDescriptors() in master checks for A and C permissions 
> while getTableNames() checks for any privilege on the table. The reasoning is 
> explained here: 
> https://issues.apache.org/jira/browse/HBASE-12564?focusedCommentId=14234504=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234504
>  
> We should change the shell command for {{list}} to use the getTableNames() 
> version because of this. Otherwise a user having only R or W cannot list the 
> table name. 
> This has been reported from a user here: 
> https://community.hortonworks.com/questions/10742/why-does-a-user-need-create-permission-for-list-co.html#comment-11000.
>  
> While we are at it, should we revisit the fact that you cannot get a table 
> descriptor if you have only R or W? It seems strange that you cannot even 
> know the CF names of a table that you can read from. I could not find info 
> about the "describe" privileges on SQL databases. However, if there are use 
> cases where Table descriptor might contain sensitive info, the current 
> semantics seems fine. cc [~apurtell] and [~mbertozzi]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112925#comment-15112925
 ] 

Enis Soztutar commented on HBASE-15132:
---

I was checking why we are not wrapping the post call with doAs() as well. I 
think the doAs in the pre is not needed at all, since the handler is already in 
user context, no? 

> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132-branch-1.v6.patch, HBASE-15132.v1.patch, 
> HBASE-15132.v2.patch, HBASE-15132.v4.patch, HBASE-15132.v5.patch, 
> HBASE-15132.v6.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112954#comment-15112954
 ] 

Ted Yu commented on HBASE-15132:


w.r.t. test failure in TestModifyNamespaceProcedure, the patch is not related 
to either namespace nor procedure V2.
I ran it locally and it passed.

> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132-branch-1.v6.patch, HBASE-15132.v1.patch, 
> HBASE-15132.v2.patch, HBASE-15132.v4.patch, HBASE-15132.v5.patch, 
> HBASE-15132.v6.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-9393:
-
Attachment: HBASE-9393.v2.patch

Patch addressing test failures and code refactor comment.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, 
> HBASE-9393.v2.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-13082.
---
Resolution: Fixed

Reresolving. Wrong issue. Meant to do the backport issue.

> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, 
> HBASE-13082_12.patch, HBASE-13082_13.patch, HBASE-13082_14.patch, 
> HBASE-13082_15.patch, HBASE-13082_16.patch, HBASE-13082_17.patch, 
> HBASE-13082_18.patch, HBASE-13082_19.patch, HBASE-13082_1_WIP.patch, 
> HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, 
> HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, 
> HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, 
> LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait for memory fetches).
> There are some drawbacks too:
> * All calls to RegionScanner need to be remain synchronized
> * Implementors of coprocessors need to be diligent in following the locking 
> contract. For example Phoenix does not lock RegionScanner.nextRaw() and 
> required in the documentation (not picking on Phoenix, this one is my fault 
> as I told them it's OK)
> * possible starving of flushes and compaction with heavy read load. 
> RegionScanner operations would keep getting the locks and the 
> flushes/compactions would not be able finalize the set of files.
> I'll have a patch soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15147) Shell should use Admin.listTableNames() instead of Admin.listTables()

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112853#comment-15112853
 ] 

Enis Soztutar commented on HBASE-15147:
---

bq. HBase encryption can put key material in CF descriptors, and there can be 
arbitrary user supplied attributes on CF and table descriptors. 
I see. Then we can do the stripping of information in HTD/HCD depending on 
perms in a follow up jira if needed. 

> Shell should use Admin.listTableNames() instead of Admin.listTables() 
> --
>
> Key: HBASE-15147
> URL: https://issues.apache.org/jira/browse/HBASE-15147
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: hbase-15147_v1.patch
>
>
> It seems that getTableDescriptors() in master checks for A and C permissions 
> while getTableNames() checks for any privilege on the table. The reasoning is 
> explained here: 
> https://issues.apache.org/jira/browse/HBASE-12564?focusedCommentId=14234504=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234504
>  
> We should change the shell command for {{list}} to use the getTableNames() 
> version because of this. Otherwise a user having only R or W cannot list the 
> table name. 
> This has been reported from a user here: 
> https://community.hortonworks.com/questions/10742/why-does-a-user-need-create-permission-for-list-co.html#comment-11000.
>  
> While we are at it, should we revisit the fact that you cannot get a table 
> descriptor if you have only R or W? It seems strange that you cannot even 
> know the CF names of a table that you can read from. I could not find info 
> about the "describe" privileges on SQL databases. However, if there are use 
> cases where Table descriptor might contain sensitive info, the current 
> semantics seems fine. cc [~apurtell] and [~mbertozzi]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14918) In-Memory MemStore Flush and Compaction

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112950#comment-15112950
 ] 

stack commented on HBASE-14918:
---

[~anoop.hbase] You see above sir?

> In-Memory MemStore Flush and Compaction
> ---
>
> Key: HBASE-14918
> URL: https://issues.apache.org/jira/browse/HBASE-14918
> Project: HBase
>  Issue Type: Umbrella
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Fix For: 0.98.18
>
>
> A memstore serves as the in-memory component of a store unit, absorbing all 
> updates to the store. From time to time these updates are flushed to a file 
> on disk, where they are compacted (by eliminating redundancies) and 
> compressed (i.e., written in a compressed format to reduce their storage 
> size).
> We aim to speed up data access, and therefore suggest to apply in-memory 
> memstore flush. That is to flush the active in-memory segment into an 
> intermediate buffer where it can be accessed by the application. Data in the 
> buffer is subject to compaction and can be stored in any format that allows 
> it to take up smaller space in RAM. The less space the buffer consumes the 
> longer it can reside in memory before data is flushed to disk, resulting in 
> better performance.
> Specifically, the optimization is beneficial for workloads with 
> medium-to-high key churn which incur many redundant cells, like persistent 
> messaging. 
> We suggest to structure the solution as 4 subtasks (respectively, patches). 
> (1) Infrastructure - refactoring of the MemStore hierarchy, introducing 
> segment (StoreSegment) as first-class citizen, and decoupling memstore 
> scanner from the memstore implementation;
> (2) Adding StoreServices facility at the region level to allow memstores 
> update region counters and access region level synchronization mechanism;
> (3) Implementation of a new memstore (CompactingMemstore) with non-optimized 
> immutable segment representation, and 
> (4) Memory optimization including compressed format representation and off 
> heap allocations.
> This Jira continues the discussion in HBASE-13408.
> Design documents, evaluation results and previous patches can be found in 
> HBASE-13408. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112949#comment-15112949
 ] 

Ted Yu commented on HBASE-15132:


bq. since the handler is already in user context, no?
I don't think so.
Here is the stack trace:
{code}
MasterRpcServices.dispatchMergingRegions(RpcController, 
MasterProtos$DispatchMergingRegionsRequest) line: 531   
MasterProtos$MasterService$2.callBlockingMethod(Descriptors$MethodDescriptor, 
RpcController, Message) line: 58210   
RpcServer.call(BlockingService, MethodDescriptor, Message, CellScanner, long, 
MonitoredRPCHandler) line: 2231   
CallRunner.run() line: 109  
BalancedQueueRpcExecutor(RpcExecutor).consumerLoop(BlockingQueue) 
line: 133 
RpcExecutor$1.run() line: 108   
Thread.run() line: 745  
{code}

> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132-branch-1.v6.patch, HBASE-15132.v1.patch, 
> HBASE-15132.v2.patch, HBASE-15132.v4.patch, HBASE-15132.v5.patch, 
> HBASE-15132.v6.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14919) Infrastructure refactoring for In-Memory Flush

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14919:
--
Summary: Infrastructure refactoring for In-Memory Flush  (was: 
Infrastructure refactoring)

> Infrastructure refactoring for In-Memory Flush
> --
>
> Key: HBASE-14919
> URL: https://issues.apache.org/jira/browse/HBASE-14919
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Attachments: HBASE-14919-V01.patch, HBASE-14919-V01.patch, 
> HBASE-14919-V02.patch, HBASE-14919-V03.patch, HBASE-14919-V04.patch, 
> HBASE-14919-V04.patch, HBASE-14919-V05.patch, HBASE-14919-V06.patch
>
>
> Refactoring the MemStore hierarchy, introducing segment (StoreSegment) as 
> first-class citizen and decoupling memstore scanner from the memstore 
> implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-14918) In-Memory MemStore Flush and Compaction

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112950#comment-15112950
 ] 

stack edited comment on HBASE-14918 at 1/22/16 7:50 PM:


[~anoop.hbase] You see above sir? We want CellBlocks right? Not HFile blocks? 
Do we need to relate the two? Should an HFile block be CellBlock?


was (Author: stack):
[~anoop.hbase] You see above sir?

> In-Memory MemStore Flush and Compaction
> ---
>
> Key: HBASE-14918
> URL: https://issues.apache.org/jira/browse/HBASE-14918
> Project: HBase
>  Issue Type: Umbrella
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Fix For: 0.98.18
>
>
> A memstore serves as the in-memory component of a store unit, absorbing all 
> updates to the store. From time to time these updates are flushed to a file 
> on disk, where they are compacted (by eliminating redundancies) and 
> compressed (i.e., written in a compressed format to reduce their storage 
> size).
> We aim to speed up data access, and therefore suggest to apply in-memory 
> memstore flush. That is to flush the active in-memory segment into an 
> intermediate buffer where it can be accessed by the application. Data in the 
> buffer is subject to compaction and can be stored in any format that allows 
> it to take up smaller space in RAM. The less space the buffer consumes the 
> longer it can reside in memory before data is flushed to disk, resulting in 
> better performance.
> Specifically, the optimization is beneficial for workloads with 
> medium-to-high key churn which incur many redundant cells, like persistent 
> messaging. 
> We suggest to structure the solution as 4 subtasks (respectively, patches). 
> (1) Infrastructure - refactoring of the MemStore hierarchy, introducing 
> segment (StoreSegment) as first-class citizen, and decoupling memstore 
> scanner from the memstore implementation;
> (2) Adding StoreServices facility at the region level to allow memstores 
> update region counters and access region level synchronization mechanism;
> (3) Implementation of a new memstore (CompactingMemstore) with non-optimized 
> immutable segment representation, and 
> (4) Memory optimization including compressed format representation and off 
> heap allocations.
> This Jira continues the discussion in HBASE-13408.
> Design documents, evaluation results and previous patches can be found in 
> HBASE-13408. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14969) Add throughput controller for flush

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112243#comment-15112243
 ] 

Yu Li commented on HBASE-14969:
---

Checking UT failures:
* TestImportExport.testDurability:
{noformat}
java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error 
while running command to get file permissions :
ExitCodeException exitCode=127: /bin/ls: error while loading shared libraries: 
libc.so.6: failed to map segment from shared object:
Permission denied
{noformat}
* TestImportExport.testWithMultipleDeleteFamilyMarkersOfSameRowSameFamily:
{noformat}
java.io.IOException: java.util.concurrent.ExecutionException: ExitCodeException 
exitCode=137: 
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
{noformat}

Both have nothing to do with changes here.

> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.branch-1.patch, HBASE-14969.patch, 
> HBASE-14969_v10.patch, HBASE-14969_v2.patch, HBASE-14969_v3.patch, 
> HBASE-14969_v4.patch, HBASE-14969_v5.patch, HBASE-14969_v6.patch, 
> HBASE-14969_v9.patch, fd78628_9909808_compat_report.html, 
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike 
> caused by huge IO pressure like network/disk overflow. However, even with 
> this control on, we are still observing disk utils near 100%, and by analysis 
> we think this is caused by flush, especially when we increase the setting of 
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a 
> supplement of HBASE-8329 to better control IO pressure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15133) Data loss after compaction when a row has more than Integer.MAX_VALUE columns

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112279#comment-15112279
 ] 

Ted Yu commented on HBASE-15133:


TestStochasticLoadBalancer test failure was not related - the test is flaky.

> Data loss after compaction when a row has more than Integer.MAX_VALUE columns
> -
>
> Key: HBASE-15133
> URL: https://issues.apache.org/jira/browse/HBASE-15133
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
> Attachments: HBASE-15133-v1.patch, HBASE-15133.patch, 
> HBASE-15133.patch
>
>
> We have lost the data in our development environment when a row has more than 
> Integer.MAX_VALUE columns after compaction.
> I think the reason is type of StoreScanner's countPerRow is int.
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L67
> After changing the type to long, it seems to be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112092#comment-15112092
 ] 

Hadoop QA commented on HBASE-15132:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
21m 40s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 100m 24s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m 30s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_79. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 239m 45s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783745/HBASE-15132.v6.patch |
| JIRA Issue | HBASE-15132 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / b6f091e |
| findbugs | v3.0.0 

[jira] [Commented] (HBASE-15133) Data loss after compaction when a row has more than Integer.MAX_VALUE columns

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112276#comment-15112276
 ] 

Hadoop QA commented on HBASE-15133:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
23m 27s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 117m 28s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 120m 7s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
14s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 284m 48s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | 
hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
| JDK v1.7.0_79 Failed junit tests | 
hadoop.hbase.master.balancer.TestStochasticLoadBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783755/HBASE-15133.patch |
| JIRA Issue | HBASE-15133 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency 

[jira] [Updated] (HBASE-15133) Data loss after compaction when a row has more than Integer.MAX_VALUE columns

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15133:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
   1.2.0
   2.0.0
   Status: Resolved  (was: Patch Available)

Thanks for the patch, Toshihiro

Thanks for the reviews.

> Data loss after compaction when a row has more than Integer.MAX_VALUE columns
> -
>
> Key: HBASE-15133
> URL: https://issues.apache.org/jira/browse/HBASE-15133
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15133-v1.patch, HBASE-15133.patch, 
> HBASE-15133.patch
>
>
> We have lost the data in our development environment when a row has more than 
> Integer.MAX_VALUE columns after compaction.
> I think the reason is type of StoreScanner's countPerRow is int.
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L67
> After changing the type to long, it seems to be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warning in AuthenticationTokenSecretManager

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15148:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the patch, Yu.

> Resolve IS2_INCONSISTENT_SYNC findbugs warning in 
> AuthenticationTokenSecretManager
> --
>
> Key: HBASE-15148
> URL: https://issues.apache.org/jira/browse/HBASE-15148
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15148.patch, HBASE-15148.patch
>
>
> After efforts in HBASE-15118, we still see IS2_INCONSISTENT_SYNC warning in 
> AuthenticationTokenSecretManager in [HadoopQA report | 
> https://builds.apache.org/job/PreCommit-HBASE-Build/197/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html]
>  for HBASE-13960:
> {noformat}
> In class 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager
> Field 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.lastKeyUpdate
> Synchronized 50% of the time
> Unsynchronized access at AuthenticationTokenSecretManager.java:[line 343]
> Synchronized access at AuthenticationTokenSecretManager.java:[line 278]
> {noformat}
> Checking the code, we could see {{synchronized (this)}} in line 343 is 
> synchronizing on the instance of the subclass 
> {{AuthenticationTokenSecretManager$LeaderElector}} while {{lastKeyUpdate}} is 
> a variable of the parent class instance
> Will fix the issue in this JIRA



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warning in AuthenticationTokenSecretManager

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15148:
---
Summary: Resolve IS2_INCONSISTENT_SYNC findbugs warning in 
AuthenticationTokenSecretManager  (was: Resolve IS2_INCONSISTENT_SYNC findbugs 
warnings in AuthenticationTokenSecretManager)

> Resolve IS2_INCONSISTENT_SYNC findbugs warning in 
> AuthenticationTokenSecretManager
> --
>
> Key: HBASE-15148
> URL: https://issues.apache.org/jira/browse/HBASE-15148
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15148.patch, HBASE-15148.patch
>
>
> After efforts in HBASE-15118, we still see IS2_INCONSISTENT_SYNC warning in 
> AuthenticationTokenSecretManager in [HadoopQA report | 
> https://builds.apache.org/job/PreCommit-HBASE-Build/197/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html]
>  for HBASE-13960:
> {noformat}
> In class 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager
> Field 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.lastKeyUpdate
> Synchronized 50% of the time
> Unsynchronized access at AuthenticationTokenSecretManager.java:[line 343]
> Synchronized access at AuthenticationTokenSecretManager.java:[line 278]
> {noformat}
> Checking the code, we could see {{synchronized (this)}} in line 343 is 
> synchronizing on the instance of the subclass 
> {{AuthenticationTokenSecretManager$LeaderElector}} while {{lastKeyUpdate}} is 
> a variable of the parent class instance
> Will fix the issue in this JIRA



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14969) Add throughput controller for flush

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1511#comment-1511
 ] 

Hadoop QA commented on HBASE-14969:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 16 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
48s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
20s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
49s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 8s {color} 
| {color:red} hbase-server-jdk1.8.0_66 with JDK v1.8.0_66 generated 6 new 
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 40s {color} 
| {color:red} hbase-server-jdk1.7.0_91 with JDK v1.7.0_91 generated 6 new 
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s 
{color} | {color:red} Patch generated 6 new checkstyle issues in hbase-server 
(total was 159, now 164). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
1s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m 
23s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 82m 46s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 31s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 199m 41s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_91 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
\\
\\
|| Subsystem || Report/Notes ||
| 

[jira] [Commented] (HBASE-13960) HConnection stuck with UnknownHostException

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112235#comment-15112235
 ] 

Ted Yu commented on HBASE-13960:


I am fine with latest patch. 

[~apurtell]:
What do you think ?


> HConnection stuck with UnknownHostException 
> 
>
> Key: HBASE-13960
> URL: https://issues.apache.org/jira/browse/HBASE-13960
> Project: HBase
>  Issue Type: Bug
>  Components: hbase
>Affects Versions: 0.98.8
>Reporter: Kurt Young
>Assignee: Yu Li
> Attachments: HBASE-13960-0.98-v1.patch, HBASE-13960-update.patch, 
> HBASE-13960-update.v2.patch, HBASE-13960-v2.patch
>
>
> when put/get from hbase, if we meet a temporary dns failure causes resolve 
> RS's host, the error will never recovered. put/get will failed with 
> UnknownHostException forever. 
> I checked the code, and the reason maybe:
> 1. when RegionServerCallable or MultiServerCallable prepare(), it gets a  
> ClientService.BlockingInterface stub from Hconnection
> 2. In HConnectionImplementation::getClient, it caches the stub with a 
> BlockingRpcChannelImplementation
> 3. In BlockingRpcChannelImplementation(), 
>  this.isa = new InetSocketAddress(sn.getHostname(), sn.getPort()); If we 
> meet a  temporary dns failure then the "address" in isa will be null.
> 4. then we launch the real rpc call, the following stack is:
> Caused by: java.net.UnknownHostException: unknown host: xxx.host2
>   at 
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.(RpcClient.java:385)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1523)
>   at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1435)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1654)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1712)
> Besides, i noticed there is a protection in RpcClient:
> if (remoteId.getAddress().isUnresolved()) {
> throw new UnknownHostException("unknown host: " + 
> remoteId.getAddress().getHostName());
>   }
> shouldn't we do something when this situation occurred? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-15146:
--
Attachment: HBASE-15146.6.patch

> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch, HBASE-15146.3.patch, HBASE-15146.4.patch, 
> HBASE-15146.5.patch, HBASE-15146.6.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of 
> different queues. Those queues have different capacity limits. Currently the 
> dispatch method can block trying to add the the blocking queue in any of the 
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15142) Procedure v2 - Basic WebUI listing the procedures

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113291#comment-15113291
 ] 

Ted Yu commented on HBASE-15142:


lgtm

> Procedure v2 - Basic WebUI listing the procedures
> -
>
> Key: HBASE-15142
> URL: https://issues.apache.org/jira/browse/HBASE-15142
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, UI
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15142-v0.patch, proc-webui.png
>
>
> Basic table showing the list of procedures 
> pending/in-execution/recently-completed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113280#comment-15113280
 ] 

Hudson commented on HBASE-13082:


SUCCESS: Integrated in HBase-1.3 #509 (See 
[https://builds.apache.org/job/HBase-1.3/509/])
Revert "HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (stack: 
rev a5228a0b4d8b4c6b9bab58e1eee015393edaaade)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactedHFilesDischarger.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischargeHandler.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischarger.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java


> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, 
> HBASE-13082_12.patch, HBASE-13082_13.patch, HBASE-13082_14.patch, 
> HBASE-13082_15.patch, HBASE-13082_16.patch, HBASE-13082_17.patch, 
> HBASE-13082_18.patch, HBASE-13082_19.patch, HBASE-13082_1_WIP.patch, 
> HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, 
> HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, 
> HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, 
> LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait 

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113281#comment-15113281
 ] 

Hudson commented on HBASE-14970:


SUCCESS: Integrated in HBase-1.3 #509 (See 
[https://builds.apache.org/job/HBase-1.3/509/])
Revert "HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (stack: 
rev a5228a0b4d8b4c6b9bab58e1eee015393edaaade)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactedHFilesDischarger.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischargeHandler.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischarger.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java


> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15147) Shell should use Admin.listTableNames() instead of Admin.listTables()

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113243#comment-15113243
 ] 

Andrew Purtell commented on HBASE-15147:


bq. Then we can do the stripping of information in HTD/HCD depending on perms 
in a follow up jira if needed. 

Earlier thinking was whitelisting of information in descriptors would be a 
burden to maintain so only principals with C or A perms should be allowed to 
see descriptors. Seeing table names is fine for any perms (as well as region 
names, etc., since anyone must be able to read META to accomplish anything). 

> Shell should use Admin.listTableNames() instead of Admin.listTables() 
> --
>
> Key: HBASE-15147
> URL: https://issues.apache.org/jira/browse/HBASE-15147
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: hbase-15147_v1.patch
>
>
> It seems that getTableDescriptors() in master checks for A and C permissions 
> while getTableNames() checks for any privilege on the table. The reasoning is 
> explained here: 
> https://issues.apache.org/jira/browse/HBASE-12564?focusedCommentId=14234504=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234504
>  
> We should change the shell command for {{list}} to use the getTableNames() 
> version because of this. Otherwise a user having only R or W cannot list the 
> table name. 
> This has been reported from a user here: 
> https://community.hortonworks.com/questions/10742/why-does-a-user-need-create-permission-for-list-co.html#comment-11000.
>  
> While we are at it, should we revisit the fact that you cannot get a table 
> descriptor if you have only R or W? It seems strange that you cannot even 
> know the CF names of a table that you can read from. I could not find info 
> about the "describe" privileges on SQL databases. However, if there are use 
> cases where Table descriptor might contain sensitive info, the current 
> semantics seems fine. cc [~apurtell] and [~mbertozzi]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113298#comment-15113298
 ] 

Hudson commented on HBASE-13082:


SUCCESS: Integrated in HBase-1.3-IT #454 (See 
[https://builds.apache.org/job/HBase-1.3-IT/454/])
Revert "HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (stack: 
rev a5228a0b4d8b4c6b9bab58e1eee015393edaaade)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischargeHandler.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactedHFilesDischarger.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischarger.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java


> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, 
> HBASE-13082_12.patch, HBASE-13082_13.patch, HBASE-13082_14.patch, 
> HBASE-13082_15.patch, HBASE-13082_16.patch, HBASE-13082_17.patch, 
> HBASE-13082_18.patch, HBASE-13082_19.patch, HBASE-13082_1_WIP.patch, 
> HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, 
> HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, 
> HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, 
> LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores 

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113299#comment-15113299
 ] 

Hudson commented on HBASE-14970:


SUCCESS: Integrated in HBase-1.3-IT #454 (See 
[https://builds.apache.org/job/HBase-1.3-IT/454/])
Revert "HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (stack: 
rev a5228a0b4d8b4c6b9bab58e1eee015393edaaade)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischargeHandler.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactedHFilesDischarger.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischarger.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java


> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warnings in AuthenticationTokenSecretManager

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112102#comment-15112102
 ] 

Yu Li commented on HBASE-15148:
---

In the latest report, from the log of the failed TestGenerateDelegationToken 
case:
{noformat}
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
{noformat}
I should be an env issue

In previous report, there are mainly two failures: 
TestRegionMergeTransactionOnCluster and TestStochasticLoadBalancer, and they 
are frequently flaked in recent runs like HBASE-15152.

Given the status, I believe the patch here is safe.

> Resolve IS2_INCONSISTENT_SYNC findbugs warnings in 
> AuthenticationTokenSecretManager
> ---
>
> Key: HBASE-15148
> URL: https://issues.apache.org/jira/browse/HBASE-15148
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15148.patch, HBASE-15148.patch
>
>
> After efforts in HBASE-15118, we still see IS2_INCONSISTENT_SYNC warning in 
> AuthenticationTokenSecretManager in [HadoopQA report | 
> https://builds.apache.org/job/PreCommit-HBASE-Build/197/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html]
>  for HBASE-13960:
> {noformat}
> In class 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager
> Field 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.lastKeyUpdate
> Synchronized 50% of the time
> Unsynchronized access at AuthenticationTokenSecretManager.java:[line 343]
> Synchronized access at AuthenticationTokenSecretManager.java:[line 278]
> {noformat}
> Checking the code, we could see {{synchronized (this)}} in line 343 is 
> synchronizing on the instance of the subclass 
> {{AuthenticationTokenSecretManager$LeaderElector}} while {{lastKeyUpdate}} is 
> a variable of the parent class instance
> Will fix the issue in this JIRA



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15150) Fix TestDurablity in branch-1.1

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112079#comment-15112079
 ] 

Hudson commented on HBASE-15150:


FAILURE: Integrated in HBase-1.1-JDK8 #1730 (See 
[https://builds.apache.org/job/HBase-1.1-JDK8/1730/])
HBASE-15150 Fix TestDurablity in branch-1.1 (Yu Li) (stack: rev 
15b1f4e0957fee77dc20525654e7c9619174517b)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
Revert "HBASE-15150 Fix TestDurablity in branch-1.1 (Yu Li)" Revert (stack: rev 
b3d1ee4b819ae9f4a7ccc9389c2b58090d3d)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
HBASE-15150 Fix TestDurablity in branch-1.1 (Yu Li) (stack: rev 
7ecba62c47563f0c37194e31281becf0cca8a4ce)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15150v2.branch-1.1.patch, HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Commented] (HBASE-15152) Automatically include prefix-tree module in MR jobs if present

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112080#comment-15112080
 ] 

Hudson commented on HBASE-15152:


FAILURE: Integrated in HBase-1.1-JDK8 #1730 (See 
[https://builds.apache.org/job/HBase-1.1-JDK8/1730/])
HBASE-15152 Automatically include prefix-tree module in MR jobs if (jmhsieh: 
rev 8c858c214a90f70df40d900d6f65c5bfe6dd7b65)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java


> Automatically include prefix-tree module in MR jobs if present
> --
>
> Key: HBASE-15152
> URL: https://issues.apache.org/jira/browse/HBASE-15152
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.3, 1.0.4, 0.98.18
>
> Attachments: hbase-15152.patch
>
>
> I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
> found in the YarnChildren processes.  
> {code}
> 2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
> hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> {code}
> This is related to HBASE-7434 and HBASE-7936 which address compile time 
> concerns.  This fix makes it so that jar inclusion is done at run time, and 
> continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warnings in AuthenticationTokenSecretManager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112088#comment-15112088
 ] 

Hadoop QA commented on HBASE-15148:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
22m 48s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 88m 28s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 53s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 212m 36s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests | 
hadoop.hbase.security.token.TestGenerateDelegationToken |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783746/HBASE-15148.patch |
| JIRA Issue | HBASE-15148 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| 

[jira] [Commented] (HBASE-15152) Automatically include prefix-tree module in MR jobs if present

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112075#comment-15112075
 ] 

Hudson commented on HBASE-15152:


FAILURE: Integrated in HBase-1.1-JDK7 #1643 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1643/])
HBASE-15152 Automatically include prefix-tree module in MR jobs if (jmhsieh: 
rev 8c858c214a90f70df40d900d6f65c5bfe6dd7b65)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java


> Automatically include prefix-tree module in MR jobs if present
> --
>
> Key: HBASE-15152
> URL: https://issues.apache.org/jira/browse/HBASE-15152
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.3, 1.0.4, 0.98.18
>
> Attachments: hbase-15152.patch
>
>
> I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
> found in the YarnChildren processes.  
> {code}
> 2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
> hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> {code}
> This is related to HBASE-7434 and HBASE-7936 which address compile time 
> concerns.  This fix makes it so that jar inclusion is done at run time, and 
> continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15150) Fix TestDurablity in branch-1.1

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112074#comment-15112074
 ] 

Hudson commented on HBASE-15150:


FAILURE: Integrated in HBase-1.1-JDK7 #1643 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1643/])
HBASE-15150 Fix TestDurablity in branch-1.1 (Yu Li) (stack: rev 
15b1f4e0957fee77dc20525654e7c9619174517b)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
Revert "HBASE-15150 Fix TestDurablity in branch-1.1 (Yu Li)" Revert (stack: rev 
b3d1ee4b819ae9f4a7ccc9389c2b58090d3d)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestDurability.java
HBASE-15150 Fix TestDurablity in branch-1.1 (Yu Li) (stack: rev 
7ecba62c47563f0c37194e31281becf0cca8a4ce)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15150v2.branch-1.1.patch, HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Commented] (HBASE-13259) mmap() based BucketCache IOEngine

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112116#comment-15112116
 ] 

ramkrishna.s.vasudevan commented on HBASE-13259:


I can work on rebasing this and getting this in. It should help us in our 
future work also. 

> mmap() based BucketCache IOEngine
> -
>
> Key: HBASE-13259
> URL: https://issues.apache.org/jira/browse/HBASE-13259
> Project: HBase
>  Issue Type: New Feature
>  Components: BlockCache
>Affects Versions: 0.98.10
>Reporter: Zee Chen
>Assignee: Zee Chen
>Priority: Critical
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, ioread-1.svg, 
> mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch
>
>
> Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data 
> from kernel space to user space. This is a good choice when the total working 
> set size is much bigger than the available RAM and the latency is dominated 
> by IO access. However, when the entire working set is small enough to fit in 
> the RAM, using mmap() (and subsequent memcpy()) to move data from kernel 
> space to user space is faster. I have run some short keyval gets tests and 
> the results indicate a reduction of 2%-7% of kernel CPU on my system, 
> depending on the load. On the gets, the latency histograms from mmap() are 
> identical to those from pread(), but peak throughput is close to 40% higher.
> This patch modifies ByteByfferArray to allow it to specify a backing file.
> Example for using this feature: set  hbase.bucketcache.ioengine to 
> mmap:/dev/shm/bucketcache.0 in hbase-site.xml.
> Attached perf measured CPU usage breakdown in flames graph.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warning in AuthenticationTokenSecretManager

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112333#comment-15112333
 ] 

Hudson commented on HBASE-15148:


SUCCESS: Integrated in HBase-1.2-IT #408 (See 
[https://builds.apache.org/job/HBase-1.2-IT/408/])
HBASE-15148 Resolve IS2_INCONSISTENT_SYNC findbugs warning in (tedyu: rev 
cd920940b736f1674780a4f8f5fd888023a3551e)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java


> Resolve IS2_INCONSISTENT_SYNC findbugs warning in 
> AuthenticationTokenSecretManager
> --
>
> Key: HBASE-15148
> URL: https://issues.apache.org/jira/browse/HBASE-15148
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15148.patch, HBASE-15148.patch
>
>
> After efforts in HBASE-15118, we still see IS2_INCONSISTENT_SYNC warning in 
> AuthenticationTokenSecretManager in [HadoopQA report | 
> https://builds.apache.org/job/PreCommit-HBASE-Build/197/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html]
>  for HBASE-13960:
> {noformat}
> In class 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager
> Field 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.lastKeyUpdate
> Synchronized 50% of the time
> Unsynchronized access at AuthenticationTokenSecretManager.java:[line 343]
> Synchronized access at AuthenticationTokenSecretManager.java:[line 278]
> {noformat}
> Checking the code, we could see {{synchronized (this)}} in line 343 is 
> synchronizing on the instance of the subclass 
> {{AuthenticationTokenSecretManager$LeaderElector}} while {{lastKeyUpdate}} is 
> a variable of the parent class instance
> Will fix the issue in this JIRA



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15129) Set default value for hbase.fs.tmp.dir rather than fully depend on hbase-default.xml

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112346#comment-15112346
 ] 

Hadoop QA commented on HBASE-15129:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m 
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
47s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 52s 
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 26s 
{color} | {color:red} hbase-server in master has 2 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 19s 
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 29s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 20s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 2s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 1s 
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 115m 40s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 10s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_91. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s 
{color} | {color:green} hbase-common in the patch passed with JDK v1.7.0_91. 

[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14841:
---
Attachment: HBASE-14841_3.patch

Updated patch. Now findIndex() does a look up using the BB itself and then if 
not found then does a copy and goes for the addEntry call. Adds support for 
future write path support with offheap cells in WAL.

> Allow Dictionary to work with BytebufferedCells
> ---
>
> Key: HBASE-14841
> URL: https://issues.apache.org/jira/browse/HBASE-14841
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-14841.patch, HBASE-14841_1.patch, 
> HBASE-14841_2.patch, HBASE-14841_3.patch
>
>
> This is part of HBASE-14832 where we need to ensure that while BBCells are 
> getting compacted the TagCompression part should be working with BBCells.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14841:
---
Status: Patch Available  (was: Open)

> Allow Dictionary to work with BytebufferedCells
> ---
>
> Key: HBASE-14841
> URL: https://issues.apache.org/jira/browse/HBASE-14841
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-14841.patch, HBASE-14841_1.patch, 
> HBASE-14841_2.patch, HBASE-14841_3.patch
>
>
> This is part of HBASE-14832 where we need to ensure that while BBCells are 
> getting compacted the TagCompression part should be working with BBCells.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14841) Allow Dictionary to work with BytebufferedCells

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14841:
---
Status: Open  (was: Patch Available)

> Allow Dictionary to work with BytebufferedCells
> ---
>
> Key: HBASE-14841
> URL: https://issues.apache.org/jira/browse/HBASE-14841
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver, Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-14841.patch, HBASE-14841_1.patch, 
> HBASE-14841_2.patch
>
>
> This is part of HBASE-14832 where we need to ensure that while BBCells are 
> getting compacted the TagCompression part should be working with BBCells.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15133) Data loss after compaction when a row has more than Integer.MAX_VALUE columns

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112332#comment-15112332
 ] 

Hudson commented on HBASE-15133:


SUCCESS: Integrated in HBase-1.2-IT #408 (See 
[https://builds.apache.org/job/HBase-1.2-IT/408/])
HBASE-15133 Data loss after compaction when a row has more than (tedyu: rev 
936203684fe2c4751d9e4d3aaf232bed1d896198)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java


> Data loss after compaction when a row has more than Integer.MAX_VALUE columns
> -
>
> Key: HBASE-15133
> URL: https://issues.apache.org/jira/browse/HBASE-15133
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15133-v1.patch, HBASE-15133.patch, 
> HBASE-15133.patch
>
>
> We have lost the data in our development environment when a row has more than 
> Integer.MAX_VALUE columns after compaction.
> I think the reason is type of StoreScanner's countPerRow is int.
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L67
> After changing the type to long, it seems to be fixed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15153) Apply checkFamilies addendum on increment to 1.1 and 1.0

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112345#comment-15112345
 ] 

Hadoop QA commented on HBASE-15153:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
50s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s 
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
25s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 27s 
{color} | {color:red} hbase-server in branch-1.1 has 80 extant Findbugs 
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 49s 
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s 
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 5m 
32s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1 
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
40s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 43s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 128m 11s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_66. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 128m 17s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
28s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 296m 39s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | 
hadoop.hbase.procedure.TestZKProcedureControllers |
| JDK v1.8.0_66 Timed out junit tests | 
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
| JDK v1.7.0_91 Failed junit tests | 
hadoop.hbase.regionserver.TestFailedAppendAndSync |
| JDK v1.7.0_91 Timed out junit tests | 

[jira] [Commented] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function creates region with wrong end key boundary

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113344#comment-15113344
 ] 

Ted Yu commented on HBASE-15125:


Can you prepare patch for branch-1 ?
{code}
The text leading up to this was:
--
|diff --git 
a/hbase-server/src/test/java/org/apache/hadoop/hbase/util/BaseTestHBaseFsck.java
 
b/hbase-server/src/test/java/org/apache/hadoop/hbase/util/BaseTestHBaseFsck.java
|index bbb6b53..35560f5 100644
|--- 
a/hbase-server/src/test/java/org/apache/hadoop/hbase/util/BaseTestHBaseFsck.java
|+++ 
b/hbase-server/src/test/java/org/apache/hadoop/hbase/util/BaseTestHBaseFsck.java
--
File to patch: ^C
{code}

> HBaseFsck's adoptHdfsOrphan function creates region with wrong end key 
> boundary
> ---
>
> Key: HBASE-15125
> URL: https://issues.apache.org/jira/browse/HBASE-15125
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15125-V001.patch, HBASE-15125-v002.patch, 
> HBASE-15125-v003.patch, HBASE-15125-v004.patch, HBASE-15125-v004.patch
>
>
> There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this 
> function will create a region,which want to cover all the orphan regions.But 
> the end key of this new region was set incorrectly.Correct region's boundary 
> should be [startKey,endKey),but this function create a region with boundary 
> of [startKey,endKey],this bug will leads to scan operation omit some data.
> I think we should create the region like bellow,
> // create new region on hdfs. move data into place.
> HRegionInfo hri = new HRegionInfo(template.getTableName(), 
> orphanRegionRange.getFirst(),
> Bytes.add(orphanRegionRange.getSecond(), new byte[1]));



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15100) Master WALProcs still never clean up

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113365#comment-15113365
 ] 

Elliott Clark commented on HBASE-15100:
---

I didn't know if Enis logs changed anything.

+1 lgtm

> Master WALProcs still never clean up
> 
>
> Key: HBASE-15100
> URL: https://issues.apache.org/jira/browse/HBASE-15100
> Project: HBase
>  Issue Type: Bug
>  Components: master, proc-v2
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Attachments: HBASE-15100-logs.zip, HBASE-15100-v0.patch, 
> TestWalProcedure.java, procs.log
>
>
> {code}
> bin/hdfs dfs -ls /hbase/MasterProcWALs | wc -l
> 218631
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15100) Master WALProcs still never clean up

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113340#comment-15113340
 ] 

Elliott Clark commented on HBASE-15100:
---

v0 looks good. Should we get this in so that any 1.2.0 rc testing would verify 
fixes ?

> Master WALProcs still never clean up
> 
>
> Key: HBASE-15100
> URL: https://issues.apache.org/jira/browse/HBASE-15100
> Project: HBase
>  Issue Type: Bug
>  Components: master, proc-v2
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Attachments: HBASE-15100-logs.zip, HBASE-15100-v0.patch, 
> TestWalProcedure.java, procs.log
>
>
> {code}
> bin/hdfs dfs -ls /hbase/MasterProcWALs | wc -l
> 218631
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15100) Master WALProcs still never clean up

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113351#comment-15113351
 ] 

Matteo Bertozzi commented on HBASE-15100:
-

sure, I was just waiting a +1

> Master WALProcs still never clean up
> 
>
> Key: HBASE-15100
> URL: https://issues.apache.org/jira/browse/HBASE-15100
> Project: HBase
>  Issue Type: Bug
>  Components: master, proc-v2
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Attachments: HBASE-15100-logs.zip, HBASE-15100-v0.patch, 
> TestWalProcedure.java, procs.log
>
>
> {code}
> bin/hdfs dfs -ls /hbase/MasterProcWALs | wc -l
> 218631
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14919) Infrastructure refactoring for In-Memory Flush

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113370#comment-15113370
 ] 

Hadoop QA commented on HBASE-14919:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 9s 
{color} | {color:red} Patch generated 6 new checkstyle issues in hbase-server 
(total was 81, now 66). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s 
{color} | {color:red} The patch has 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
24m 37s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m 49s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 93m 49s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_91. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 253m 48s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-22 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783896/HBASE-14919-V06.patch 
|
| JIRA Issue | HBASE-14919 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux dfe7a8961350 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 

[jira] [Commented] (HBASE-15153) Apply checkFamilies addendum on increment to 1.1 and 1.0

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113373#comment-15113373
 ] 

Hudson commented on HBASE-15153:


FAILURE: Integrated in HBase-1.1-JDK7 #1644 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1644/])
HBASE-15153 Apply checkFamilies addendum on increment to 1.1 and 1.0 (stack: 
rev aa0e492f8f42c1ef53a7966f447d58027c105481)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Apply checkFamilies addendum on increment to 1.1 and 1.0
> 
>
> Key: HBASE-15153
> URL: https://issues.apache.org/jira/browse/HBASE-15153
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15153.branch-1.1.patch, 15153v2.branch-1.1.patch, 
> 15153v3.branch-1.1.patch
>
>
> branch-1.1 and branch-1.0 need this:
> --- 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++ 
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -7377,6 +7377,7 @@ public class HRegion implements HeapSize, 
> PropagatingConfigurationObserver, Regi
>  checkReadOnly();
>  checkResources();
>  checkRow(mutation.getRow(), op.toString());
> +checkFamilies(mutation.getFamilyCellMap().keySet());
>  startRegionOperation(op);
>  this.writeRequestsCount.increment();
>  try {
> Its an addendum on HBASE-15091 so fix for this oversight is in branch-1.2+.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113393#comment-15113393
 ] 

Hadoop QA commented on HBASE-15146:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m 
2s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
32s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 51s 
{color} | {color:red} Patch generated 7 new checkstyle issues in hbase-client 
(total was 44, now 51). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 3m 59s 
{color} | {color:red} Patch generated 12 new checkstyle issues in hbase-server 
(total was 105, now 117). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
21m 54s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 44s {color} 
| {color:red} hbase-client in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 51s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 58s {color} 
| {color:red} hbase-client in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 59s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s 
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 53s {color} 
| {color:black} {color} 

[jira] [Updated] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function creates region with wrong end key boundary

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15125:
---
Attachment: HBASE-15125-v004.patch

Reattach for QA run.

> HBaseFsck's adoptHdfsOrphan function creates region with wrong end key 
> boundary
> ---
>
> Key: HBASE-15125
> URL: https://issues.apache.org/jira/browse/HBASE-15125
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15125-V001.patch, HBASE-15125-v002.patch, 
> HBASE-15125-v003.patch, HBASE-15125-v004.patch, HBASE-15125-v004.patch
>
>
> There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this 
> function will create a region,which want to cover all the orphan regions.But 
> the end key of this new region was set incorrectly.Correct region's boundary 
> should be [startKey,endKey),but this function create a region with boundary 
> of [startKey,endKey],this bug will leads to scan operation omit some data.
> I think we should create the region like bellow,
> // create new region on hdfs. move data into place.
> HRegionInfo hri = new HRegionInfo(template.getTableName(), 
> orphanRegionRange.getFirst(),
> Bytes.add(orphanRegionRange.getSecond(), new byte[1]));



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15137) CallTimeoutException should be grounds for PFFE

[Mikhail Antonov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113335#comment-15113335
 ] 

Mikhail Antonov commented on HBASE-15137:
-

checkstyle complained because I aligned my changes with existing codestyle 
(misformatted)

> CallTimeoutException should be grounds for PFFE
> ---
>
> Key: HBASE-15137
> URL: https://issues.apache.org/jira/browse/HBASE-15137
> Project: HBase
>  Issue Type: Bug
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HBASE-15137.mantonov.patch, HBASE-15137.patch
>
>
> If a region server is backed up enough that lots of calls are timing out then 
> we should think about treating a server as failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15153) Apply checkFamilies addendum on increment to 1.1 and 1.0

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113353#comment-15113353
 ] 

Hudson commented on HBASE-15153:


FAILURE: Integrated in HBase-1.1-JDK8 #1731 (See 
[https://builds.apache.org/job/HBase-1.1-JDK8/1731/])
HBASE-15153 Apply checkFamilies addendum on increment to 1.1 and 1.0 (stack: 
rev aa0e492f8f42c1ef53a7966f447d58027c105481)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Apply checkFamilies addendum on increment to 1.1 and 1.0
> 
>
> Key: HBASE-15153
> URL: https://issues.apache.org/jira/browse/HBASE-15153
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15153.branch-1.1.patch, 15153v2.branch-1.1.patch, 
> 15153v3.branch-1.1.patch
>
>
> branch-1.1 and branch-1.0 need this:
> --- 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++ 
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -7377,6 +7377,7 @@ public class HRegion implements HeapSize, 
> PropagatingConfigurationObserver, Regi
>  checkReadOnly();
>  checkResources();
>  checkRow(mutation.getRow(), op.toString());
> +checkFamilies(mutation.getFamilyCellMap().keySet());
>  startRegionOperation(op);
>  this.writeRequestsCount.increment();
>  try {
> Its an addendum on HBASE-15091 so fix for this oversight is in branch-1.2+.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function creates region with wrong end key boundary

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113362#comment-15113362
 ] 

Ted Yu commented on HBASE-15125:


{code}
Running org.apache.hadoop.hbase.util.TestHBaseFsckOneRS
Tests run: 41, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 313.319 sec - 
in org.apache.hadoop.hbase.util.TestHBaseFsckOneRS
Running org.apache.hadoop.hbase.util.TestHBaseFsckTwoRS
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.853 sec - in 
org.apache.hadoop.hbase.util.TestHBaseFsckTwoRS
{code}
Is it possible to move the new test from TestHBaseFsckOneRS to 
TestHBaseFsckTwoRS so that the runtime for the two tests balance ?

Thanks


> HBaseFsck's adoptHdfsOrphan function creates region with wrong end key 
> boundary
> ---
>
> Key: HBASE-15125
> URL: https://issues.apache.org/jira/browse/HBASE-15125
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15125-V001.patch, HBASE-15125-v002.patch, 
> HBASE-15125-v003.patch, HBASE-15125-v004.patch, HBASE-15125-v004.patch
>
>
> There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this 
> function will create a region,which want to cover all the orphan regions.But 
> the end key of this new region was set incorrectly.Correct region's boundary 
> should be [startKey,endKey),but this function create a region with boundary 
> of [startKey,endKey],this bug will leads to scan operation omit some data.
> I think we should create the region like bellow,
> // create new region on hdfs. move data into place.
> HRegionInfo hri = new HRegionInfo(template.getTableName(), 
> orphanRegionRange.getFirst(),
> Bytes.add(orphanRegionRange.getSecond(), new byte[1]));



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14919) Infrastructure refactoring for In-Memory Flush

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113417#comment-15113417
 ] 

Hadoop QA commented on HBASE-14919:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 5s 
{color} | {color:red} Patch generated 6 new checkstyle issues in hbase-server 
(total was 81, now 66). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 4s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 105m 27s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 99m 55s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_91. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
18s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 270m 55s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-22 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783896/HBASE-14919-V06.patch 
|
| JIRA Issue | HBASE-14919 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 6178dd24c1a9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 

[jira] [Comment Edited] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113576#comment-15113576
 ] 

Ted Yu edited comment on HBASE-9393 at 1/23/16 5:30 AM:


For branch-1,
{code}
+if (stream != null && stream.getWrappedStream() instanceof DFSInputStream) 
{
{code}
The instanceof check can be replaced with iterating 
stream.getWrappedStream().getClass().getInterfaces() and seeing if one of the 
interfaces is CanUnbuffer


was (Author: yuzhih...@gmail.com):
{code}
+if (stream != null && stream.getWrappedStream() instanceof DFSInputStream) 
{
{code}
The instanceof check can be replaced with iterating 
stream.getWrappedStream().getClass().getInterfaces() and seeing if one of the 
interfaces is CanUnbuffer

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, 
> HBASE-9393.v2.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-8676) hbase.hstore.compaction.max.size parameter has no effect in minor compaction

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-8676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113599#comment-15113599
 ] 

Anoop Sam John commented on HBASE-8676:
---

{quote}
hfile3 t3 100G
t1->t2>t3(older->newer),
{quote}
So hfile3, a newer one with bigger size, is a bulk loaded one?

> hbase.hstore.compaction.max.size parameter has no effect in minor compaction
> 
>
> Key: HBASE-8676
> URL: https://issues.apache.org/jira/browse/HBASE-8676
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.94.7
>Reporter: Calvin Qu
>
> In hbase-site.xml,I set(10G)
>   
> hbase.hstore.compaction.max.size
> 10737418240
>   
> but in regionserver's log,a minor compaction deals with 10 Hfiles,include a 
> 220G HFile larger than hbase.hstore.compaction.max.size=10G.
> 2013-06-03 10:27:51,147 DEBUG org.apache.hadoop.hbase.regionserver.Store: 
> 8b6a4d4aae3099730b353183cf754ec4 - t: Initiating minorcompaction
> 2013-06-03 10:27:51,149 INFO org.apache.hadoop.hbase.regionserver.HRegion: 
> Starting compaction on t in region 
> transdb,1651763699103243149223370692144515807,1369372490168.8b6a4d4aae3099730b353183cf754ec4.
> 2013-06-03 10:27:51,150 DEBUG 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction 
> requested: 
> regionName=transdb,1651763699103243149223370692144515807,1369372490168.8b6a4d4aae3099730b353183cf754ec4.,
>  storeName=t, fileCount=10, fileSize=234.4g (106.0m, 2.0g, 2.1g, 10.7m, 1.7g, 
> 2.1g, 1.9g, 2.0g, 932.8m, 221.5g), priority=-110, time=2671513455305347; 
> Because: Opening Region; compaction_queue=(0:0), split_queue=0
> 2013-06-03 10:27:51,151 INFO org.apache.hadoop.hbase.regionserver.Store: 
> Starting compaction of 10 file(s) in t of 
> transdb,1651763699103243149223370692144515807,1369372490168.8b6a4d4aae3099730b353183cf754ec4.
>  into 
> tmpdir=hdfs://namenode01:9000/hbase/transdb/8b6a4d4aae3099730b353183cf754ec4/.tmp,
>  seqid=0, totalSize=234.4g
> 2013-06-03 10:27:51,152 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
> Compacting 
> hdfs://namenode01:9000/hbase/transdb/8b6a4d4aae3099730b353183cf754ec4/t/0a5fb3caf6be490ba739b77fcaa4b274.c7ce1266ef1f8696ede82f08fceb28ff-hdfs://namenode01:9000/hbase/transdb/c7ce1266ef1f8696ede82f08fceb28ff/t/0a5fb3caf6be490ba739b77fcaa4b274-top,
>  keycount=7948080, bloomtype=NONE, size=106.0m, encoding=NONE
> 2013-06-03 10:27:51,153 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
> Compacting 
> hdfs://namenode01:9000/hbase/transdb/8b6a4d4aae3099730b353183cf754ec4/t/6bd849c93e3a45d4a4c8a5ab691f9b22.c7ce1266ef1f8696ede82f08fceb28ff-hdfs://namenode01:9000/hbase/transdb/c7ce1266ef1f8696ede82f08fceb28ff/t/6bd849c93e3a45d4a4c8a5ab691f9b22-top,
>  keycount=153911753, bloomtype=NONE, size=2.0g, encoding=NONE
> 2013-06-03 10:27:51,153 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
> Compacting 
> hdfs://namenode01:9000/hbase/transdb/8b6a4d4aae3099730b353183cf754ec4/t/191ef6d1076341e1b7fb5c775d0f9fdb.c7ce1266ef1f8696ede82f08fceb28ff-hdfs://namenode01:9000/hbase/transdb/c7ce1266ef1f8696ede82f08fceb28ff/t/191ef6d1076341e1b7fb5c775d0f9fdb-top,
>  keycount=159372725, bloomtype=NONE, size=2.1g, encoding=NONE
> 2013-06-03 10:27:51,154 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
> Compacting 
> hdfs://namenode01:9000/hbase/transdb/8b6a4d4aae3099730b353183cf754ec4/t/d907a7dca7d8466f843f126429f05665.c7ce1266ef1f8696ede82f08fceb28ff-hdfs://namenode01:9000/hbase/transdb/c7ce1266ef1f8696ede82f08fceb28ff/t/d907a7dca7d8466f843f126429f05665-top,
>  keycount=808800, bloomtype=NONE, size=10.7m, encoding=NONE
> 2013-06-03 10:27:51,154 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
> Compacting 
> hdfs://namenode01:9000/hbase/transdb/8b6a4d4aae3099730b353183cf754ec4/t/35399e73e3764d0b88251c7285cf5406.c7ce1266ef1f8696ede82f08fceb28ff-hdfs://namenode01:9000/hbase/transdb/c7ce1266ef1f8696ede82f08fceb28ff/t/35399e73e3764d0b88251c7285cf5406-top,
>  keycount=133146091, bloomtype=NONE, size=1.7g, encoding=NONE
> 2013-06-03 10:27:51,154 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
> Compacting 
> hdfs://namenode01:9000/hbase/transdb/8b6a4d4aae3099730b353183cf754ec4/t/9d5583ffde334bc79a692d400af7bae0.c7ce1266ef1f8696ede82f08fceb28ff-hdfs://namenode01:9000/hbase/transdb/c7ce1266ef1f8696ede82f08fceb28ff/t/9d5583ffde334bc79a692d400af7bae0-top,
>  keycount=159372708, bloomtype=NONE, size=2.1g, encoding=NONE
> 2013-06-03 10:27:51,155 DEBUG org.apache.hadoop.hbase.regionserver.Compactor: 
> Compacting 
> hdfs://namenode01:9000/hbase/transdb/8b6a4d4aae3099730b353183cf754ec4/t/506254eac1bb455283f578e71af23153.c7ce1266ef1f8696ede82f08fceb28ff-hdfs://namenode01:9000/hbase/transdb/c7ce1266ef1f8696ede82f08fceb28ff/t/506254eac1bb455283f578e71af23153-top,
>  keycount=152578349, 

[jira] [Commented] (HBASE-15100) Master WALProcs still never clean up

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113612#comment-15113612
 ] 

Hudson commented on HBASE-15100:


FAILURE: Integrated in HBase-1.2 #519 (See 
[https://builds.apache.org/job/HBase-1.2/519/])
HBASE-15100 Master WALProcs are deleted out of order ending up with 
(matteo.bertozzi: rev 8a34c14129758adcea62299dfaaeb193d0279758)
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
* 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java
* 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStore.java


> Master WALProcs still never clean up
> 
>
> Key: HBASE-15100
> URL: https://issues.apache.org/jira/browse/HBASE-15100
> Project: HBase
>  Issue Type: Bug
>  Components: master, proc-v2
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.1.3
>
> Attachments: HBASE-15100-logs.zip, HBASE-15100-v0.patch, 
> TestWalProcedure.java, procs.log
>
>
> {code}
> bin/hdfs dfs -ls /hbase/MasterProcWALs | wc -l
> 218631
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15100) Master WALProcs still never clean up

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113623#comment-15113623
 ] 

Hudson commented on HBASE-15100:


FAILURE: Integrated in HBase-1.1-JDK8 #1732 (See 
[https://builds.apache.org/job/HBase-1.1-JDK8/1732/])
HBASE-15100 Master WALProcs are deleted out of order ending up with 
(matteo.bertozzi: rev 20d5577d2e1b8e395e66be88a66a70bec8665f4f)
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
* 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStore.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java
* 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java


> Master WALProcs still never clean up
> 
>
> Key: HBASE-15100
> URL: https://issues.apache.org/jira/browse/HBASE-15100
> Project: HBase
>  Issue Type: Bug
>  Components: master, proc-v2
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.1.3
>
> Attachments: HBASE-15100-logs.zip, HBASE-15100-v0.patch, 
> TestWalProcedure.java, procs.log
>
>
> {code}
> bin/hdfs dfs -ls /hbase/MasterProcWALs | wc -l
> 218631
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14409) Clarify use of hbase.hstore.compaction.max.size in ExploringCompactionPolicy

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113621#comment-15113621
 ] 

Anoop Sam John commented on HBASE-14409:


See HBASE-15055

> Clarify use of hbase.hstore.compaction.max.size in ExploringCompactionPolicy
> 
>
> Key: HBASE-14409
> URL: https://issues.apache.org/jira/browse/HBASE-14409
> Project: HBase
>  Issue Type: Bug
>  Components: Compaction
>Affects Versions: 1.0.2, 1.1.2
>Reporter: Lars George
>Assignee: stack
>Priority: Critical
>
> As discussed in https://issues.apache.org/jira/browse/HBASE-7842:
> Why is the {{ExploringCompactionPolicy}} overloading the 
> {{hbase.hstore.compaction.max.size}} parameter, which is used in the original 
> ratio-based policy _just_ to exclude store files that are larger than this 
> threshold. The ECP does the same, but later on uses the same threshold (if 
> set) to drop a possible selection when the sum of all store files in the 
> selection exceeds this limit. Why?
> Here the code:
> {code}
> if (size > comConf.getMaxCompactSize()) {
>   continue;
> }
> {code}
> The ref guide says this:
> {noformat}
> * Do size-based sanity checks against each StoreFile in this set of 
> StoreFiles.
> ** If the size of this StoreFile is larger than 
> `hbase.hstore.compaction.max.size`, take it out of consideration.
> ** If the size is greater than or equal to 
> `hbase.hstore.compaction.min.size`, sanity-check it against the file-based 
> ratio to see whether it is too large to be considered.
> {noformat}
> This seems wrong, no? It does not do this by each store file, but by the 
> current selection candidate. It still speaks of the max size key, but here in 
> the traditional sense, i.e. eliminate single store files that exceed the 
> limit. But that is not what the code does at this spot. 
> We should either remove that check, since larger files are already removed in 
> {{selectCompaction()}} of the base class, or we should see what was meant to 
> happen here and clarify/fix the code and/or description.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113586#comment-15113586
 ] 

Anoop Sam John commented on HBASE-9393:
---

unbuffer support is added from 2.7.0 (See 
https://issues.apache.org/jira/browse/HDFS-7694)
So for older version based clusters, we can not solve this?

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch, 
> HBASE-9393.v2.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113604#comment-15113604
 ] 

Hadoop QA commented on HBASE-15132:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} master passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 16s 
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server 
(total was 123, now 124). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
21m 39s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m 20s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_66. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 108m 37s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_91. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
13s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 247m 20s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783972/HBASE-15132.v8.patch |
| JIRA Issue | HBASE-15132 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux b3aa04d663ca 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT 

[jira] [Commented] (HBASE-14916) Add checkstyle_report.py to other branches

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113624#comment-15113624
 ] 

Sean Busbey commented on HBASE-14916:
-

yep. presuming we're getting as much information out of the yetus checkstyle 
plugin as we were getting out of the report.

> Add checkstyle_report.py to other branches
> --
>
> Key: HBASE-14916
> URL: https://issues.apache.org/jira/browse/HBASE-14916
> Project: HBase
>  Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-14916-branch-1-v2.patch, 
> HBASE-14916-branch-1-v3.patch, HBASE-14916-branch-1.patch
>
>
> Given test-patch.sh is always run from master, and that it now uses 
> checkstyle_report.py, we should pull back the script to other branches too.
> Otherwise we see error like: 
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/jenkins.build/dev-support/test-patch.sh:
>  line 662: 
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/dev-support/checkstyle_report.py:
>  No such file or directory
> [reference|https://builds.apache.org/job/PreCommit-HBASE-Build/16734//consoleFull]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15097) When the scan operation covered two regions,sometimes the final results have duplicated rows.

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113578#comment-15113578
 ] 

Anoop Sam John commented on HBASE-15097:


bq.,I said no,if we set the correct stop row,then it will not happen,because 
the error will not happen at the last region.
I mean when the user specified Scan do not have any stop row.  Then as per the 
patch also, we wont be setting the stop row.  In that case will it be possible 
to get a row out of it boundary from this region?

> When the scan operation covered two regions,sometimes the final results have 
> duplicated rows.
> -
>
> Key: HBASE-15097
> URL: https://issues.apache.org/jira/browse/HBASE-15097
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 1.1.2
> Environment: centos 6.5
> hbase 1.1.2 
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15097-v001.patch, HBASE-15097-v002.patch, 
> HBASE-15097-v003.patch, HBASE-15097-v004.patch, output.log, rowkey.txt, 
> snapshot2016-01-13 pm 8.42.37.png
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When the scan operation‘s start key and end key covered two regions,the first 
> region returned the rows which were beyond of its' end key.So,this finally 
> leads to duplicated rows in the results.
> To avoid this problem,we should add a judgment before setting the variable 
> "stopRow" in the class of HRegion,like follow:
> if (Bytes.equals(scan.getStopRow(), HConstants.EMPTY_END_ROW) && 
> !scan.isGetScan()) {
> this.stopRow = null;
> } else {
> if (Bytes.compareTo(scan.getStopRow(), 
> this.getRegionInfo().getEndKey()) >= 0) {
> this.stopRow = this.getRegionInfo().getEndKey();
> } else {
> this.stopRow = scan.getStopRow();
> }
> }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15100) Master WALProcs still never clean up

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113609#comment-15113609
 ] 

Hudson commented on HBASE-15100:


FAILURE: Integrated in HBase-1.3 #510 (See 
[https://builds.apache.org/job/HBase-1.3/510/])
HBASE-15100 Master WALProcs are deleted out of order ending up with 
(matteo.bertozzi: rev 931e1b03ab67182e1f60ca497b7f99e3b2b65d33)
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
* 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormatReader.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/ProcedureInfo.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
* 
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStore.java
* 
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java


> Master WALProcs still never clean up
> 
>
> Key: HBASE-15100
> URL: https://issues.apache.org/jira/browse/HBASE-15100
> Project: HBase
>  Issue Type: Bug
>  Components: master, proc-v2
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Matteo Bertozzi
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.1.3
>
> Attachments: HBASE-15100-logs.zip, HBASE-15100-v0.patch, 
> TestWalProcedure.java, procs.log
>
>
> {code}
> bin/hdfs dfs -ls /hbase/MasterProcWALs | wc -l
> 218631
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113592#comment-15113592
 ] 

ramkrishna.s.vasudevan commented on HBASE-14970:


While committing I commited the _8 version of the patch I think. I can see that 
the commit does not have the fix for TestHfileOutputFormat. _9 version of the 
patch actually solved that problem and the QA build was without the 
TestHfileOutputFormat issue. Can I go ahead and recommit _9 version of the 
patch?

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15055) Major compaction is not triggered when both of TTL and hbase.hstore.compaction.max.size are set

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113618#comment-15113618
 ] 

Anoop Sam John commented on HBASE-15055:


Patch looks good now.

bq.// Test depends on this not being set to pass.  Default breaks test.
bq.82   this.conf.setLong("hbase.hstore.compaction.min.size", minSize);
Why is this change in test?

> Major compaction is not triggered when both of TTL and 
> hbase.hstore.compaction.max.size are set
> ---
>
> Key: HBASE-15055
> URL: https://issues.apache.org/jira/browse/HBASE-15055
> Project: HBase
>  Issue Type: Bug
>Reporter: Eungsop Yoo
>Assignee: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-15055-v1.patch, HBASE-15055-v10.patch, 
> HBASE-15055-v11.patch, HBASE-15055-v2.patch, HBASE-15055-v3.patch, 
> HBASE-15055-v4.patch, HBASE-15055-v5.patch, HBASE-15055-v6.patch, 
> HBASE-15055-v7.patch, HBASE-15055-v8.patch, HBASE-15055-v9.patch, 
> HBASE-15055.patch
>
>
> Some large files may be skipped by hbase.hstore.compaction.max.size in 
> candidate selection. It causes skipping of major compaction. So the TTL 
> expired records are still remained in the disks and keep consuming disks.
> To resolve this issue, I suggest that to skip large files only if there is no 
> TTL expired record.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15113613#comment-15113613
 ] 

ramkrishna.s.vasudevan commented on HBASE-14970:


Ping [~saint@gmail.com]

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd*

[stack (JIRA)]

stack created HBASE-15159:
-

 Summary: Fix merge of MVCC and SequenceID performance regression 
in branch-1.0 for checkAnd*
 Key: HBASE-15159
 URL: https://issues.apache.org/jira/browse/HBASE-15159
 Project: HBase
  Issue Type: Sub-task
Reporter: stack


Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for 
checkAnd* too in branch-1. Should be quick enough to do.  Only prereq would be 
HBASE-15157, tooling we have to do anyways, so we can show we've made 
improvement. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112690#comment-15112690
 ] 

stack commented on HBASE-14460:
---

Hey [~bbeaudreault] I made HBASE-15159 a subtask here. I don't think it'd be 
too hard making the branch-1 hack for increments work for checkAnd*... let me 
take a look. Doing it on a per table or per column-family basis might be tough. 
Let me report back on HBASE-15159.

> [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed 
> Increments, CheckAndPuts, batch operations
> -
>
> Key: HBASE-14460
> URL: https://issues.apache.org/jira/browse/HBASE-14460
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 0.94.test.patch, 0.98.test.patch, 
> 1.0.80.flamegraph-7932.svg, 14460.txt, 14460.v0.branch-1.0.patch, 
> 98.80.flamegraph-11428.svg, HBASE-14460-discussion.patch, client.test.patch, 
> flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg, 
> flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg, 
> flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg, 
> hack.flamegraph-16593.svg, hack.uncommitted.patch, m.test.patch, 
> region_lock.png, testincrement.094.patch, testincrement.098.patch, 
> testincrement.master.patch
>
>
> As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation 
> between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of 
> sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to 
> 'catch up' to our current point before we can read the last Increment value 
> that we need to update.
> We can say that our Increment is just done wrong, we should just be writing 
> Increments and summing on read, but checkAndPut as well as batching 
> operations have the same issue. Fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15150) Fix TestDurablity in branch-1.1

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112695#comment-15112695
 ] 

Hudson commented on HBASE-15150:


SUCCESS: Integrated in HBase-1.0 #1137 (See 
[https://builds.apache.org/job/HBase-1.0/1137/])
HBASE-15150 Fix TestDurablity in branch-1.1 (Yu Li) (stack: rev 
64e22626adc20162cfb5da71f0fc68797a888fa4)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15150v2.branch-1.1.patch, HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Commented] (HBASE-15152) Automatically include prefix-tree module in MR jobs if present

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112696#comment-15112696
 ] 

Hudson commented on HBASE-15152:


SUCCESS: Integrated in HBase-1.0 #1137 (See 
[https://builds.apache.org/job/HBase-1.0/1137/])
HBASE-15152 Automatically include prefix-tree module in MR jobs if (jmhsieh: 
rev 24002e23623472d9e6d5e5a63c834eb442873bf1)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java


> Automatically include prefix-tree module in MR jobs if present
> --
>
> Key: HBASE-15152
> URL: https://issues.apache.org/jira/browse/HBASE-15152
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.3, 1.0.4, 0.98.18
>
> Attachments: hbase-15152.patch
>
>
> I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
> found in the YarnChildren processes.  
> {code}
> 2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
> hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> {code}
> This is related to HBASE-7434 and HBASE-7936 which address compile time 
> concerns.  This fix makes it so that jar inclusion is done at run time, and 
> continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15147) Shell should use Admin.listTableNames() instead of Admin.listTables()

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112697#comment-15112697
 ] 

Hudson commented on HBASE-15147:


SUCCESS: Integrated in HBase-1.0 #1137 (See 
[https://builds.apache.org/job/HBase-1.0/1137/])
HBASE-15147 Shell should use Admin.listTableNames() instead of (enis: rev 
886c70d0d95b95ddd928cd5bc1e1fc83b1de2f42)
* hbase-shell/src/main/ruby/hbase/admin.rb


> Shell should use Admin.listTableNames() instead of Admin.listTables() 
> --
>
> Key: HBASE-15147
> URL: https://issues.apache.org/jira/browse/HBASE-15147
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: hbase-15147_v1.patch
>
>
> It seems that getTableDescriptors() in master checks for A and C permissions 
> while getTableNames() checks for any privilege on the table. The reasoning is 
> explained here: 
> https://issues.apache.org/jira/browse/HBASE-12564?focusedCommentId=14234504=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234504
>  
> We should change the shell command for {{list}} to use the getTableNames() 
> version because of this. Otherwise a user having only R or W cannot list the 
> table name. 
> This has been reported from a user here: 
> https://community.hortonworks.com/questions/10742/why-does-a-user-need-create-permission-for-list-co.html#comment-11000.
>  
> While we are at it, should we revisit the fact that you cannot get a table 
> descriptor if you have only R or W? It seems strange that you cannot even 
> know the CF names of a table that you can read from. I could not find info 
> about the "describe" privileges on SQL databases. However, if there are use 
> cases where Table descriptor might contain sensitive info, the current 
> semantics seems fine. cc [~apurtell] and [~mbertozzi]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112701#comment-15112701
 ] 

Ted Yu commented on HBASE-9393:
---

{code}
1772  boolean useHBaseChecksum = 
this.streamWrapper.shouldUseHBaseChecksum();
1773  final FSDataInputStream stream = 
this.streamWrapper.getStream(useHBaseChecksum);
1774  if (stream.getWrappedStream() instanceof DFSInputStream) {
1775stream.unbuffer();
1776  }
{code}
The above code is repeated. Suggest refactoring into a method.

Please run PE tool for other read actions.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15152) Automatically include prefix-tree module in MR jobs if present

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112708#comment-15112708
 ] 

stack commented on HBASE-15152:
---

Thanks for looking at failing tests [~jmhsieh]

I think I should pull stochastic load balancer again... till someone takes a 
look at it. Will keep an eye on it.



> Automatically include prefix-tree module in MR jobs if present
> --
>
> Key: HBASE-15152
> URL: https://issues.apache.org/jira/browse/HBASE-15152
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.3, 1.0.4, 0.98.18
>
> Attachments: hbase-15152.patch
>
>
> I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
> found in the YarnChildren processes.  
> {code}
> 2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
> hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> {code}
> This is related to HBASE-7434 and HBASE-7936 which address compile time 
> concerns.  This fix makes it so that jar inclusion is done at run time, and 
> continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15023) Reenable TestShell and TestStochasticLoadBalancer

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112711#comment-15112711
 ] 

stack commented on HBASE-15023:
---

TestStochasticBalancer is reported failing on master over in HBASE-15152.

Next time it hangs, going to revert TestStochasticBalancer again.

> Reenable TestShell and TestStochasticLoadBalancer
> -
>
> Key: HBASE-15023
> URL: https://issues.apache.org/jira/browse/HBASE-15023
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 15023.patch, 15023.patch
>
>
> Parent disabled these tests when test runs were flakier than they are now. 
> Try reenabling them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15152) Automatically include prefix-tree module in MR jobs if present

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112723#comment-15112723
 ] 

Hudson commented on HBASE-15152:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1162 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1162/])
HBASE-15152 Automatically include prefix-tree module in MR jobs if (jmhsieh: 
rev 6187bea711521063abfcd0b228e31b622ebfa0ce)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java


> Automatically include prefix-tree module in MR jobs if present
> --
>
> Key: HBASE-15152
> URL: https://issues.apache.org/jira/browse/HBASE-15152
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.3, 1.0.4, 0.98.18
>
> Attachments: hbase-15152.patch
>
>
> I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
> found in the YarnChildren processes.  
> {code}
> 2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
> hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> {code}
> This is related to HBASE-7434 and HBASE-7936 which address compile time 
> concerns.  This fix makes it so that jar inclusion is done at run time, and 
> continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112727#comment-15112727
 ] 

Ashish Singhi commented on HBASE-9393:
--

bq. Please run PE tool for other read actions.
Can you suggest which one's to run? I will run them now.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112731#comment-15112731
 ] 

Ted Yu commented on HBASE-9393:
---

How about the following:

randomSeekScan
randomRead
sequentialRead
filterScan

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112733#comment-15112733
 ] 

Ashish Singhi commented on HBASE-9393:
--

Ok. Let me try.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch, HBASE-9393.v1.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15157) Add *PerformanceTest for Append, CheckAnd*

[Jean-Marc Spaggiari (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112409#comment-15112409
 ] 

Jean-Marc Spaggiari commented on HBASE-15157:
-

Big +1 for that! Also very interested to see an increment option. It end up 
being a chekandput, but might have some slight differences that we might want 
to get tested...

> Add *PerformanceTest for Append, CheckAnd*
> --
>
> Key: HBASE-15157
> URL: https://issues.apache.org/jira/browse/HBASE-15157
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance, test
>Reporter: stack
>
> In a sibling issue we add IncrementPerformanceTest which is handy checking 
> Increment performance. Make a general tool to do all check-and-set type ops 
> like increment, append, checkAndPut, checkAndDelete... just to make sure no 
> regression in future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15158) Change order in which we do write pipeline operations; do all under row locks!

[Jean-Marc Spaggiari (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112407#comment-15112407
 ] 

Jean-Marc Spaggiari commented on HBASE-15158:
-

Does this impact the multi-wall performances?

> Change order in which we do write pipeline operations; do all under row locks!
> --
>
> Key: HBASE-15158
> URL: https://issues.apache.org/jira/browse/HBASE-15158
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 15158.patch
>
>
> Change how we do our write pipeline. I want to do all write pipeline ops 
> under row lock so I lean on this fact fixing performance regression in 
> check-and-set type operations like increment, append, and checkAnd* (see 
> sibling issue HBASE-15082).
> To be specific, we write like this now:
> {code}
> # take rowlock
> # start mvcc
> # append to WAL
> # add to memstore
> # let go of rowlock
> # sync WAL
> # in case of error: rollback memstore
> {code}
> Instead, write like this:
> {code}
> # take rowlock
> # start mvcc
> # append to WAL
> # sync WAL
> # add to memstore
> # let go of rowlock
> ... no need to do rollback.
> {code}
> The old ordering was put in place because it got better performance in a time 
> when WAL was different and before row locks were read/write (HBASE-12751).
> Testing in branch-1 shows that a reordering and skipping mvcc waits gets us 
> back to the performance we had before we unified mvcc and sequenceid 
> (HBASE-8763). Tests in HBASE-15046 show that at the macro level using our 
> usual perf tools, reordering pipeline seems to cause no slowdown (see 
> HBASE-15046). A rough compare of increments with reordered write pipeline 
> seems to have us getting back a bunch of our performance (see tail of 
> https://issues.apache.org/jira/browse/HBASE-15082?focusedCommentId=15111703=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15111703
>  and subsequent comment).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14969) Add throughput controller for flush

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112438#comment-15112438
 ] 

Duo Zhang commented on HBASE-14969:
---

[~apurtell] In this patch we move and remove some classes marked as 
{{IA.CONFIG}} and map the old class name to new class name in Factory 
class(which means we are still compatible with old config file). Does this 
break our compatibility assumptions? Thanks.

> Add throughput controller for flush
> ---
>
> Key: HBASE-14969
> URL: https://issues.apache.org/jira/browse/HBASE-14969
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-14969.branch-1.patch, HBASE-14969.patch, 
> HBASE-14969_v10.patch, HBASE-14969_v2.patch, HBASE-14969_v3.patch, 
> HBASE-14969_v4.patch, HBASE-14969_v5.patch, HBASE-14969_v6.patch, 
> HBASE-14969_v9.patch, fd78628_9909808_compat_report.html, 
> load-nothrottling.log, load-throttling.log
>
>
> In HBASE-8329 we added a throughput controller for compaction, to avoid spike 
> caused by huge IO pressure like network/disk overflow. However, even with 
> this control on, we are still observing disk utils near 100%, and by analysis 
> we think this is caused by flush, especially when we increase the setting of 
> {{hbase.hstore.flusher.count}}
> In this JIRA, we propose to add throughput control feature for flush, as a 
> supplement of HBASE-8329 to better control IO pressure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warning in AuthenticationTokenSecretManager

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112444#comment-15112444
 ] 

Hudson commented on HBASE-15148:


FAILURE: Integrated in HBase-1.3 #508 (See 
[https://builds.apache.org/job/HBase-1.3/508/])
HBASE-15148 Resolve IS2_INCONSISTENT_SYNC findbugs warning in (tedyu: rev 
37d15e05d8858b9b3dc60f6251e29dd428f07e7f)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java


> Resolve IS2_INCONSISTENT_SYNC findbugs warning in 
> AuthenticationTokenSecretManager
> --
>
> Key: HBASE-15148
> URL: https://issues.apache.org/jira/browse/HBASE-15148
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15148.patch, HBASE-15148.patch
>
>
> After efforts in HBASE-15118, we still see IS2_INCONSISTENT_SYNC warning in 
> AuthenticationTokenSecretManager in [HadoopQA report | 
> https://builds.apache.org/job/PreCommit-HBASE-Build/197/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html]
>  for HBASE-13960:
> {noformat}
> In class 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager
> Field 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.lastKeyUpdate
> Synchronized 50% of the time
> Unsynchronized access at AuthenticationTokenSecretManager.java:[line 343]
> Synchronized access at AuthenticationTokenSecretManager.java:[line 278]
> {noformat}
> Checking the code, we could see {{synchronized (this)}} in line 343 is 
> synchronizing on the instance of the subclass 
> {{AuthenticationTokenSecretManager$LeaderElector}} while {{lastKeyUpdate}} is 
> a variable of the parent class instance
> Will fix the issue in this JIRA



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)