[jira] [Updated] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-15146:
--
Attachment: HBASE-15146.2.patch

Don't throw, and make sure to decrement the size.

> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15128) Disable region splits and merges in HBCK

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110500#comment-15110500
 ] 

Hadoop QA commented on HBASE-15128:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 5s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
58s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s 
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 24s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 24s {color} | 
{color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 24s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 22s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79. 
{color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 22s {color} | 
{color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 22s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 3s 
{color} | {color:red} Patch generated 6 new checkstyle issues in hbase-client 
(total was 473, now 478). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 11s 
{color} | {color:red} Patch generated 7 new checkstyle issues in hbase-server 
(total was 339, now 345). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 20m 57s 
{color} | {color:blue} Rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 20m 57s 
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
21m 55s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
54s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s 
{color} | {color:red} hbase-server introduced 5 new FindBugs issues. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 28s 
{color} | {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 5 new 
issues (was 1, now 6). 

[jira] [Updated] (HBASE-15150) Fix TestDurablity in branch-1.1

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-15150:
--
Attachment: HBASE-15150.branch-1.1.patch

A straight forward patch

> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Created] (HBASE-15150) Fix TestDurablity in branch-1.1

[Yu Li (JIRA)]

Yu Li created HBASE-15150:
-

 Summary: Fix TestDurablity in branch-1.1
 Key: HBASE-15150
 URL: https://issues.apache.org/jira/browse/HBASE-15150
 Project: HBase
  Issue Type: Bug
Reporter: Yu Li
Assignee: Yu Li


>From [UT 
>report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
> of HBASE-15120 against branch-1.1, we could see below failure:
{noformat}
testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
  Time elapsed: 0.111 sec  <<< FAILURE!
java.lang.AssertionError: expected null, but 

[jira] [Commented] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warnings in AuthenticationTokenSecretManager

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110489#comment-15110489
 ] 

Yu Li commented on HBASE-15148:
---

>From the console output:
{noformat}


 Patch findbugs detection




cd 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/hbase-server
/home/jenkins/tools/maven/latest/bin/mvn 
-Dmaven.repo.local=/home/jenkins/yetus-m2/hbase-master-1 -DHBasePatchProcess 
test-compile findbugs:findbugs -DskipTests=true > 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/patchprocess/patch-findbugs-hbase-server.txt
 2>&1
Elapsed:   2m 50s
Starting with 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/patchprocess/branch-findbugs-hbase-server-warnings.xml
Merging 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/patchprocess/patch-findbugs-hbase-server-warnings.xml
Writing 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/patchprocess/combined-findbugs-hbase-server.xml
Found 0 new Findbugs warnings and 1 newly fixed warnings.
{noformat}
The findbugs issue got resolved with the patch. It seems the output of HadoopQA 
report has some problem.

Regarding the UT failures, none of them are introduced by change here.

> Resolve IS2_INCONSISTENT_SYNC findbugs warnings in 
> AuthenticationTokenSecretManager
> ---
>
> Key: HBASE-15148
> URL: https://issues.apache.org/jira/browse/HBASE-15148
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15148.patch
>
>
> After efforts in HBASE-15118, we still see IS2_INCONSISTENT_SYNC warning in 
> AuthenticationTokenSecretManager in [HadoopQA report | 
> https://builds.apache.org/job/PreCommit-HBASE-Build/197/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html]
>  for HBASE-13960:
> {noformat}
> In class 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager
> Field 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.lastKeyUpdate
> Synchronized 50% of the time
> Unsynchronized access at AuthenticationTokenSecretManager.java:[line 343]
> Synchronized access at AuthenticationTokenSecretManager.java:[line 278]
> {noformat}
> Checking the code, we could see {{synchronized (this)}} in line 343 is 
> synchronizing on the instance of the subclass 
> {{AuthenticationTokenSecretManager$LeaderElector}} while {{lastKeyUpdate}} is 
> a variable of the parent class instance
> Will fix the issue in this JIRA



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110546#comment-15110546
 ] 

Ashish Singhi commented on HBASE-9393:
--

[~anoop.hbase] and [~ramkrishna.s.vasude...@gmail.com] thanks for comment.
I think we are purposefully doing pread here as we are passing value false to 
the HFileBlock read. For meta and small scan I see pread is set to true. When 
replacing the unbuffer with pread in our internal test we found that there was 
a performance degrade of about 9% and with the attached patch there was a 
slight(4%) improvement!! with below commands,
{noformat}
hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestPerf 
--presplit=134 --rows=100 randomWrite 14
hbase org.apache.hadoop.hbase.PerformanceEvaluation --table=TestPerf 
--rows=100 randomRead 14
{noformat}

bq. Or we can just make sure this unbuffer() call happens at the end of the 
scan on this HFile. (When StoreFileScanner done with its Cells for the scan and 
getting closed)
Yes I am working on it, on first try I saw that CLOSE_WAIT are not getting 
closed, I need to dig more in it to find out what is the difference it makes to 
close the socket in close call or immediately after the read. I will come up 
soon with the result.

[~tedyu], Thanks a lot for checking the test failure and for the help on the 
same.

[~vinayrpet], Thanks, will add a check that if stream is instance of 
DFSInputStream then only call unbuffer().

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15109) Region server failed to start when "fs.hdfs.impl.disable.cache" is set to true

[Pankaj Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110566#comment-15110566
 ] 

Pankaj Kumar commented on HBASE-15109:
--

Two approach can be adopted here when fs.hdfs.impl.disable.cache is set true,
- During installation, skip suppressing HDFS shutdownhook.
{code}
boolean disabledFsCache = 
fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
if (!disabledFsCache && 
!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
!ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
  throw new RuntimeException("Failed suppression of fs shutdown hook: " 
+
hdfsClientFinalizer);
}
{code}

   OR

- we can disable shutdownhook feature completely by having check in 
HRegionServer when fs.hdfs.impl.disable.cache is set true ,
{code}
boolean disabledFsCache = 
fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
if(!disabledFsCache){
ShutdownHook.install(conf, fs, this, Thread.currentThread());
}
{code}


> Region server failed to start when "fs.hdfs.impl.disable.cache" is set to true
> --
>
> Key: HBASE-15109
> URL: https://issues.apache.org/jira/browse/HBASE-15109
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 1.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
>
> Region server failed to start during installing ShutdownHook when 
> "fs.hdfs.impl.disable.cache" is set to true in core-site.xml at HBase side.
> {code}
> 2016-01-14 15:30:56,358 | FATAL | regionserver/ds2/192.168.152.230:21302 | 
> ABORTING region server ds2,21302,1452756654352: Unhandled: Failed suppression 
> of fs shutdown hook: 
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@571527d6 | 
> org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2054)
> java.lang.RuntimeException: Failed suppression of fs shutdown hook: 
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@571527d6
> at 
> org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:204)
> at 
> org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:84)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:893)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> During installation, first time it will try to suppress the HDFS shutdownhook 
> by removing the hdfsclientfinalizer from HDFS. Since 
> fs.hdfs.impl.disable.cache is enabled, so removal will be unsuccessful from 
> HDFS  (FS is not cached) and RuntimeException will be thrown with "Failed 
> suppression of fs shutdown hook" message.
> In ShutdownHook,
> {code}
> if (!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
> !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> throw new RuntimeException("Failed suppression of fs shutdown hook: " +
> hdfsClientFinalizer);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15109) Region server failed to start when "fs.hdfs.impl.disable.cache" is set to true

[Pankaj Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pankaj Kumar updated HBASE-15109:
-
Status: Patch Available  (was: Open)

> Region server failed to start when "fs.hdfs.impl.disable.cache" is set to true
> --
>
> Key: HBASE-15109
> URL: https://issues.apache.org/jira/browse/HBASE-15109
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 1.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Attachments: HBASE-15109-branch-1.patch
>
>
> Region server failed to start during installing ShutdownHook when 
> "fs.hdfs.impl.disable.cache" is set to true in core-site.xml at HBase side.
> {code}
> 2016-01-14 15:30:56,358 | FATAL | regionserver/ds2/192.168.152.230:21302 | 
> ABORTING region server ds2,21302,1452756654352: Unhandled: Failed suppression 
> of fs shutdown hook: 
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@571527d6 | 
> org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2054)
> java.lang.RuntimeException: Failed suppression of fs shutdown hook: 
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@571527d6
> at 
> org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:204)
> at 
> org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:84)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:893)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> During installation, first time it will try to suppress the HDFS shutdownhook 
> by removing the hdfsclientfinalizer from HDFS. Since 
> fs.hdfs.impl.disable.cache is enabled, so removal will be unsuccessful from 
> HDFS  (FS is not cached) and RuntimeException will be thrown with "Failed 
> suppression of fs shutdown hook" message.
> In ShutdownHook,
> {code}
> if (!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
> !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> throw new RuntimeException("Failed suppression of fs shutdown hook: " +
> hdfsClientFinalizer);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15109) Region server failed to start when "fs.hdfs.impl.disable.cache" is set to true

[Pankaj Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pankaj Kumar updated HBASE-15109:
-
Attachment: HBASE-15109-branch-1.patch

Patch with approach 1, please review.

> Region server failed to start when "fs.hdfs.impl.disable.cache" is set to true
> --
>
> Key: HBASE-15109
> URL: https://issues.apache.org/jira/browse/HBASE-15109
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 1.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Attachments: HBASE-15109-branch-1.patch
>
>
> Region server failed to start during installing ShutdownHook when 
> "fs.hdfs.impl.disable.cache" is set to true in core-site.xml at HBase side.
> {code}
> 2016-01-14 15:30:56,358 | FATAL | regionserver/ds2/192.168.152.230:21302 | 
> ABORTING region server ds2,21302,1452756654352: Unhandled: Failed suppression 
> of fs shutdown hook: 
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@571527d6 | 
> org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2054)
> java.lang.RuntimeException: Failed suppression of fs shutdown hook: 
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@571527d6
> at 
> org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:204)
> at 
> org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:84)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:893)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> During installation, first time it will try to suppress the HDFS shutdownhook 
> by removing the hdfsclientfinalizer from HDFS. Since 
> fs.hdfs.impl.disable.cache is enabled, so removal will be unsuccessful from 
> HDFS  (FS is not cached) and RuntimeException will be thrown with "Failed 
> suppression of fs shutdown hook" message.
> In ShutdownHook,
> {code}
> if (!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
> !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> throw new RuntimeException("Failed suppression of fs shutdown hook: " +
> hdfsClientFinalizer);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-15146:
--
Fix Version/s: 1.2.0
Affects Version/s: 1.2.0
   Status: Patch Available  (was: Open)

> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-15146:
--
Description: 
Blocking on the epoll thread is awful. The new rpc scheduler can have lots of 
different queues. Those queues have different capacity limits. Currently the 
dispatch method can block trying to add the the blocking queue in any of the 
schedulers.

This causes readers to block, tcp acks are delayed, and everything slows down.

> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of 
> different queues. Those queues have different capacity limits. Currently the 
> dispatch method can block trying to add the the blocking queue in any of the 
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14970:
---
Status: Open  (was: Patch Available)

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1_1.patch, HBASE-14970_branch-1_2.patch, 
> HBASE-14970_branch-1_4.patch, HBASE-14970_branch-1_5.patch, 
> HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110529#comment-15110529
 ] 

Anoop Sam John commented on HBASE-15132:


Implement new method in BaseMasterAndRegionObserver also so that don't have to 
do dummy impl in VisibilityController.
We don't want a post hook?
bq.Call {@link org.apache.hadoop.hbase.coprocessor.ObserverContext#bypass()} to 
skip the merge.
Doc says bypass is having effect but the code is not considering that.
{code}
if (User.isHBaseSecurityEnabled(master.getConfiguration())) {
531   User user = RpcServer.getRequestUser();
532   try {
533 user.getUGI().doAs(new PrivilegedExceptionAction() {
534   @Override
535   public Void run() throws Exception {
536 master.cpHost.preMerge(regionInfoA, regionInfoB);
537 return null;
538   }
539 });
{code}
When a hook is added, it must be called irrespective of security is enabled or 
not.
This flow will be executed by the normal RPC user who called merge() right? 
Still do we need this doAs way?


> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132.v1.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110559#comment-15110559
 ] 

Yu Li commented on HBASE-15120:
---

Checking the report, the UT failure is introduced by HBASE-15031 and have filed 
HBASE-15150 to fix it. The javadoc and findbugs issues are irrelative to the 
changes here.

> Backport HBASE-14883 to branch-1.1
> --
>
> Key: HBASE-15120
> URL: https://issues.apache.org/jira/browse/HBASE-15120
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Minor
> Attachments: HBASE-15120.branch-1.1.patch, 
> HBASE-15120.branch-1.1.patch
>
>
> When checking branch-1.1 UT in HBASE-13590, found 
> TestSplitTransactionOnCluster#testFailedSplit will fail at a 12/50 chance. 
> The issue is fixed by HBASE-14883 but the change didn't go into branch-1.1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14970:
---
Status: Patch Available  (was: Open)

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14970:
---
Attachment: HBASE-14970_9.branch-1.patch

This is the patch I will commit. The TestHFileOuputFormat was a failure but 
that class is unnecessary now considering that we already have 
TestHFileOutputFormat2.  The testHRegion seems to pass locally. And 
testSecureIPC seems to be an environement issue.

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110459#comment-15110459
 ] 

ramkrishna.s.vasudevan commented on HBASE-14970:


Forgot to add that TestHFileOutputFormat is also fixed in the _9 version of the 
patch.

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[CIS-CMMI-3] IllegalArgumentException: Row length 41221 is > 32767

[Kshitij Shukla]

Hello everyone,

Software stack is *nutch-branch-2.3.1, gora-hbase 0.6.1 Hadoop 2.5.2, 
hbase-0.98.8-hadoop2


* I have added a set of seeds to crawl using this command
*
./bin/crawl /largeSeeds 1 http://localhost:8983/solr/ddcd 4*

For first iteration all of the commands(*inject, **generate, **fetch, 
**parse, **update-table, **Indexer & delete duplicates.*) got executed 
successfully.
For second iteration, *"CrawlDB update" *command got failed (please see 
error log for reference), because of failure of this command the whole 
process gets terminated.



LOG 
START
16/01/20 02:45:19 INFO parse.ParserJob: ParserJob: finished at 
2016-01-20 02:45:19, time elapsed: 00:06:57

CrawlDB update for 1
/usr/share/searchEngine/nutch-branch-2.3.1/runtime/deploy/bin/nutch 
updatedb -D mapred.reduce.tasks=2 -D mapred.child.java.opts=-Xmx1000m -D 
mapred.reduce.tasks.speculative.execution=false -D 
mapred.map.tasks.speculative.execution=false -D 
mapred.compress.map.output=true 1453230757-13191 -crawlId 1
16/01/20 02:45:27 INFO crawl.DbUpdaterJob: DbUpdaterJob: starting at 
2016-01-20 02:45:27
16/01/20 02:45:27 INFO crawl.DbUpdaterJob: DbUpdaterJob: batchId: 
1453230757-13191
16/01/20 02:45:27 INFO plugin.PluginRepository: Plugins: looking in: 
/tmp/hadoop-root/hadoop-unjar5654418190157422003/classes/plugins
16/01/20 02:45:28 INFO plugin.PluginRepository: Plugin Auto-activation 
mode: [true]

16/01/20 02:45:28 INFO plugin.PluginRepository: Registered Plugins:
16/01/20 02:45:28 INFO plugin.PluginRepository: HTTP Framework 
(lib-http)
16/01/20 02:45:28 INFO plugin.PluginRepository: Html Parse Plug-in 
(parse-html)
16/01/20 02:45:28 INFO plugin.PluginRepository: MetaTags 
(parse-metatags)
16/01/20 02:45:28 INFO plugin.PluginRepository: the nutch core 
extension points (nutch-extensionpoints)
16/01/20 02:45:28 INFO plugin.PluginRepository: Basic Indexing 
Filter (index-basic)

16/01/20 02:45:28 INFO plugin.PluginRepository: XML Libraries (lib-xml)
16/01/20 02:45:28 INFO plugin.PluginRepository: Anchor Indexing 
Filter (index-anchor)
16/01/20 02:45:28 INFO plugin.PluginRepository: Basic URL Normalizer 
(urlnormalizer-basic)
16/01/20 02:45:28 INFO plugin.PluginRepository: Language 
Identification Parser/Filter (language-identifier)
16/01/20 02:45:28 INFO plugin.PluginRepository: Metadata Indexing 
Filter (index-metadata)
16/01/20 02:45:28 INFO plugin.PluginRepository: CyberNeko HTML 
Parser (lib-nekohtml)
16/01/20 02:45:28 INFO plugin.PluginRepository: Subcollection 
indexing and query filter (subcollection)
16/01/20 02:45:28 INFO plugin.PluginRepository: SOLRIndexWriter 
(indexer-solr)
16/01/20 02:45:28 INFO plugin.PluginRepository: Rel-Tag microformat 
Parser/Indexer/Querier (microformats-reltag)
16/01/20 02:45:28 INFO plugin.PluginRepository: Http / Https 
Protocol Plug-in (protocol-httpclient)
16/01/20 02:45:28 INFO plugin.PluginRepository: JavaScript Parser 
(parse-js)
16/01/20 02:45:28 INFO plugin.PluginRepository: Tika Parser Plug-in 
(parse-tika)
16/01/20 02:45:28 INFO plugin.PluginRepository: Top Level Domain 
Plugin (tld)
16/01/20 02:45:28 INFO plugin.PluginRepository: Regex URL Filter 
Framework (lib-regex-filter)
16/01/20 02:45:28 INFO plugin.PluginRepository: Regex URL Normalizer 
(urlnormalizer-regex)
16/01/20 02:45:28 INFO plugin.PluginRepository: Link Analysis 
Scoring Plug-in (scoring-link)
16/01/20 02:45:28 INFO plugin.PluginRepository: OPIC Scoring Plug-in 
(scoring-opic)
16/01/20 02:45:28 INFO plugin.PluginRepository: More Indexing Filter 
(index-more)
16/01/20 02:45:28 INFO plugin.PluginRepository: Http Protocol 
Plug-in (protocol-http)
16/01/20 02:45:28 INFO plugin.PluginRepository: Creative Commons 
Plugins (creativecommons)

16/01/20 02:45:28 INFO plugin.PluginRepository: Registered Extension-Points:
16/01/20 02:45:28 INFO plugin.PluginRepository: Parse Filter 
(org.apache.nutch.parse.ParseFilter)
16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch Index Cleaning 
Filter (org.apache.nutch.indexer.IndexCleaningFilter)
16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch Content Parser 
(org.apache.nutch.parse.Parser)
16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch URL Filter 
(org.apache.nutch.net.URLFilter)
16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch Scoring 
(org.apache.nutch.scoring.ScoringFilter)
16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch URL Normalizer 
(org.apache.nutch.net.URLNormalizer)
16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch Protocol 
(org.apache.nutch.protocol.Protocol)
16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch Index Writer 
(org.apache.nutch.indexer.IndexWriter)
16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch Indexing 
Filter 

[jira] [Updated] (HBASE-15128) Disable region splits and merges in HBCK

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-15128:
--
Attachment: HBASE-15128_v1.patch

add relates ruby scripts.  

> Disable region splits and merges in HBCK
> 
>
> Key: HBASE-15128
> URL: https://issues.apache.org/jira/browse/HBASE-15128
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Heng Chen
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15128.patch, HBASE-15128_v1.patch
>
>
> In large clusters where region splits are frequent, and HBCK runs take 
> longer, the concurrent splits cause further problems in HBCK since HBCK 
> assumes a static state for the region partition map. We have just seen a case 
> where HBCK undo's a concurrently splitting region causing number of 
> inconsistencies to go up. 
> We can have a mode in master where splits and merges are disabled like the 
> balancer and catalog janitor switches. Master will reject the split requests 
> if regionservers decide to split. This switch can be turned on / off by the 
> admins and also automatically by HBCK while it is running (similar to 
> balancer switch being disabled by HBCK). 
> HBCK  should also disable the Catalog Janitor just in case. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15150) Fix TestDurablity in branch-1.1

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-15150:
--
Status: Patch Available  (was: Open)

> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110402#comment-15110402
 ] 

Hadoop QA commented on HBASE-14970:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 17 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
22s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
45s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 11s 
{color} | {color:red} hbase-client in branch-1 has 7 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 30s 
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 4s {color} 
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was 
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 51s {color} 
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new 
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 16s 
{color} | {color:red} Patch generated 5 new checkstyle issues in hbase-server 
(total was 178, now 180). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
11m 20s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 18s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 37s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_79. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 48s {color} 
| {color:red} hbase-server in the patch failed with JDK 

[jira] [Commented] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function creates region with wrong end key boundary

[chenrongwei (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110250#comment-15110250
 ] 

chenrongwei commented on HBASE-15125:
-

I means maybe hbck did two steps to fix orphan regions. First create new region 
cover all orphan HFiles. Second,just fix meta data which not refers to any 
actual physical data if there still are holes in region chain. For example, the 
hole [cx\00,d) will be fixed by the next step operation fix meta.

> HBaseFsck's adoptHdfsOrphan function creates region with wrong end key 
> boundary
> ---
>
> Key: HBASE-15125
> URL: https://issues.apache.org/jira/browse/HBASE-15125
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15125-V001.patch, HBASE-15125-v002.patch, 
> HBASE-15125-v003.patch, HBASE-15125-v004.patch
>
>
> There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this 
> function will create a region,which want to cover all the orphan regions.But 
> the end key of this new region was set incorrectly.Correct region's boundary 
> should be [startKey,endKey),but this function create a region with boundary 
> of [startKey,endKey],this bug will leads to scan operation omit some data.
> I think we should create the region like bellow,
> // create new region on hdfs. move data into place.
> HRegionInfo hri = new HRegionInfo(template.getTableName(), 
> orphanRegionRange.getFirst(),
> Bytes.add(orphanRegionRange.getSecond(), new byte[1]));



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warnings in AuthenticationTokenSecretManager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110310#comment-15110310
 ] 

Hadoop QA commented on HBASE-15148:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m 
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
37s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 11s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 28s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 7s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 124m 28s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 10s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
18s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 277m 48s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Timed out junit tests | 
org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed |
| JDK v1.7.0_79 Failed junit tests | 
hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| JDK v1.7.0_79 Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
|   | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
|   | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 

[jira] [Commented] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function creates region with wrong end key boundary

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110353#comment-15110353
 ] 

Anoop Sam John commented on HBASE-15125:


+1

> HBaseFsck's adoptHdfsOrphan function creates region with wrong end key 
> boundary
> ---
>
> Key: HBASE-15125
> URL: https://issues.apache.org/jira/browse/HBASE-15125
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15125-V001.patch, HBASE-15125-v002.patch, 
> HBASE-15125-v003.patch, HBASE-15125-v004.patch
>
>
> There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this 
> function will create a region,which want to cover all the orphan regions.But 
> the end key of this new region was set incorrectly.Correct region's boundary 
> should be [startKey,endKey),but this function create a region with boundary 
> of [startKey,endKey],this bug will leads to scan operation omit some data.
> I think we should create the region like bellow,
> // create new region on hdfs. move data into place.
> HRegionInfo hri = new HRegionInfo(template.getTableName(), 
> orphanRegionRange.getFirst(),
> Bytes.add(orphanRegionRange.getSecond(), new byte[1]));



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Vinayakumar B (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110247#comment-15110247
 ] 

Vinayakumar B commented on HBASE-9393:
--

I think those test failures, are because of local file system is being used to 
read/write HFiles, gives {{BufferedFSInputStream}} which doesnt implement 
CanUnBuffer.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set incorrect 'storesFirstKey'

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110403#comment-15110403
 ] 

Hadoop QA commented on HBASE-15126:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m 
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 42s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m 
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
32m 4s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 149m 11s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 146m 12s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_79. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
33s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 359m 25s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | hadoop.hbase.quotas.TestQuotaThrottle |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783507/HBASE-15126-v006.patch
 |
| JIRA Issue | HBASE-15126 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (HBASE-15120) Backport HBASE-14883 to branch-1.1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110277#comment-15110277
 ] 

Hadoop QA commented on HBASE-15120:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
25s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s 
{color} | {color:green} branch-1.1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
16s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s 
{color} | {color:red} hbase-server in branch-1.1 has 79 extant Findbugs 
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 35s 
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
35s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
10m 50s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
52s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 35s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 44s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 21s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
23s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 215m 2s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | hadoop.hbase.master.TestTableLockManager |
|   | hadoop.hbase.util.TestHBaseFsck |
|   | hadoop.hbase.regionserver.TestFailedAppendAndSync |
|   | hadoop.hbase.regionserver.wal.TestDurability |
| JDK v1.8.0 Timed out junit tests | 
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
| JDK v1.7.0_79 Failed junit tests | 
hadoop.hbase.regionserver.wal.TestDurability |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783510/HBASE-15120.branch-1.1.patch
 |
| JIRA Issue | HBASE-15120 |
| Optional Tests |  

[jira] [Commented] (HBASE-14918) In-Memory MemStore Flush and Compaction

[Eshcar Hillel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110274#comment-15110274
 ] 

Eshcar Hillel commented on HBASE-14918:
---

I submitted a patch in task 1 two days ago but didn't receive any QA report 
since.
Any problems with the QA system?

> In-Memory MemStore Flush and Compaction
> ---
>
> Key: HBASE-14918
> URL: https://issues.apache.org/jira/browse/HBASE-14918
> Project: HBase
>  Issue Type: Umbrella
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Fix For: 0.98.18
>
>
> A memstore serves as the in-memory component of a store unit, absorbing all 
> updates to the store. From time to time these updates are flushed to a file 
> on disk, where they are compacted (by eliminating redundancies) and 
> compressed (i.e., written in a compressed format to reduce their storage 
> size).
> We aim to speed up data access, and therefore suggest to apply in-memory 
> memstore flush. That is to flush the active in-memory segment into an 
> intermediate buffer where it can be accessed by the application. Data in the 
> buffer is subject to compaction and can be stored in any format that allows 
> it to take up smaller space in RAM. The less space the buffer consumes the 
> longer it can reside in memory before data is flushed to disk, resulting in 
> better performance.
> Specifically, the optimization is beneficial for workloads with 
> medium-to-high key churn which incur many redundant cells, like persistent 
> messaging. 
> We suggest to structure the solution as 4 subtasks (respectively, patches). 
> (1) Infrastructure - refactoring of the MemStore hierarchy, introducing 
> segment (StoreSegment) as first-class citizen, and decoupling memstore 
> scanner from the memstore implementation;
> (2) Adding StoreServices facility at the region level to allow memstores 
> update region counters and access region level synchronization mechanism;
> (3) Implementation of a new memstore (CompactingMemstore) with non-optimized 
> immutable segment representation, and 
> (4) Memory optimization including compressed format representation and off 
> heap allocations.
> This Jira continues the discussion in HBASE-13408.
> Design documents, evaluation results and previous patches can be found in 
> HBASE-13408. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110341#comment-15110341
 ] 

Hadoop QA commented on HBASE-15082:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 18 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 39s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 48s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 12m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
57s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 16s 
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 16s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s 
{color} | {color:red} hbase-examples in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 39s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 19s 
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-common 
(total was 97, now 99). {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 5m 5s 
{color} | {color:red} Patch generated 58 new checkstyle issues in hbase-server 
(total was 863, now 834). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
36m 0s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 38s 
{color} | {color:red} hbase-server introduced 2 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 2s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 9m 16s 
{color} | {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 3 
new issues (was 1, now 4). {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 36s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 33s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 43s 
{color} | {color:green} hbase-common in the patch passed with JDK v1.8.0. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 50s 
{color} | {color:green} hbase-examples in the patch passed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 55s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. 

[jira] [Commented] (HBASE-15019) Replication stuck when HDFS is restarted

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110361#comment-15110361
 ] 

Hadoop QA commented on HBASE-15019:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 6s 
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server 
(total was 17, now 18). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
20m 52s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 23s 
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 49s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 209m 27s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_79 Failed junit tests | 
hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783523/HBASE-15019-v3.patch |
| JIRA Issue | HBASE-15019 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT 

[jira] [Commented] (HBASE-15125) HBaseFsck's adoptHdfsOrphan function creates region with wrong end key boundary

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110352#comment-15110352
 ] 

Anoop Sam John commented on HBASE-15125:


bq. will be fixed by the next step operation fix meta.
Then it is fine.

> HBaseFsck's adoptHdfsOrphan function creates region with wrong end key 
> boundary
> ---
>
> Key: HBASE-15125
> URL: https://issues.apache.org/jira/browse/HBASE-15125
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Attachments: HBASE-15125-V001.patch, HBASE-15125-v002.patch, 
> HBASE-15125-v003.patch, HBASE-15125-v004.patch
>
>
> There is a bug in HBaseFsck's adoptHdfsOrphan function.At the last of this 
> function will create a region,which want to cover all the orphan regions.But 
> the end key of this new region was set incorrectly.Correct region's boundary 
> should be [startKey,endKey),but this function create a region with boundary 
> of [startKey,endKey],this bug will leads to scan operation omit some data.
> I think we should create the region like bellow,
> // create new region on hdfs. move data into place.
> HRegionInfo hri = new HRegionInfo(template.getTableName(), 
> orphanRegionRange.getFirst(),
> Bytes.add(orphanRegionRange.getSecond(), new byte[1]));



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15019) Replication stuck when HDFS is restarted

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-15019:

Attachment: HBASE-15019-v4.patch

-04

  - fix checkstyle error

> Replication stuck when HDFS is restarted
> 
>
> Key: HBASE-15019
> URL: https://issues.apache.org/jira/browse/HBASE-15019
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, wal
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15019-v0_branch-1.2.patch, HBASE-15019-v1.patch, 
> HBASE-15019-v1_0.98.patch, HBASE-15019-v1_branch-1.2.patch, 
> HBASE-15019-v2.patch, HBASE-15019-v3.patch, HBASE-15019-v4.patch
>
>
> RS is normally working and writing on the WAL.
> HDFS is killed and restarted, and the RS try to do a roll.
> The close fail, but the roll succeed (because hdfs is now up) and everything 
> works.
> {noformat}
> 2015-12-11 21:52:28,058 ERROR 
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter: Got IOException 
> while writing trailer
> java.io.IOException: All datanodes 10.51.30.152:50010 are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1147)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:945)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:496)
> 2015-12-11 21:52:28,059 ERROR 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Failed close of HLog writer
> java.io.IOException: All datanodes 10.51.30.152:50010 are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1147)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:945)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:496)
> 2015-12-11 21:52:28,059 WARN org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
> Riding over HLog close failure! error count=1
> {noformat}
> The problem is on the replication side. that log we rolled and we were not 
> able to close
> is waiting for a lease recovery.
> {noformat}
> 2015-12-11 21:16:31,909 ERROR 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory: Can't open after 267 
> attempts and 301124ms 
> {noformat}
> the WALFactory notify us about that, but there is nothing on the RS side that 
> perform the WAL recovery.
> {noformat}
> 2015-12-11 21:11:30,921 WARN 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory: Lease should have 
> recovered. This is not expected. Will retry
> java.io.IOException: Cannot obtain block length for 
> LocatedBlock{BP-1547065147-10.51.30.152-1446756937665:blk_1073801614_61243; 
> getBlockSize()=83; corrupt=false; offset=0; locs=[10.51.30.154:50010, 
> 10.51.30.152:50010, 10.51.30.155:50010]}
>   at 
> org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:358)
>   at 
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:300)
>   at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:237)
>   at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:230)
>   at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1448)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:301)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:297)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:297)
>   at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
>   at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:116)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:89)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:77)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:68)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:508)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:321)
> {noformat}
> the only way to trigger a WAL recovery is to restart and force the master to 
> trigger the lease recovery on WAL split. 
> but there is a case where restarting will not help. If the RS keeps going 
> rolling and 

[jira] [Updated] (HBASE-15152) Automatically include prefix-tree module in MR jobs if present

[Jonathan Hsieh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-15152:
---
Summary: Automatically include prefix-tree module in MR jobs if present  
(was: Automatically include prefix-tree module in MR jobs.)

> Automatically include prefix-tree module in MR jobs if present
> --
>
> Key: HBASE-15152
> URL: https://issues.apache.org/jira/browse/HBASE-15152
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.3.0, 1.2.1
>
> Attachments: hbase-15152.patch
>
>
> I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
> found in the YarnChildren processes.  
> {code}
> 2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
> hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> {code}
> This is related to HBASE-7434 and HBASE-7936 which address compile time 
> concerns.  This fix makes it so that jar inclusion is done at run time, and 
> continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15124) Document the new 'normalization' feature in refguid

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-15124:

Fix Version/s: (was: 1.2.0)
   1.2.1
   1.3.0
   2.0.0

> Document the new 'normalization' feature in refguid
> ---
>
> Key: HBASE-15124
> URL: https://issues.apache.org/jira/browse/HBASE-15124
> Project: HBase
>  Issue Type: Task
>  Components: documentation
>Reporter: stack
>Assignee: Mikhail Antonov
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1
>
>
> A nice new feature is coming in to 1.2.0, normalization. A small bit of doc 
> on it in refguide would help.
> Should define what normalization is.
> Should say a sentence or two on how it works and when.
> Throw in the output of shell commands.
> A paragraph or so. I can help.
> Marking critical against 1.2.0. Not a blocker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15152) Automatically include prefix-tree module in MR jobs.

[Jonathan Hsieh (JIRA)]

Jonathan Hsieh created HBASE-15152:
--

 Summary: Automatically include prefix-tree module in MR jobs.
 Key: HBASE-15152
 URL: https://issues.apache.org/jira/browse/HBASE-15152
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Affects Versions: 1.0.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
 Fix For: 2.0.0, 1.3.0, 1.2.1


I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
found in the YarnChildren processes.  

{code}
2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
java.lang.ClassNotFoundException: 
org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
{code}

This is related to HBASE-7434 and HBASE-7936 which address compile time 
concerns.  This fix makes it so that jar inclusion is done at run time, and 
continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set incorrect 'storesFirstKey'

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110799#comment-15110799
 ] 

Ted Yu commented on HBASE-15126:


{code}
Failed tests: 
  
TestReplicationKillSlaveRS.killOneSlaveRS:34->TestReplicationKillRS.loadTableAndKillRS:88
 Waited too much time for queueFailover replication. Waited 18130ms.
{code}
The above test failure was not related to patch.
TestQuotaThrottle failure was not related either.

> HBaseFsck's checkRegionBoundaries function set incorrect 'storesFirstKey'
> -
>
> Key: HBASE-15126
> URL: https://issues.apache.org/jira/browse/HBASE-15126
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch, 
> HBASE-15126-v003.patch, HBASE-15126-v004.patch, HBASE-15126-v005.patch, 
> HBASE-15126-v006.patch
>
>
> HBaseFsck's checkRegionBoundaries function set the 
> 'currentRegionBoundariesInformation.storesFirstKey'  was incorrect.I think it 
> should be set like below,
> currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey);
> but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just 
> set to 'storeFirstKey',which will cause to 
> comparator.compare(currentRegionBoundariesInformation.storesFirstKey,
> currentRegionBoundariesInformation.metaFirstKey)  get a wrong 
> result.  Because it just compared the rowkey's length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110673#comment-15110673
 ] 

Sean Busbey commented on HBASE-15146:
-

please ping me when this goes in.

> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of 
> different queues. Those queues have different capacity limits. Currently the 
> dispatch method can block trying to add the the blocking queue in any of the 
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15126) HBaseFsck's checkRegionBoundaries function set incorrect 'storesFirstKey'

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15126:
---
Flags: Patch  (was: Patch,Important)
 Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
   2.0.0

> HBaseFsck's checkRegionBoundaries function set incorrect 'storesFirstKey'
> -
>
> Key: HBASE-15126
> URL: https://issues.apache.org/jira/browse/HBASE-15126
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch, 
> HBASE-15126-v003.patch, HBASE-15126-v004.patch, HBASE-15126-v005.patch, 
> HBASE-15126-v006.patch
>
>
> HBaseFsck's checkRegionBoundaries function set the 
> 'currentRegionBoundariesInformation.storesFirstKey'  was incorrect.I think it 
> should be set like below,
> currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey);
> but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just 
> set to 'storeFirstKey',which will cause to 
> comparator.compare(currentRegionBoundariesInformation.storesFirstKey,
> currentRegionBoundariesInformation.metaFirstKey)  get a wrong 
> result.  Because it just compared the rowkey's length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14918) In-Memory MemStore Flush and Compaction

[Eshcar Hillel (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110790#comment-15110790
 ] 

Eshcar Hillel commented on HBASE-14918:
---

I went through the code in HBASE-10713, and it seems we can come up with a 
design for task #4 of a compacted memstore which stores the data in a flat 
format (in the issue they are called CellBlocks) instead of in java skip-list. 
[~anoop.hbase] would you be interested to collaborate on this? If you are, we 
can schedule an off-list chat to discuss the details of the design.

> In-Memory MemStore Flush and Compaction
> ---
>
> Key: HBASE-14918
> URL: https://issues.apache.org/jira/browse/HBASE-14918
> Project: HBase
>  Issue Type: Umbrella
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
> Fix For: 0.98.18
>
>
> A memstore serves as the in-memory component of a store unit, absorbing all 
> updates to the store. From time to time these updates are flushed to a file 
> on disk, where they are compacted (by eliminating redundancies) and 
> compressed (i.e., written in a compressed format to reduce their storage 
> size).
> We aim to speed up data access, and therefore suggest to apply in-memory 
> memstore flush. That is to flush the active in-memory segment into an 
> intermediate buffer where it can be accessed by the application. Data in the 
> buffer is subject to compaction and can be stored in any format that allows 
> it to take up smaller space in RAM. The less space the buffer consumes the 
> longer it can reside in memory before data is flushed to disk, resulting in 
> better performance.
> Specifically, the optimization is beneficial for workloads with 
> medium-to-high key churn which incur many redundant cells, like persistent 
> messaging. 
> We suggest to structure the solution as 4 subtasks (respectively, patches). 
> (1) Infrastructure - refactoring of the MemStore hierarchy, introducing 
> segment (StoreSegment) as first-class citizen, and decoupling memstore 
> scanner from the memstore implementation;
> (2) Adding StoreServices facility at the region level to allow memstores 
> update region counters and access region level synchronization mechanism;
> (3) Implementation of a new memstore (CompactingMemstore) with non-optimized 
> immutable segment representation, and 
> (4) Memory optimization including compressed format representation and off 
> heap allocations.
> This Jira continues the discussion in HBASE-13408.
> Design documents, evaluation results and previous patches can be found in 
> HBASE-13408. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15126) HBaseFsck's checkRegionBoundaries function sets incorrect 'storesFirstKey'

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15126:
---
Summary: HBaseFsck's checkRegionBoundaries function sets incorrect 
'storesFirstKey'  (was: HBaseFsck's checkRegionBoundaries function set 
incorrect 'storesFirstKey')

> HBaseFsck's checkRegionBoundaries function sets incorrect 'storesFirstKey'
> --
>
> Key: HBASE-15126
> URL: https://issues.apache.org/jira/browse/HBASE-15126
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch, 
> HBASE-15126-v003.patch, HBASE-15126-v004.patch, HBASE-15126-v005.patch, 
> HBASE-15126-v006.patch
>
>
> HBaseFsck's checkRegionBoundaries function set the 
> 'currentRegionBoundariesInformation.storesFirstKey'  was incorrect.I think it 
> should be set like below,
> currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey);
> but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just 
> set to 'storeFirstKey',which will cause to 
> comparator.compare(currentRegionBoundariesInformation.storesFirstKey,
> currentRegionBoundariesInformation.metaFirstKey)  get a wrong 
> result.  Because it just compared the rowkey's length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110653#comment-15110653
 ] 

Hadoop QA commented on HBASE-15146:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 46s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 6s 
{color} | {color:red} Patch generated 12 new checkstyle issues in hbase-server 
(total was 105, now 117). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
20m 45s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 12s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 3s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
8s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 29s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | hadoop.hbase.ipc.TestAsyncIPC |
|   | hadoop.hbase.ipc.TestIPC |
| JDK v1.7.0_79 Failed junit tests | hadoop.hbase.ipc.TestAsyncIPC |
|   | hadoop.hbase.ipc.TestIPC |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783577/HBASE-15146.2.patch |
| JIRA Issue | HBASE-15146 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  

[jira] [Updated] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14970:
---
   Resolution: Fixed
Fix Version/s: 1.3.0
   Status: Resolved  (was: Patch Available)

Pushed to branch-1  (1.3). Thanks for the reviews [~anoopsamjohn].

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warnings in AuthenticationTokenSecretManager

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110683#comment-15110683
 ] 

Sean Busbey commented on HBASE-15148:
-

The findbugs statement says

{quote}
-1  findbugs4m 11s  hbase-server in master has 1 extant Findbugs 
warnings.
+1  findbugs3m 17s  the patch passed
{quote}

which means "there was 1 finbugs warning but the patch doesn't add any new 
ones", which is correct.

Resubmitted to verify UT failures.

> Resolve IS2_INCONSISTENT_SYNC findbugs warnings in 
> AuthenticationTokenSecretManager
> ---
>
> Key: HBASE-15148
> URL: https://issues.apache.org/jira/browse/HBASE-15148
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15148.patch
>
>
> After efforts in HBASE-15118, we still see IS2_INCONSISTENT_SYNC warning in 
> AuthenticationTokenSecretManager in [HadoopQA report | 
> https://builds.apache.org/job/PreCommit-HBASE-Build/197/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html]
>  for HBASE-13960:
> {noformat}
> In class 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager
> Field 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.lastKeyUpdate
> Synchronized 50% of the time
> Unsynchronized access at AuthenticationTokenSecretManager.java:[line 343]
> Synchronized access at AuthenticationTokenSecretManager.java:[line 278]
> {noformat}
> Checking the code, we could see {{synchronized (this)}} in line 343 is 
> synchronizing on the instance of the subclass 
> {{AuthenticationTokenSecretManager$LeaderElector}} while {{lastKeyUpdate}} is 
> a variable of the parent class instance
> Will fix the issue in this JIRA



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15147) Shell should use Admin.listTableNames() instead of Admin.listTables()

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110690#comment-15110690
 ] 

Sean Busbey commented on HBASE-15147:
-

+1 no need for a new test while changing implementation of extant functionality.

> Shell should use Admin.listTableNames() instead of Admin.listTables() 
> --
>
> Key: HBASE-15147
> URL: https://issues.apache.org/jira/browse/HBASE-15147
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>
> Attachments: hbase-15147_v1.patch
>
>
> It seems that getTableDescriptors() in master checks for A and C permissions 
> while getTableNames() checks for any privilege on the table. The reasoning is 
> explained here: 
> https://issues.apache.org/jira/browse/HBASE-12564?focusedCommentId=14234504=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234504
>  
> We should change the shell command for {{list}} to use the getTableNames() 
> version because of this. Otherwise a user having only R or W cannot list the 
> table name. 
> This has been reported from a user here: 
> https://community.hortonworks.com/questions/10742/why-does-a-user-need-create-permission-for-list-co.html#comment-11000.
>  
> While we are at it, should we revisit the fact that you cannot get a table 
> descriptor if you have only R or W? It seems strange that you cannot even 
> know the CF names of a table that you can read from. I could not find info 
> about the "describe" privileges on SQL databases. However, if there are use 
> cases where Table descriptor might contain sensitive info, the current 
> semantics seems fine. cc [~apurtell] and [~mbertozzi]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15148) Resolve IS2_INCONSISTENT_SYNC findbugs warnings in AuthenticationTokenSecretManager

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110719#comment-15110719
 ] 

Yu Li commented on HBASE-15148:
---

I see, thanks for the clarification Sean!

> Resolve IS2_INCONSISTENT_SYNC findbugs warnings in 
> AuthenticationTokenSecretManager
> ---
>
> Key: HBASE-15148
> URL: https://issues.apache.org/jira/browse/HBASE-15148
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15148.patch
>
>
> After efforts in HBASE-15118, we still see IS2_INCONSISTENT_SYNC warning in 
> AuthenticationTokenSecretManager in [HadoopQA report | 
> https://builds.apache.org/job/PreCommit-HBASE-Build/197/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html]
>  for HBASE-13960:
> {noformat}
> In class 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager
> Field 
> org.apache.hadoop.hbase.security.token.AuthenticationTokenSecretManager.lastKeyUpdate
> Synchronized 50% of the time
> Unsynchronized access at AuthenticationTokenSecretManager.java:[line 343]
> Synchronized access at AuthenticationTokenSecretManager.java:[line 278]
> {noformat}
> Checking the code, we could see {{synchronized (this)}} in line 343 is 
> synchronizing on the instance of the subclass 
> {{AuthenticationTokenSecretManager$LeaderElector}} while {{lastKeyUpdate}} is 
> a variable of the parent class instance
> Will fix the issue in this JIRA



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110646#comment-15110646
 ] 

Hadoop QA commented on HBASE-14970:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 18 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
29s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 16s 
{color} | {color:red} hbase-client in branch-1 has 7 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 19s 
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 1s {color} 
| {color:red} hbase-server-jdk1.8.0 with JDK v1.8.0 generated 6 new issues (was 
6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 55s {color} 
| {color:red} hbase-server-jdk1.7.0_79 with JDK v1.7.0_79 generated 6 new 
issues (was 6, now 6). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s 
{color} | {color:red} Patch generated 5 new checkstyle issues in hbase-server 
(total was 178, now 180). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
13m 39s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 15s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 43s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 44s 
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_79. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 98m 10s 
{color} | {color:green} hbase-server in the patch passed 

[jira] [Created] (HBASE-15151) turn off strict findbugs in precommit tests

[Sean Busbey (JIRA)]

Sean Busbey created HBASE-15151:
---

 Summary: turn off strict findbugs in precommit tests
 Key: HBASE-15151
 URL: https://issues.apache.org/jira/browse/HBASE-15151
 Project: HBase
  Issue Type: Task
  Components: build, test
Reporter: Sean Busbey


the "-1" for extant findbugs warnings has confused interpretation of our 
precommit checks enough that we should switch to non-strict mode.

It will still record the number of findbugs warnings present before the patch, 
but it'll vote "0" rather than calling attention to things via a -1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15152) Automatically include prefix-tree module in MR jobs.

[Jonathan Hsieh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-15152:
---
Attachment: hbase-15152.patch

> Automatically include prefix-tree module in MR jobs.
> 
>
> Key: HBASE-15152
> URL: https://issues.apache.org/jira/browse/HBASE-15152
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.3.0, 1.2.1
>
> Attachments: hbase-15152.patch
>
>
> I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
> found in the YarnChildren processes.  
> {code}
> 2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
> hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> {code}
> This is related to HBASE-7434 and HBASE-7936 which address compile time 
> concerns.  This fix makes it so that jar inclusion is done at run time, and 
> continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15126) HBaseFsck's checkRegionBoundaries function sets incorrect 'storesFirstKey'

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15126:
---
   Resolution: Fixed
Fix Version/s: (was: 1.3.0)
   Status: Resolved  (was: Patch Available)

Interesting, the bug actually doesn't exist in branch-1

Thanks for the patch, chenrongwei

> HBaseFsck's checkRegionBoundaries function sets incorrect 'storesFirstKey'
> --
>
> Key: HBASE-15126
> URL: https://issues.apache.org/jira/browse/HBASE-15126
> Project: HBase
>  Issue Type: Bug
>  Components: hbck
>Affects Versions: 2.0.0
>Reporter: chenrongwei
>Assignee: chenrongwei
> Fix For: 2.0.0
>
> Attachments: HBASE-15126-v001.patch, HBASE-15126-v002.patch, 
> HBASE-15126-v003.patch, HBASE-15126-v004.patch, HBASE-15126-v005.patch, 
> HBASE-15126-v006.patch
>
>
> HBaseFsck's checkRegionBoundaries function set the 
> 'currentRegionBoundariesInformation.storesFirstKey'  was incorrect.I think it 
> should be set like below,
> currentRegionBoundariesInformation.storesFirstKey = keyOnly(storeFirstKey);
> but current the 'currentRegionBoundariesInformation.storesFirstKey ' is just 
> set to 'storeFirstKey',which will cause to 
> comparator.compare(currentRegionBoundariesInformation.storesFirstKey,
> currentRegionBoundariesInformation.metaFirstKey)  get a wrong 
> result.  Because it just compared the rowkey's length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110808#comment-15110808
 ] 

ramkrishna.s.vasudevan commented on HBASE-14970:


TestSecureIPC fails with and without patch. Will commit this patch now.

> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15109) Region server failed to start when "fs.hdfs.impl.disable.cache" is set to true

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110845#comment-15110845
 ] 

Hadoop QA commented on HBASE-15109:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
19s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 7m 3s 
{color} | {color:red} branch/. no findbugs output file 
(./target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 4s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
11m 25s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 7m 14s 
{color} | {color:red} patch/. no findbugs output file 
(./target/findbugsXml.xml) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 55s {color} 
| {color:red} root in the patch failed with JDK v1.8.0. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 102m 36s 
{color} | {color:green} root in the patch passed with JDK v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 16s 
{color} | {color:red} Patch generated 24 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 167m 56s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | hadoop.hbase.security.TestSecureIPC |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783581/HBASE-15109-branch-1.patch
 |
| JIRA Issue | HBASE-15109 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110922#comment-15110922
 ] 

Hudson commented on HBASE-14970:


FAILURE: Integrated in HBase-1.3 #505 (See 
[https://builds.apache.org/job/HBase-1.3/505/])
HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (Ram) 
(ramkrishna: rev 58521869b06a63894e422e9c9403e48b4b12f388)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischargeHandler.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactedHFilesDischarger.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischarger.java


> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110921#comment-15110921
 ] 

Hudson commented on HBASE-13082:


FAILURE: Integrated in HBase-1.3 #505 (See 
[https://builds.apache.org/job/HBase-1.3/505/])
HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (Ram) 
(ramkrishna: rev 58521869b06a63894e422e9c9403e48b4b12f388)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischargeHandler.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreFileManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactedHFilesDischarger.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischarger.java


> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, 
> HBASE-13082_12.patch, HBASE-13082_13.patch, HBASE-13082_14.patch, 
> HBASE-13082_15.patch, HBASE-13082_16.patch, HBASE-13082_17.patch, 
> HBASE-13082_18.patch, HBASE-13082_19.patch, HBASE-13082_1_WIP.patch, 
> HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, 
> HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, 
> HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, 
> LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the cores wait 

[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110929#comment-15110929
 ] 

stack commented on HBASE-15146:
---

You have thread dumps showing how awful this is [~eclark]?

If queue is 'too big', we go back to the client. Client retries? You have it 
running over there? How does it look when we hit the limit?

Patch LGTM





> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of 
> different queues. Those queues have different capacity limits. Currently the 
> dispatch method can block trying to add the the blocking queue in any of the 
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15132) Master region merge RPC should authorize user request

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15132:
---
Attachment: HBASE-15132.v2.patch

> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132.v1.patch, HBASE-15132.v2.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9393) Hbase does not closing a closed socket resulting in many CLOSE_WAIT

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110958#comment-15110958
 ] 

Ashish Singhi commented on HBASE-9393:
--

I have tried with Apache Hadoop 2.6.0 and Custom Hadoop almost equal to 2.7.2
But I strongly think this is not related to Hadoop.

> Hbase does not closing a closed socket resulting in many CLOSE_WAIT 
> 
>
> Key: HBASE-9393
> URL: https://issues.apache.org/jira/browse/HBASE-9393
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.94.2, 0.98.0
> Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 
> 7279 regions
>Reporter: Avi Zrachya
>Assignee: Ashish Singhi
> Attachments: HBASE-9393.patch
>
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect 
> to the datanode because too many mapped sockets from one host to another on 
> the same port.
> The example below is with low CLOSE_WAIT count because we had to restart 
> hbase to solve the porblem, later in time it will incease to 60-100K sockets 
> on CLOSE_WAIT
> [root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
> 13156
> [root@hd2-region3 ~]# ps -ef |grep 21592
> root 17255 17219  0 12:26 pts/000:00:00 grep 21592
> hbase21592 1 17 Aug29 ?03:29:06 
> /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m 
> -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode 
> -Dhbase.log.dir=/var/log/hbase 
> -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15150) Fix TestDurablity in branch-1.1

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110903#comment-15110903
 ] 

Hadoop QA commented on HBASE-15150:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
21s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} branch-1.1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
19s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
19s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s 
{color} | {color:red} hbase-server in branch-1.1 has 79 extant Findbugs 
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 41s 
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
11m 50s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 38s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 7s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 57s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
23s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 226m 49s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | 
hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
| JDK v1.7.0_79 Failed junit tests | hadoop.hbase.regionserver.TestWALLockup |
|   | hadoop.hbase.regionserver.TestFailedAppendAndSync |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783579/HBASE-15150.branch-1.1.patch
 |
| JIRA Issue | HBASE-15150 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed 

[jira] [Updated] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-15146:
--
Attachment: HBASE-15146.3.patch

> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch, HBASE-15146.3.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of 
> different queues. Those queues have different capacity limits. Currently the 
> dispatch method can block trying to add the the blocking queue in any of the 
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15019) Replication stuck when HDFS is restarted

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110996#comment-15110996
 ] 

Sean Busbey commented on HBASE-15019:
-

the test failures don't look related. what do you think [~stack]?

> Replication stuck when HDFS is restarted
> 
>
> Key: HBASE-15019
> URL: https://issues.apache.org/jira/browse/HBASE-15019
> Project: HBase
>  Issue Type: Bug
>  Components: Replication, wal
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.3, 0.98.16.1
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-15019-v0_branch-1.2.patch, HBASE-15019-v1.patch, 
> HBASE-15019-v1_0.98.patch, HBASE-15019-v1_branch-1.2.patch, 
> HBASE-15019-v2.patch, HBASE-15019-v3.patch, HBASE-15019-v4.patch
>
>
> RS is normally working and writing on the WAL.
> HDFS is killed and restarted, and the RS try to do a roll.
> The close fail, but the roll succeed (because hdfs is now up) and everything 
> works.
> {noformat}
> 2015-12-11 21:52:28,058 ERROR 
> org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter: Got IOException 
> while writing trailer
> java.io.IOException: All datanodes 10.51.30.152:50010 are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1147)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:945)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:496)
> 2015-12-11 21:52:28,059 ERROR 
> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Failed close of HLog writer
> java.io.IOException: All datanodes 10.51.30.152:50010 are bad. Aborting...
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1147)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:945)
>   at 
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:496)
> 2015-12-11 21:52:28,059 WARN org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
> Riding over HLog close failure! error count=1
> {noformat}
> The problem is on the replication side. that log we rolled and we were not 
> able to close
> is waiting for a lease recovery.
> {noformat}
> 2015-12-11 21:16:31,909 ERROR 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory: Can't open after 267 
> attempts and 301124ms 
> {noformat}
> the WALFactory notify us about that, but there is nothing on the RS side that 
> perform the WAL recovery.
> {noformat}
> 2015-12-11 21:11:30,921 WARN 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory: Lease should have 
> recovered. This is not expected. Will retry
> java.io.IOException: Cannot obtain block length for 
> LocatedBlock{BP-1547065147-10.51.30.152-1446756937665:blk_1073801614_61243; 
> getBlockSize()=83; corrupt=false; offset=0; locs=[10.51.30.154:50010, 
> 10.51.30.152:50010, 10.51.30.155:50010]}
>   at 
> org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:358)
>   at 
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:300)
>   at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:237)
>   at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:230)
>   at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1448)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:301)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:297)
>   at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>   at 
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:297)
>   at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
>   at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:116)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:89)
>   at 
> org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:77)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:68)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:508)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:321)
> {noformat}
> the only way to trigger a WAL recovery is to restart and force the master to 
> trigger the lease recovery on WAL split. 
> but there is a case where restarting will not help. If 

[jira] [Updated] (HBASE-15091) Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15091:
--
Summary: Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID 
performance regression in branch-1.0"  (was: Forward-port to 1.2 HBASE-15031 
"Fix merge of MVCC and SequenceID performance regression in branch-1.0")

> Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID 
> performance regression in branch-1.0"
> 
>
> Key: HBASE-15091
> URL: https://issues.apache.org/jira/browse/HBASE-15091
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: 15091v2.branch-1.2.patch, 15091v3.branch-1.2.patch, 
> 15091v4.branch-1.2.patch, 15091v5.branch-1.2.patch, 15091v6.branch-1.2.patch, 
> 15091v6.branch-1.patch, 15091v7.branch-1.2.patch, 15091v8.branch-1.2.patch, 
> 15091v9.branch-1.2.patch, HBASE-15091-branch-1.2.patch, 
> HBASE-15091-branch-1.2_v1.patch, HBASE-15091.png, 
> HBASE-15091.v1.branch-1.2.patch, HBASE-15091.v9.branch-1.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15095) isReturnResult=false on fast path in branch-1.1 and branch-1.0 is not respected

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111076#comment-15111076
 ] 

stack commented on HBASE-15095:
---

I should have noticed that the slow/default path also needed this fix. We get a 
test failure in TestDurability (which is a weird place for this to show, i know 
but hurray for unit tests... even ones that are messy)... see HBASE-15150

> isReturnResult=false  on fast path in branch-1.1 and branch-1.0 is not 
> respected
> 
>
> Key: HBASE-15095
> URL: https://issues.apache.org/jira/browse/HBASE-15095
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 1.1.2, 1.0.3
>Reporter: stack
>Assignee: Heng Chen
> Fix For: 1.0.3, 1.1.3
>
> Attachments: HBASE-15095-branch-1.0.patch, 
> HBASE-15095-branch-1.0.patch, HBASE-15095-branch-1.1.patch
>
>
> We don't pay attention to the isReturnResult when we go the fast path 
> increment. Fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15152) Automatically include prefix-tree module in MR jobs if present

[Jonathan Hsieh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Hsieh updated HBASE-15152:
---
Status: Patch Available  (was: Open)

> Automatically include prefix-tree module in MR jobs if present
> --
>
> Key: HBASE-15152
> URL: https://issues.apache.org/jira/browse/HBASE-15152
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.3.0, 1.2.1
>
> Attachments: hbase-15152.patch
>
>
> I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
> found in the YarnChildren processes.  
> {code}
> 2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
> hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> {code}
> This is related to HBASE-7434 and HBASE-7936 which address compile time 
> concerns.  This fix makes it so that jar inclusion is done at run time, and 
> continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110975#comment-15110975
 ] 

Elliott Clark commented on HBASE-15146:
---

I have some thread dumps (I'll get them when I get to work). They are a little 
deceptive since they are only blocking 5 threads. However since those are the 
threads feeding everything it's like everything is slow.

bq.If queue is 'too big', we go back to the client. Client retries?
Yeah client retries.

bq.You have it running over there? How does it look when we hit the limit?
Yeah have it running with the first version. However a co-worker found the bug 
about not reducing queue size. So need to get that up.

> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch, HBASE-15146.3.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of 
> different queues. Those queues have different capacity limits. Currently the 
> dispatch method can block trying to add the the blocking queue in any of the 
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15152) Automatically include prefix-tree module in MR jobs if present

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110989#comment-15110989
 ] 

stack commented on HBASE-15152:
---

+1

> Automatically include prefix-tree module in MR jobs if present
> --
>
> Key: HBASE-15152
> URL: https://issues.apache.org/jira/browse/HBASE-15152
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 1.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0, 1.3.0, 1.2.1
>
> Attachments: hbase-15152.patch
>
>
> I was running some MR jobs tests and ended up with PrefixTreeCodec class not 
> found in the YarnChildren processes.  
> {code}
> 2016-01-21 06:24:26,844 WARN  [main] mapreduce.TableMapReduceUtil(785): The  
> hbase-prefix-tree module jar containing PrefixTreeCodec is not present.
> java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.code.prefixtree.PrefixTreeCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> {code}
> This is related to HBASE-7434 and HBASE-7936 which address compile time 
> concerns.  This fix makes it so that jar inclusion is done at run time, and 
> continues if it is not present (for mr unit tests that don't depend on it)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111080#comment-15111080
 ] 

Enis Soztutar commented on HBASE-15132:
---

bq. I am not aware of use case which needs the post hook. When the requirement 
comes, we can add later.
We should stick with the standard coprocessor pattern. Always add preXXX() and 
postXXX() please. 

I think the method name should be preDispatchMerge() or 
preDispatchRegionMerge(). We already have preMerge() in the regionserver 
coprocessor. 

> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132.v1.patch, HBASE-15132.v2.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15153) Apply checkFamilies addendum on increment to 1.1 and 1.0

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15153:
--
Attachment: 15153.branch-1.1.patch

> Apply checkFamilies addendum on increment to 1.1 and 1.0
> 
>
> Key: HBASE-15153
> URL: https://issues.apache.org/jira/browse/HBASE-15153
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
> Attachments: 15153.branch-1.1.patch
>
>
> branch-1.1 and branch-1.0 need this:
> --- 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++ 
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -7377,6 +7377,7 @@ public class HRegion implements HeapSize, 
> PropagatingConfigurationObserver, Regi
>  checkReadOnly();
>  checkResources();
>  checkRow(mutation.getRow(), op.toString());
> +checkFamilies(mutation.getFamilyCellMap().keySet());
>  startRegionOperation(op);
>  this.writeRequestsCount.increment();
>  try {
> Its an addendum on HBASE-15091 so fix for this oversight is in branch-1.2+.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15153) Apply checkFamilies addendum on increment to 1.1 and 1.0

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15153:
--
Fix Version/s: 1.0.4
   1.1.4
   Status: Patch Available  (was: Open)

> Apply checkFamilies addendum on increment to 1.1 and 1.0
> 
>
> Key: HBASE-15153
> URL: https://issues.apache.org/jira/browse/HBASE-15153
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15153.branch-1.1.patch
>
>
> branch-1.1 and branch-1.0 need this:
> --- 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++ 
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -7377,6 +7377,7 @@ public class HRegion implements HeapSize, 
> PropagatingConfigurationObserver, Regi
>  checkReadOnly();
>  checkResources();
>  checkRow(mutation.getRow(), op.toString());
> +checkFamilies(mutation.getFamilyCellMap().keySet());
>  startRegionOperation(op);
>  this.writeRequestsCount.increment();
>  try {
> Its an addendum on HBASE-15091 so fix for this oversight is in branch-1.2+.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15019) Replication stuck when HDFS is restarted

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110978#comment-15110978
 ] 

Hadoop QA commented on HBASE-15019:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
23m 18s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 105m 21s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
8s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 43s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Timed out junit tests | 
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface |
|   | org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
|   | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783597/HBASE-15019-v4.patch |
| JIRA Issue | HBASE-15019 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 

[jira] [Updated] (HBASE-15091) Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15091:
--
Fix Version/s: 1.3.0

> Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID 
> performance regression in branch-1.0"
> 
>
> Key: HBASE-15091
> URL: https://issues.apache.org/jira/browse/HBASE-15091
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
> Attachments: 15031.addendum, 15091v2.branch-1.2.patch, 
> 15091v3.branch-1.2.patch, 15091v4.branch-1.2.patch, 15091v5.branch-1.2.patch, 
> 15091v6.branch-1.2.patch, 15091v6.branch-1.patch, 15091v7.branch-1.2.patch, 
> 15091v8.branch-1.2.patch, 15091v9.branch-1.2.patch, 
> HBASE-15091-branch-1.2.patch, HBASE-15091-branch-1.2_v1.patch, 
> HBASE-15091.png, HBASE-15091.v1.branch-1.2.patch, 
> HBASE-15091.v9.branch-1.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15091) Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111050#comment-15111050
 ] 

stack commented on HBASE-15091:
---

I pushed this to branch-1 too. The push included the 15031.addendum.

> Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID 
> performance regression in branch-1.0"
> 
>
> Key: HBASE-15091
> URL: https://issues.apache.org/jira/browse/HBASE-15091
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
> Attachments: 15031.addendum, 15091v2.branch-1.2.patch, 
> 15091v3.branch-1.2.patch, 15091v4.branch-1.2.patch, 15091v5.branch-1.2.patch, 
> 15091v6.branch-1.2.patch, 15091v6.branch-1.patch, 15091v7.branch-1.2.patch, 
> 15091v8.branch-1.2.patch, 15091v9.branch-1.2.patch, 
> HBASE-15091-branch-1.2.patch, HBASE-15091-branch-1.2_v1.patch, 
> HBASE-15091.png, HBASE-15091.v1.branch-1.2.patch, 
> HBASE-15091.v9.branch-1.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15150) Fix TestDurablity in branch-1.1

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15150:
--
Fix Version/s: 1.0.4

> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15150v2.branch-1.1.patch, HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Updated] (HBASE-15150) Fix TestDurablity in branch-1.1

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15150:
--
Component/s: test

> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15150v2.branch-1.1.patch, HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110983#comment-15110983
 ] 

stack commented on HBASE-14970:
---

Thanks for getting this patch in. Nice one.

bq. TestSecureIPC fails with and without patch. Will commit this patch now.

Yes. HBASE-14865. Reverted it from branch-1. Needs newer version of hadoop.

What about the ASF license warnings? They have substance? I skimmed the patch 
and they do not seem related.

Needs a release note.

Good one.





> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15109) Region server failed to start when "fs.hdfs.impl.disable.cache" is set to true

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111013#comment-15111013
 ] 

Ted Yu commented on HBASE-15109:


{code}
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 16.217 sec <<< 
FAILURE! - in org.apache.hadoop.hbase.security.TestSecureIPC
testSaslNoCommonQop(org.apache.hadoop.hbase.security.TestSecureIPC)  Time 
elapsed: 11.526 sec  <<< FAILURE!
java.lang.AssertionError: 
Expected: (an instance of javax.security.sasl.SaslException and exception with 
message a string containing "No common protection layer between client and 
server")
 but: an instance of javax.security.sasl.SaslException  
is a org.ietf.jgss.GSSException
Stacktrace was: GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)
at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
{code}
Related to HBASE-14865.

Suggest re-attaching patch for new QA run.

> Region server failed to start when "fs.hdfs.impl.disable.cache" is set to true
> --
>
> Key: HBASE-15109
> URL: https://issues.apache.org/jira/browse/HBASE-15109
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 1.0.0
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Attachments: HBASE-15109-branch-1.patch
>
>
> Region server failed to start during installing ShutdownHook when 
> "fs.hdfs.impl.disable.cache" is set to true in core-site.xml at HBase side.
> {code}
> 2016-01-14 15:30:56,358 | FATAL | regionserver/ds2/192.168.152.230:21302 | 
> ABORTING region server ds2,21302,1452756654352: Unhandled: Failed suppression 
> of fs shutdown hook: 
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@571527d6 | 
> org.apache.hadoop.hbase.regionserver.HRegionServer.abort(HRegionServer.java:2054)
> java.lang.RuntimeException: Failed suppression of fs shutdown hook: 
> org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@571527d6
> at 
> org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:204)
> at 
> org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:84)
> at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:893)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> During installation, first time it will try to suppress the HDFS shutdownhook 
> by removing the hdfsclientfinalizer from HDFS. Since 
> fs.hdfs.impl.disable.cache is enabled, so removal will be unsuccessful from 
> HDFS  (FS is not cached) and RuntimeException will be thrown with "Failed 
> suppression of fs shutdown hook" message.
> In ShutdownHook,
> {code}
> if (!fsShutdownHooks.containsKey(hdfsClientFinalizer) &&
> !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> throw new RuntimeException("Failed suppression of fs shutdown hook: " +
> hdfsClientFinalizer);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15091) Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15091:
--
Attachment: 15031.addendum

One line addendum to fix a dropped check for non-existent families that was 
turned up by my attempt at applying this patch to branch-1. Pushed to 
branch-1.2. Will include it in the push to branch-1.

> Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID 
> performance regression in branch-1.0"
> 
>
> Key: HBASE-15091
> URL: https://issues.apache.org/jira/browse/HBASE-15091
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: 15031.addendum, 15091v2.branch-1.2.patch, 
> 15091v3.branch-1.2.patch, 15091v4.branch-1.2.patch, 15091v5.branch-1.2.patch, 
> 15091v6.branch-1.2.patch, 15091v6.branch-1.patch, 15091v7.branch-1.2.patch, 
> 15091v8.branch-1.2.patch, 15091v9.branch-1.2.patch, 
> HBASE-15091-branch-1.2.patch, HBASE-15091-branch-1.2_v1.patch, 
> HBASE-15091.png, HBASE-15091.v1.branch-1.2.patch, 
> HBASE-15091.v9.branch-1.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15150) Fix TestDurablity in branch-1.1

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15150:
--
Attachment: 15150v2.branch-1.1.patch

Thanks [~carp84] for figuring this issue. Proper fix is what was done in 
HBASE-15095; it just needed to be put on the default path too...I should have 
noticed that about the HBASE-15095 when it was going in.

> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.1.4
>
> Attachments: 15150v2.branch-1.1.patch, HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Updated] (HBASE-15150) Fix TestDurablity in branch-1.1

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15150:
--
Fix Version/s: 1.1.4

> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.1.4
>
> Attachments: 15150v2.branch-1.1.patch, HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Created] (HBASE-15153) Apply checkFamilies addendum on increment to 1.1 and 1.0

[stack (JIRA)]

stack created HBASE-15153:
-

 Summary: Apply checkFamilies addendum on increment to 1.1 and 1.0
 Key: HBASE-15153
 URL: https://issues.apache.org/jira/browse/HBASE-15153
 Project: HBase
  Issue Type: Sub-task
Reporter: stack
Assignee: stack


branch-1.1 and branch-1.0 need this:

--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
@@ -7377,6 +7377,7 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver, Regi
 checkReadOnly();
 checkResources();
 checkRow(mutation.getRow(), op.toString());
+checkFamilies(mutation.getFamilyCellMap().keySet());
 startRegionOperation(op);
 this.writeRequestsCount.increment();
 try {

Its an addendum on HBASE-15091 so fix for this oversight is in branch-1.2+.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111088#comment-15111088
 ] 

Enis Soztutar commented on HBASE-15132:
---

Can you also add a unit test as well. We unfortunately do not have end-to-end 
test coverage of the ACLs yet, so I do not want to block the patch for adding 
one. However, maybe we can do a manual test at least. 

> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132.v1.patch, HBASE-15132.v2.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15153) Apply checkFamilies addendum on increment to 1.1 and 1.0

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1533#comment-1533
 ] 

Hadoop QA commented on HBASE-15153:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s {color} 
| {color:red} HBASE-15153 does not apply to branch-1.1. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12783628/15153.branch-1.1.patch
 |
| JIRA Issue | HBASE-15153 |
| Powered by | Apache Yetus 0.1.0   http://yetus.apache.org |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/237/console |


This message was automatically generated.



> Apply checkFamilies addendum on increment to 1.1 and 1.0
> 
>
> Key: HBASE-15153
> URL: https://issues.apache.org/jira/browse/HBASE-15153
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15153.branch-1.1.patch
>
>
> branch-1.1 and branch-1.0 need this:
> --- 
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> +++ 
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> @@ -7377,6 +7377,7 @@ public class HRegion implements HeapSize, 
> PropagatingConfigurationObserver, Regi
>  checkReadOnly();
>  checkResources();
>  checkRow(mutation.getRow(), op.toString());
> +checkFamilies(mutation.getFamilyCellMap().keySet());
>  startRegionOperation(op);
>  this.writeRequestsCount.increment();
>  try {
> Its an addendum on HBASE-15091 so fix for this oversight is in branch-1.2+.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15147) Shell should use Admin.listTableNames() instead of Admin.listTables()

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-15147:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 1.0.4
   Status: Resolved  (was: Patch Available)

I have committed this. Thanks for reviews. 

> Shell should use Admin.listTableNames() instead of Admin.listTables() 
> --
>
> Key: HBASE-15147
> URL: https://issues.apache.org/jira/browse/HBASE-15147
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: hbase-15147_v1.patch
>
>
> It seems that getTableDescriptors() in master checks for A and C permissions 
> while getTableNames() checks for any privilege on the table. The reasoning is 
> explained here: 
> https://issues.apache.org/jira/browse/HBASE-12564?focusedCommentId=14234504=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234504
>  
> We should change the shell command for {{list}} to use the getTableNames() 
> version because of this. Otherwise a user having only R or W cannot list the 
> table name. 
> This has been reported from a user here: 
> https://community.hortonworks.com/questions/10742/why-does-a-user-need-create-permission-for-list-co.html#comment-11000.
>  
> While we are at it, should we revisit the fact that you cannot get a table 
> descriptor if you have only R or W? It seems strange that you cannot even 
> know the CF names of a table that you can read from. I could not find info 
> about the "describe" privileges on SQL databases. However, if there are use 
> cases where Table descriptor might contain sensitive info, the current 
> semantics seems fine. cc [~apurtell] and [~mbertozzi]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15091) Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111201#comment-15111201
 ] 

Hudson commented on HBASE-15091:


SUCCESS: Integrated in HBase-1.2-IT #405 (See 
[https://builds.apache.org/job/HBase-1.2-IT/405/])
HBASE-15091 Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and (stack: rev 
d3ce4cdce206e080ee102d394b9038524d981aab)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID 
> performance regression in branch-1.0"
> 
>
> Key: HBASE-15091
> URL: https://issues.apache.org/jira/browse/HBASE-15091
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
> Attachments: 15031.addendum, 15091v2.branch-1.2.patch, 
> 15091v3.branch-1.2.patch, 15091v4.branch-1.2.patch, 15091v5.branch-1.2.patch, 
> 15091v6.branch-1.2.patch, 15091v6.branch-1.patch, 15091v7.branch-1.2.patch, 
> 15091v8.branch-1.2.patch, 15091v9.branch-1.2.patch, 
> HBASE-15091-branch-1.2.patch, HBASE-15091-branch-1.2_v1.patch, 
> HBASE-15091.png, HBASE-15091.v1.branch-1.2.patch, 
> HBASE-15091.v9.branch-1.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-14460:
--
Release Note: Regression addressed narrowly on branch-1 (1.0.3+, 1.1.3+, 
and 1.2.0). You need to set a  "hbase.increment.fast.but.narrow.consistency" 
and rolling restart to enable a 'fast path' for increments only (appends, 
checkAndPut, etc., are not address in branch-1). See the sub-task release notes 
for more detail. On master branch, a more substantial change is done changing 
write path so all check-and-put type operations -- i.e. increment, append, 
checkAnd*, etc. -- have the regression addressed.

> [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed 
> Increments, CheckAndPuts, batch operations
> -
>
> Key: HBASE-14460
> URL: https://issues.apache.org/jira/browse/HBASE-14460
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 0.94.test.patch, 0.98.test.patch, 
> 1.0.80.flamegraph-7932.svg, 14460.txt, 14460.v0.branch-1.0.patch, 
> 98.80.flamegraph-11428.svg, HBASE-14460-discussion.patch, client.test.patch, 
> flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg, 
> flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg, 
> flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg, 
> hack.flamegraph-16593.svg, hack.uncommitted.patch, m.test.patch, 
> region_lock.png, testincrement.094.patch, testincrement.098.patch, 
> testincrement.master.patch
>
>
> As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation 
> between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of 
> sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to 
> 'catch up' to our current point before we can read the last Increment value 
> that we need to update.
> We can say that our Increment is just done wrong, we should just be writing 
> Increments and summing on read, but checkAndPut as well as batching 
> operations have the same issue. Fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15031) Fix merge of MVCC and SequenceID performance regression in branch-1.0

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111202#comment-15111202
 ] 

Hudson commented on HBASE-15031:


SUCCESS: Integrated in HBase-1.2-IT #405 (See 
[https://builds.apache.org/job/HBase-1.2-IT/405/])
HBASE-15091 Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and (stack: rev 
d3ce4cdce206e080ee102d394b9038524d981aab)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Fix merge of MVCC and SequenceID performance regression in branch-1.0
> -
>
> Key: HBASE-15031
> URL: https://issues.apache.org/jira/browse/HBASE-15031
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 1.0.3
>Reporter: stack
>Assignee: stack
> Fix For: 1.0.3, 1.1.3
>
> Attachments: 14460.v0.branch-1.0.patch, 15031.v2.branch-1.0.patch, 
> 15031.v3.branch-1.0.patch, 15031.v4.branch-1.0.patch, 
> 15031.v5.branch-1.0.patch, 15031.v6.branch-1.0.patch, 
> 15031.v6.branch-1.0.patch, 15031.v6.branch-1.0.patch, 
> 15031.v7.branch-1.0.patch, 15031.v8.branch-1.0.patch, 15031.v8.master.patch
>
>
> Subtask with fix for branch-1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111216#comment-15111216
 ] 

Elliott Clark commented on HBASE-15146:
---

bq.please ping me when this goes in.

Will do

> Don't block on Reader threads queueing to a scheduler queue
> ---
>
> Key: HBASE-15146
> URL: https://issues.apache.org/jira/browse/HBASE-15146
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Blocker
> Fix For: 1.2.0
>
> Attachments: HBASE-15146.0.patch, HBASE-15146.1.patch, 
> HBASE-15146.2.patch, HBASE-15146.3.patch
>
>
> Blocking on the epoll thread is awful. The new rpc scheduler can have lots of 
> different queues. Those queues have different capacity limits. Currently the 
> dispatch method can block trying to add the the blocking queue in any of the 
> schedulers.
> This causes readers to block, tcp acks are delayed, and everything slows down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack resolved HBASE-14865.
---
   Resolution: Fixed
Fix Version/s: (was: 1.3.0)

Thanks [~mbertozzi] Reverted from branch-1. We can come back here if branch-1 
ups its hadoop version.

Resolving again.



> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: 14865-master-v7.patch, HBASE-14865-branch-1.2.patch, 
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch, 
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch, 
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch, 
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch, 
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of 
> authentication/integrity/privacy. It is the used to set 
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll 
> have to take a downtime. Rolling upgrade will create a situation where some 
> nodes have old value and some have new, which'll prevent any communication 
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order. 
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [CIS-CMMI-3] IllegalArgumentException: Row length 41221 is > 32767

[Vladimir Rodionov]

Maximum size of row in HBase is 32767 and your application is trying to get
rows which exceed this limit.

You probably better ask your question in gora/nutch user group.

-Vlad

On Thu, Jan 21, 2016 at 5:39 AM, Kshitij Shukla 
wrote:

> Hello everyone,
>
> Software stack is *nutch-branch-2.3.1, gora-hbase 0.6.1 Hadoop 2.5.2,
> hbase-0.98.8-hadoop2
>
> * I have added a set of seeds to crawl using this command
> *
> ./bin/crawl /largeSeeds 1 http://localhost:8983/solr/ddcd 4*
>
> For first iteration all of the commands(*inject, **generate, **fetch,
> **parse, **update-table, **Indexer & delete duplicates.*) got executed
> successfully.
> For second iteration, *"CrawlDB update" *command got failed (please see
> error log for reference), because of failure of this command the whole
> process gets terminated.
>
>
> LOG
> START
> 16/01/20 02:45:19 INFO parse.ParserJob: ParserJob: finished at 2016-01-20
> 02:45:19, time elapsed: 00:06:57
> CrawlDB update for 1
> /usr/share/searchEngine/nutch-branch-2.3.1/runtime/deploy/bin/nutch
> updatedb -D mapred.reduce.tasks=2 -D mapred.child.java.opts=-Xmx1000m -D
> mapred.reduce.tasks.speculative.execution=false -D
> mapred.map.tasks.speculative.execution=false -D
> mapred.compress.map.output=true 1453230757-13191 -crawlId 1
> 16/01/20 02:45:27 INFO crawl.DbUpdaterJob: DbUpdaterJob: starting at
> 2016-01-20 02:45:27
> 16/01/20 02:45:27 INFO crawl.DbUpdaterJob: DbUpdaterJob: batchId:
> 1453230757-13191
> 16/01/20 02:45:27 INFO plugin.PluginRepository: Plugins: looking in:
> /tmp/hadoop-root/hadoop-unjar5654418190157422003/classes/plugins
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Plugin Auto-activation
> mode: [true]
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Registered Plugins:
> 16/01/20 02:45:28 INFO plugin.PluginRepository: HTTP Framework
> (lib-http)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Html Parse Plug-in
> (parse-html)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: MetaTags
> (parse-metatags)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: the nutch core
> extension points (nutch-extensionpoints)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Basic Indexing Filter
> (index-basic)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: XML Libraries (lib-xml)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Anchor Indexing Filter
> (index-anchor)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Basic URL Normalizer
> (urlnormalizer-basic)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Language
> Identification Parser/Filter (language-identifier)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Metadata Indexing
> Filter (index-metadata)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: CyberNeko HTML Parser
> (lib-nekohtml)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Subcollection indexing
> and query filter (subcollection)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: SOLRIndexWriter
> (indexer-solr)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Rel-Tag microformat
> Parser/Indexer/Querier (microformats-reltag)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Http / Https Protocol
> Plug-in (protocol-httpclient)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: JavaScript Parser
> (parse-js)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Tika Parser Plug-in
> (parse-tika)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Top Level Domain
> Plugin (tld)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Regex URL Filter
> Framework (lib-regex-filter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Regex URL Normalizer
> (urlnormalizer-regex)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Link Analysis Scoring
> Plug-in (scoring-link)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: OPIC Scoring Plug-in
> (scoring-opic)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: More Indexing Filter
> (index-more)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Http Protocol Plug-in
> (protocol-http)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Creative Commons
> Plugins (creativecommons)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Registered
> Extension-Points:
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Parse Filter
> (org.apache.nutch.parse.ParseFilter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch Index Cleaning
> Filter (org.apache.nutch.indexer.IndexCleaningFilter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch Content Parser
> (org.apache.nutch.parse.Parser)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch URL Filter (
> org.apache.nutch.net.URLFilter)
> 16/01/20 02:45:28 INFO plugin.PluginRepository: Nutch Scoring
> (org.apache.nutch.scoring.ScoringFilter)
> 16/01/20 02:45:28 INFO 

[jira] [Commented] (HBASE-15132) Master region merge RPC should authorize user request

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110937#comment-15110937
 ] 

Ted Yu commented on HBASE-15132:


bq. We don't want a post hook?

I am not aware of use case which needs the post hook. When the requirement 
comes, we can add later.

The rest of review comments are addressed in patch v2.

> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132.v1.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111221#comment-15111221
 ] 

stack commented on HBASE-14460:
---

Added release note on approach going on in here. (Unfortunately,) We have three 
different changes made under the rubric of this issue. In the branch-1.0 (and 
branch-1.1)  fix, HBASE-15031, and in the branch-1.2+ fix, HBASE-15091, we 
address the Increment slow down only; you set a flag to take a 'fast path' for 
Increments only. The patches for branch-1.0+branch-1.1 and that for branch-1.2+ 
differ pretty substantially mostly because HBASE-12751 changed the theatre of 
operation. The fix for master branch is a general fix. It builds on the changes 
in branch-1 taking the 'fast path' only for increments but then changes the 
write path for all updates in hbase -- appends, checkAnd* AND the ho-hum 
batchMutate, doMiniBatchMutate, etc., -- to do similar, making all changes 
under row lock, including sync and mvcc catchup, so append and checkAnd* can 
take the same 'fast path' and we can continue to honor our consistency-on-a-row 
tenet. Doing all under a lock was intentionally avoided in the past for 
performance reasons but HBASE-12751 (row locks are read/write rather than 
xclusive) plus all that has changed in the write path since changes the lay of 
the land; HBASE-15046 seems to show the default path runs a little faster 
though all now done under lock (more perf testing to follow).

> [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed 
> Increments, CheckAndPuts, batch operations
> -
>
> Key: HBASE-14460
> URL: https://issues.apache.org/jira/browse/HBASE-14460
> Project: HBase
>  Issue Type: Bug
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 0.94.test.patch, 0.98.test.patch, 
> 1.0.80.flamegraph-7932.svg, 14460.txt, 14460.v0.branch-1.0.patch, 
> 98.80.flamegraph-11428.svg, HBASE-14460-discussion.patch, client.test.patch, 
> flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg, 
> flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg, 
> flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg, 
> hack.flamegraph-16593.svg, hack.uncommitted.patch, m.test.patch, 
> region_lock.png, testincrement.094.patch, testincrement.098.patch, 
> testincrement.master.patch
>
>
> As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation 
> between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of 
> sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to 
> 'catch up' to our current point before we can read the last Increment value 
> that we need to update.
> We can say that our Increment is just done wrong, we should just be writing 
> Increments and summing on read, but checkAndPut as well as batching 
> operations have the same issue. Fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15132) Master region merge RPC should authorize user request

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-15132:
---
Attachment: HBASE-15132.v4.patch

Patch v4 addresses Enis' comment with unit test.

> Master region merge RPC should authorize user request
> -
>
> Key: HBASE-15132
> URL: https://issues.apache.org/jira/browse/HBASE-15132
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: HBASE-15132.v1.patch, HBASE-15132.v2.patch, 
> HBASE-15132.v4.patch
>
>
> The normal flow for region merge is:
> 1. client sends a master RPC for dispatch merge regions
> 2. master moves the regions to the same regionserver
> 3. master calls mergeRegions RPC on the regionserver. 
> For user initiated region merge, MasterRpcServices#dispatchMergingRegions() 
> is called by HBaseAdmin.
> There is no coprocessor invocation in step 1.
> Step 3 is carried out in the "hbase" user context.
> This leaves potential security hole - any user without proper authorization 
> can merge regions of any table.
> Thanks to Enis who spotted this flaw first.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15150) Fix TestDurablity in branch-1.1

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15150:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Pushed to branch-1.1 and branch-1.0. The failed unit tests lists does not 
include TestDurability. Thanks for digging in on this one [~carp84]


> Fix TestDurablity in branch-1.1
> ---
>
> Key: HBASE-15150
> URL: https://issues.apache.org/jira/browse/HBASE-15150
> Project: HBase
>  Issue Type: Bug
>  Components: test
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 1.1.4, 1.0.4
>
> Attachments: 15150v2.branch-1.1.patch, HBASE-15150.branch-1.1.patch
>
>
> From [UT 
> report|https://builds.apache.org/job/PreCommit-HBASE-Build/145/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_79.txt]
>  of HBASE-15120 against branch-1.1, we could see below failure:
> {noformat}
> testIncrementWithReturnResultsSetToFalse(org.apache.hadoop.hbase.regionserver.wal.TestDurability)
>   Time elapsed: 0.111 sec  <<< FAILURE!
> java.lang.AssertionError: expected null, but 
> 

[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)

[Francis Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111617#comment-15111617
 ] 

Francis Liu commented on HBASE-11165:
-

Just an update, we've been running a production cluster which has 16 meta 
regions and around 260k regions for the last 2 months or so. We'll start 
getting the supporting changes in zkless (if any), HBASE-11290. Then the big 
patch, tho will try to break it up into smaller pieces if possible.

> Scaling so cluster can host 1M regions and beyond (50M regions?)
> 
>
> Key: HBASE-11165
> URL: https://issues.apache.org/jira/browse/HBASE-11165
> Project: HBase
>  Issue Type: Brainstorming
>Reporter: stack
> Attachments: HBASE-11165.zip, Region Scalability test.pdf, 
> ScalableMeta.pdf, zk_less_assignment_comparison_2.pdf
>
>
> This discussion issue comes out of "Co-locate Meta And Master HBASE-10569" 
> and comments on the doc posted there.
> A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M 
> regions maybe even 50M later.  This issue is about discussing how we will do 
> that (or if not 50M on a cluster, how otherwise we can attain same end).
> More detail to follow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-15154) Master puts up a blockcache instance

[stack (JIRA)]

stack created HBASE-15154:
-

 Summary: Master puts up a blockcache instance
 Key: HBASE-15154
 URL: https://issues.apache.org/jira/browse/HBASE-15154
 Project: HBase
  Issue Type: Bug
Reporter: stack
Priority: Critical


Master is putting up a blockcache instance. It shouldn't. This issue comes of 
the unification of Master and RegionServer where Master is a subclass of 
RegionServer. Our [~jmspaggi] found out the hard way today when he tried to 
bring up a cluster with a large offheap cache only the master member had been 
given less memory.

Marking critical rather than blocker because there is a means of configuring 
your way out of this little entanglement; set hbase.bucketcache.size to zero in 
 the master config only.

If you want to set direct memory for regionservers only and not on the master, 
you should do the following:

Set HBASE_OFFHEAPSIZE=0G

... and then turn it on for RegionServers only:

HBASE_REGIONSERVER_OPTS="HBASE_REGIONSERVER_OPTS -XX:MaxDirectMemorySize=XXXG" 
where XXX is size of the direct memory you want to allocate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15146) Don't block on Reader threads queueing to a scheduler queue

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111397#comment-15111397
 ] 

Hadoop QA commented on HBASE-15146:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s 
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 11s 
{color} | {color:red} Patch generated 12 new checkstyle issues in hbase-server 
(total was 105, now 117). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
20m 12s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 42s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 39s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
18s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 232m 41s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0 Failed junit tests | 
hadoop.hbase.security.token.TestGenerateDelegationToken |
|   | hadoop.hbase.ipc.TestDelayedRpc |
| JDK v1.7.0_79 Failed junit tests | hadoop.hbase.ipc.TestDelayedRpc |
| JDK v1.7.0_79 Timed out junit tests | org.apache.hadoop.hbase.TestZooKeeper |
|   | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
|   | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
|   | 

[jira] [Commented] (HBASE-13082) Coarsen StoreScanner locks to RegionScanner

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111600#comment-15111600
 ] 

Hudson commented on HBASE-13082:


SUCCESS: Integrated in HBase-1.3-IT #451 (See 
[https://builds.apache.org/job/HBase-1.3-IT/451/])
HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (Ram) 
(ramkrishna: rev 58521869b06a63894e422e9c9403e48b4b12f388)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactedHFilesDischarger.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischargeHandler.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischarger.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java


> Coarsen StoreScanner locks to RegionScanner
> ---
>
> Key: HBASE-13082
> URL: https://issues.apache.org/jira/browse/HBASE-13082
> Project: HBase
>  Issue Type: Bug
>  Components: Performance, Scanners
>Reporter: Lars Hofhansl
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 13082-test.txt, 13082-v2.txt, 13082-v3.txt, 
> 13082-v4.txt, 13082.txt, 13082.txt, HBASE-13082.pdf, HBASE-13082_1.pdf, 
> HBASE-13082_12.patch, HBASE-13082_13.patch, HBASE-13082_14.patch, 
> HBASE-13082_15.patch, HBASE-13082_16.patch, HBASE-13082_17.patch, 
> HBASE-13082_18.patch, HBASE-13082_19.patch, HBASE-13082_1_WIP.patch, 
> HBASE-13082_2.pdf, HBASE-13082_2_WIP.patch, HBASE-13082_3.patch, 
> HBASE-13082_4.patch, HBASE-13082_9.patch, HBASE-13082_9.patch, 
> HBASE-13082_withoutpatch.jpg, HBASE-13082_withpatch.jpg, 
> LockVsSynchronized.java, gc.png, gc.png, gc.png, hits.png, next.png, next.png
>
>
> Continuing where HBASE-10015 left of.
> We can avoid locking (and memory fencing) inside StoreScanner by deferring to 
> the lock already held by the RegionScanner.
> In tests this shows quite a scan improvement and reduced CPU (the fences make 
> the 

[jira] [Commented] (HBASE-15031) Fix merge of MVCC and SequenceID performance regression in branch-1.0

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111602#comment-15111602
 ] 

Hudson commented on HBASE-15031:


SUCCESS: Integrated in HBase-1.3-IT #451 (See 
[https://builds.apache.org/job/HBase-1.3-IT/451/])
HBASE-15091 Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and (stack: rev 
a9c008344f7555322c96f193519eefcca308dd7c)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementsFromClientSide.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideWithCoprocessor.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/Tag.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestIncrement.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideNoCodec.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementFromClientSideWithCoprocessor.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedScannerCallable.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionIncrement.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/IncrementPerformanceTest.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/io/compress/Compression.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java


> Fix merge of MVCC and SequenceID performance regression in branch-1.0
> -
>
> Key: HBASE-15031
> URL: https://issues.apache.org/jira/browse/HBASE-15031
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 1.0.3
>Reporter: stack
>Assignee: stack
> Fix For: 1.0.3, 1.1.3
>
> Attachments: 14460.v0.branch-1.0.patch, 15031.v2.branch-1.0.patch, 
> 15031.v3.branch-1.0.patch, 15031.v4.branch-1.0.patch, 
> 15031.v5.branch-1.0.patch, 15031.v6.branch-1.0.patch, 
> 15031.v6.branch-1.0.patch, 15031.v6.branch-1.0.patch, 
> 15031.v7.branch-1.0.patch, 15031.v8.branch-1.0.patch, 15031.v8.master.patch
>
>
> Subtask with fix for branch-1.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111604#comment-15111604
 ] 

Hudson commented on HBASE-14865:


SUCCESS: Integrated in HBase-1.3-IT #451 (See 
[https://builds.apache.org/job/HBase-1.3-IT/451/])
Revert "HBASE-14865 Support passing multiple QOPs to SaslClient/Server (stack: 
rev d965d14a63c1d713b8b85be23f77599431034c5e)
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureIPC.java
* 
hbase-client/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslClientHandler.java
* hbase-client/src/test/java/org/apache/hadoop/hbase/security/TestSaslUtil.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/AbstractTestSecureIPC.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/AsyncRpcChannel.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestAsyncSecureIPC.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestHBaseSaslRpcClient.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/security/SaslUtil.java


> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
>  Issue Type: Improvement
>  Components: security
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: 14865-master-v7.patch, HBASE-14865-branch-1.2.patch, 
> HBASE-14865-branch-1.patch, HBASE-14865-branch-1.patch, 
> HBASE-14865-master-v2.patch, HBASE-14865-master-v3.patch, 
> HBASE-14865-master-v4.patch, HBASE-14865-master-v5.patch, 
> HBASE-14865-master-v6.patch, HBASE-14865-master-v7.patch, 
> HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of 
> authentication/integrity/privacy. It is the used to set 
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll 
> have to take a downtime. Rolling upgrade will create a situation where some 
> nodes have old value and some have new, which'll prevent any communication 
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order. 
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15147) Shell should use Admin.listTableNames() instead of Admin.listTables()

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111601#comment-15111601
 ] 

Hudson commented on HBASE-15147:


SUCCESS: Integrated in HBase-1.3-IT #451 (See 
[https://builds.apache.org/job/HBase-1.3-IT/451/])
HBASE-15147 Shell should use Admin.listTableNames() instead of (enis: rev 
27dd1156fb86d5b5a54b4aacd2c6a4abf58def65)
* hbase-shell/src/main/ruby/hbase/admin.rb


> Shell should use Admin.listTableNames() instead of Admin.listTables() 
> --
>
> Key: HBASE-15147
> URL: https://issues.apache.org/jira/browse/HBASE-15147
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 1.0.4
>
> Attachments: hbase-15147_v1.patch
>
>
> It seems that getTableDescriptors() in master checks for A and C permissions 
> while getTableNames() checks for any privilege on the table. The reasoning is 
> explained here: 
> https://issues.apache.org/jira/browse/HBASE-12564?focusedCommentId=14234504=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234504
>  
> We should change the shell command for {{list}} to use the getTableNames() 
> version because of this. Otherwise a user having only R or W cannot list the 
> table name. 
> This has been reported from a user here: 
> https://community.hortonworks.com/questions/10742/why-does-a-user-need-create-permission-for-list-co.html#comment-11000.
>  
> While we are at it, should we revisit the fact that you cannot get a table 
> descriptor if you have only R or W? It seems strange that you cannot even 
> know the CF names of a table that you can read from. I could not find info 
> about the "describe" privileges on SQL databases. However, if there are use 
> cases where Table descriptor might contain sensitive info, the current 
> semantics seems fine. cc [~apurtell] and [~mbertozzi]. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15091) Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0"

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111599#comment-15111599
 ] 

Hudson commented on HBASE-15091:


SUCCESS: Integrated in HBase-1.3-IT #451 (See 
[https://builds.apache.org/job/HBase-1.3-IT/451/])
HBASE-15091 Forward-port to 1.2 HBASE-15031 "Fix merge of MVCC and (stack: rev 
a9c008344f7555322c96f193519eefcca308dd7c)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementsFromClientSide.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/Scan.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideWithCoprocessor.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/Tag.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueScanner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
* hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestIncrement.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSideNoCodec.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestIncrementFromClientSideWithCoprocessor.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ReversedScannerCallable.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MultiVersionConcurrencyControl.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionIncrement.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/IncrementPerformanceTest.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/io/compress/Compression.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java


> Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID 
> performance regression in branch-1.0"
> 
>
> Key: HBASE-15091
> URL: https://issues.apache.org/jira/browse/HBASE-15091
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Blocker
> Fix For: 1.2.0, 1.3.0
>
> Attachments: 15031.addendum, 15091v2.branch-1.2.patch, 
> 15091v3.branch-1.2.patch, 15091v4.branch-1.2.patch, 15091v5.branch-1.2.patch, 
> 15091v6.branch-1.2.patch, 15091v6.branch-1.patch, 15091v7.branch-1.2.patch, 
> 15091v8.branch-1.2.patch, 15091v9.branch-1.2.patch, 
> HBASE-15091-branch-1.2.patch, HBASE-15091-branch-1.2_v1.patch, 
> HBASE-15091.png, HBASE-15091.v1.branch-1.2.patch, 
> HBASE-15091.v9.branch-1.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14970) Backport HBASE-13082 and its sub-jira to branch-1

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111603#comment-15111603
 ] 

Hudson commented on HBASE-14970:


SUCCESS: Integrated in HBase-1.3-IT #451 (See 
[https://builds.apache.org/job/HBase-1.3-IT/451/])
HBASE-14970 Backport HBASE-13082 and its sub-jira to branch-1 (Ram) 
(ramkrishna: rev 58521869b06a63894e422e9c9403e48b4b12f388)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReversedStoreScanner.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/ExecutorType.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat2.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MockStoreFile.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionReplayEvents.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCompactedHFilesDischarger.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEncryptionKeyRotation.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischargeHandler.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionConfiguration.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactedHFilesDischarger.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreFileManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java


> Backport HBASE-13082 and its sub-jira to branch-1
> -
>
> Key: HBASE-14970
> URL: https://issues.apache.org/jira/browse/HBASE-14970
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.3.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 1.3.0
>
> Attachments: HBASE-13082-branch-1.patch, 
> HBASE-14970_6.branch-1.patch, HBASE-14970_7.branch-1.patch, 
> HBASE-14970_8.branch-1.patch, HBASE-14970_9.branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1.patch, 
> HBASE-14970_branch-1.patch, HBASE-14970_branch-1_1.patch, 
> HBASE-14970_branch-1_2.patch, HBASE-14970_branch-1_4.patch, 
> HBASE-14970_branch-1_5.patch, HBASE-14970_branch-1_6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)