[jira] [Updated] (HBASE-17194) Assign the new region to the idle server after splitting

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17194:
--
Status: Patch Available  (was: Open)

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> HBASE-17194.v2.patch, evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17194) Assign the new region to the idle server after splitting

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17194:
--
Attachment: HBASE-17194.v2.patch

All failed tests pass locally.

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> HBASE-17194.v2.patch, evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711216#comment-15711216
 ] 

Hadoop QA commented on HBASE-14882:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
31m 38s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 56s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 44s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841237/HBASE-14882.master.004.patch
 |
| JIRA Issue | HBASE-14882 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 6c76de6c1271 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / fb789b3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4741/testReport/ |
| modules | C: hbase-common hbase-client U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4741/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Provide a Put API that adds the provided 

[jira] [Updated] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi updated HBASE-16499:
--
Component/s: Replication

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16499) slow replication for small HBase clusters

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711200#comment-15711200
 ] 

Ashish Singhi commented on HBASE-16499:
---

Any update here, [~vik.karma] ?

> slow replication for small HBase clusters
> -
>
> Key: HBASE-16499
> URL: https://issues.apache.org/jira/browse/HBASE-16499
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>
> For small clusters 10-20 nodes we recently observed that replication is 
> progressing very slowly when we do bulk writes and there is lot of lag 
> accumulation on AgeOfLastShipped / SizeOfLogQueue. From the logs we observed 
> that the number of threads used for shipping wal edits in parallel comes from 
> the following equation in HBaseInterClusterReplicationEndpoint
> int n = Math.min(Math.min(this.maxThreads, entries.size()/100+1),
>   replicationSinkMgr.getSinks().size());
> ... 
>   for (int i=0; i entryLists.add(new ArrayList(entries.size()/n+1)); <-- 
> batch size
>   }
> ...
> for (int i=0; i  .
> // RuntimeExceptions encountered here bubble up and are handled 
> in ReplicationSource
> pool.submit(createReplicator(entryLists.get(i), i));  <-- 
> concurrency 
> futures++;
>   }
> }
> maxThreads is fixed & configurable and since we are taking min of the three 
> values n gets decided based replicationSinkMgr.getSinks().size() when we have 
> enough edits to replicate
> replicationSinkMgr.getSinks().size() is decided based on 
> int numSinks = (int) Math.ceil(slaveAddresses.size() * ratio);
> where ratio is this.ratio = conf.getFloat("replication.source.ratio", 
> DEFAULT_REPLICATION_SOURCE_RATIO);
> Currently DEFAULT_REPLICATION_SOURCE_RATIO is set to 10% so for small 
> clusters of size 10-20 RegionServers  the value we get for numSinks and hence 
> n is very small like 1 or 2. This substantially reduces the pool concurrency 
> used for shipping wal edits in parallel effectively slowing down replication 
> for small clusters and causing lot of lag accumulation in AgeOfLastShipped. 
> Sometimes it takes tens of hours to clear off the entire replication queue 
> even after the client has finished writing on the source side. 
> We are running tests by varying replication.source.ratio and have seen 
> multi-fold improvement in total replication time (will update the results 
> here). I wanted to propose here that we should increase the default value for 
> replication.source.ratio also so that we have sufficient concurrency even for 
> small clusters. We figured it out after lot of iterations and debugging so 
> probably slightly higher default will save the trouble. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying

[Xiang Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711187#comment-15711187
 ] 

Xiang Li commented on HBASE-14882:
--

Hi [~anoop.hbase]

I uploaded patch 004 for master and triggered HADOOP-QA.
Patch 004 mainly addresses:
1. Implement ExtendedCell (to override 2 write() and getSerializedSize())
2. Enhance UT. Previously, the expected value of assert was hard-coded. I 
changed them to verify if IndividualBytesFieldCell has the same 
behaviors/outputs as KeyValue when given the same inputs.

For the implementation of 2 write() and getSerializedSize(), I used the 
functions in KeyValueUtil as much as I can. I also implemented the logic myself 
according to KeyValue format, which I think might be a little more efficient, 
at least no copy is made, and write into byte buffer or output stream directly. 
The reason why I chose to use the functions in KeyValueUtil is that it saves a 
lot of efforts of the maintenance in the future, when somebody would like to 
modify the key value format. He does not need to care about the implementation 
in IndividualBytesFieldCell, as long as the functions in KeyValueUtil are 
modified. The drawback is that it might be less efficient than that implemented 
directly, when calling write().  What is your opinion?

> Provide a Put API that adds the provided family, qualifier, value without 
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch, 
> HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, 
> HBASE-14882.master.003.patch, HBASE-14882.master.004.patch
>
>
> In the Put API, we have addImmutable()
> {code}
>  /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
>   public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being 
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family, 
> qualifier and value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17110) Improve SimpleLoadBalancer to always take server-level balance into account

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711184#comment-15711184
 ] 

Hudson commented on HBASE-17110:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2051 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2051/])
HBASE-17110 Improve SimpleLoadBalancer to always take server-level (liyu: rev 
b2086873a95b6916d66c1c6734fa0e130c5aff74)
* (edit) 
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/SimpleLoadBalancer.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/LoadBalancer.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java


> Improve SimpleLoadBalancer to always take server-level balance into account
> ---
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Fix For: 2.0.0
>
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch, 
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110-V6.patch, 
> HBASE-17110-V7.patch, HBASE-17110-V8.patch, HBASE-17110.patch
>
>
> Currently with bytable strategy there might still be server-level imbalance 
> and we will improve this in this JIRA.
> Some more background:
> When operating large scale clusters(our case), some companies still prefer to 
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan 
> generation, etc. Current SimpleLoadBalancer has two modes: 
> 1. byTable, which only guarantees that the regions of one table could be 
> uniformly distributed. 
> 2. byCluster, which ignores the distribution within tables and balance the 
> regions all together.
> If the pressures on different tables are different, the first byTable option 
> is the preferable one in most case. Yet, this choice sacrifice the cluster 
> level balance and would cause some servers to have significantly higher load, 
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case,  a cluster has 3 tables and 4 servers:
> {noformat}
>   server A has 3 regions: table1:1, table2:1, table3:1
>   server B has 3 regions: table1:2, table2:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been 
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
>   server A has 2 regions: table2:1, table3:1
>   server B has 2 regions: table1:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1, 
> table2 and table3 still keep balanced. And this is the goal this JIRA tries 
> to achieve.
> Two UTs will be added as well with the last one demonstrating advantage of 
> the new strategy. Also, a onConfigurationChange method will be implemented to 
> hot control the "slop" variable.
> We have been using the strategy on our largest cluster for several months, so 
> the effect could be assured to some extent.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17151) New API to create HFile.Reader without instantiating block cache

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711185#comment-15711185
 ] 

Hudson commented on HBASE-17151:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2051 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2051/])
HBASE-17151 New API to create HFile.Reader without instantiating block (enis: 
rev b6f5d5b85ff518defc9ffb64cc1d1ab2669783d2)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java


> New API to create HFile.Reader without instantiating block cache 
> -
>
> Key: HBASE-17151
> URL: https://issues.apache.org/jira/browse/HBASE-17151
> Project: HBase
>  Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch, 
> HBASE-17151-v3.patch, HBASE-17151-v4.patch
>
>
> Currently, to create HFile.Reader instance, the CacheConfig instance is 
> required (which instantiates block cache). We need API for reading HFile w/o 
> block cache being involved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17210) Set timeout on trying rowlock according to client's RPC timeout

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711182#comment-15711182
 ] 

Ted Yu commented on HBASE-17210:


bq. so we can use System.currentTimeMillis() directly

I don't have strong opinion on this. It is up to you.

> Set timeout on trying rowlock according to client's RPC timeout
> ---
>
> Key: HBASE-17210
> URL: https://issues.apache.org/jira/browse/HBASE-17210
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-17120.v1.patch
>
>
> Now when we want to get a row lock, the timeout is fixed and default is 30s. 
> But the requests from client have different RPC timeout setting. We can use 
> the client's deadline to set timeout on tryLock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711181#comment-15711181
 ] 

Hadoop QA commented on HBASE-17114:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
42s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
54s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
31s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
15s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
19m 33s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 45s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 54s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 21s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:e01ee2f |
| JIRA Patch URL | 

[jira] [Updated] (HBASE-17194) Assign the new region to the idle server after splitting

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17194:
--
Status: Open  (was: Patch Available)

refactor due to [HBASE-17110|https://issues.apache.org/jira/browse/HBASE-17110]

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17194) Assign the new region to the idle server after splitting

[ChiaPing Tsai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711159#comment-15711159
 ] 

ChiaPing Tsai edited comment on HBASE-17194 at 12/1/16 7:22 AM:


refactoring due to 
[HBASE-17110|https://issues.apache.org/jira/browse/HBASE-17110]


was (Author: chia7712):
refactor due to [HBASE-17110|https://issues.apache.org/jira/browse/HBASE-17110]

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17057) Minor compactions should also drop page cache behind reads

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711138#comment-15711138
 ] 

Hadoop QA commented on HBASE-17057:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 47s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 137m 54s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 179m 58s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestMasterReplication |
|   | hadoop.hbase.client.TestHTableMultiplexer |
|   | hadoop.hbase.master.TestAssignmentManagerMetrics |
|   | hadoop.hbase.client.TestHTableMultiplexerFlushCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.2 Server=1.12.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841212/HBASE-17057.V1.patch |
| JIRA Issue | HBASE-17057 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 240f5c874cf2 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 
16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / b208687 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4733/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4733/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4733/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 

[jira] [Commented] (HBASE-17194) Assign the new region to the idle server after splitting

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711130#comment-15711130
 ] 

Hadoop QA commented on HBASE-17194:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 39s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 135m 0s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 177m 6s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestHTableMultiplexer |
|   | hadoop.hbase.client.TestHTableMultiplexerFlushCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.2 Server=1.12.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841209/HBASE-17194.v1.patch |
| JIRA Issue | HBASE-17194 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux b89ee8283899 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 
16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / b208687 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4734/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4734/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4734/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4734/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Assign the new region to the idle server after splitting
> 

[jira] [Commented] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1576#comment-1576
 ] 

ramkrishna.s.vasudevan commented on HBASE-16859:


But when I say two level copies - I don't mean for the actual data but the two 
level copies only happens for the UINT sizes that we normally write along with 
the actual data. Just for emphasizing saying this.

> Use Bytebuffer pool for non java clients specifically for scans/gets
> 
>
> Key: HBASE-16859
> URL: https://issues.apache.org/jira/browse/HBASE-16859
> Project: HBase
>  Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, 
> HBASE-16859_V2.patch
>
>
> In case of non java clients we still write the results and header into a on 
> demand  byte[]. This can be changed to use the BBPool (onheap or offheap 
> buffer?).
> But the basic problem is to identify if the response is for scans/gets. 
> - One easy way to do it is use the MethodDescriptor per Call and use the   
> name of the MethodDescriptor to identify it is a scan/get. But this will 
> pollute RpcServer by checking for scan/get type response.
> - Other way is always set the result to cellScanner but we know that 
> isClientCellBlockSupported is going to false for non PB clients. So ignore 
> the cellscanner and go ahead with the results in PB. But this is not clean
> - third one is that we already have a RpccallContext being passed to the RS. 
> In case of scan/gets/multiGets we already set a Rpccallback for shipped call. 
> So here on response we can check if the callback is not null and check for 
> isclientBlockSupported. In this case we can get the BB from the pool and 
> write the result and header to that BB. May be this looks clean?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17223) Make COS#newInstance(ByteOutput, int) as public API.

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1574#comment-1574
 ] 

ramkrishna.s.vasudevan commented on HBASE-17223:


ACtually am reading this PB things now than what I had previously. Will get 
back. 
One thing is that CIS is not of type ByteInput. But COS is of ByteOutput. So 
what ever you create for COS they will be for writeLazy also. Will get back 
here.

> Make COS#newInstance(ByteOutput, int) as public API.
> 
>
> Key: HBASE-17223
> URL: https://issues.apache.org/jira/browse/HBASE-17223
> Project: HBase
>  Issue Type: Sub-task
>  Components: Protobufs
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
>
> This is needed for HBASE-16859 since we need the COS#newInstance(ByteOutput) 
> version to be public to have ByteBuffByteOutput support.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17223) Make COS#newInstance(ByteOutput, int) as public API.

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711108#comment-15711108
 ] 

Anoop Sam John commented on HBASE-17223:


ALso how u think, to handle the ByteOuput based COS being a buffered OS? We 
will continue to use that?

> Make COS#newInstance(ByteOutput, int) as public API.
> 
>
> Key: HBASE-17223
> URL: https://issues.apache.org/jira/browse/HBASE-17223
> Project: HBase
>  Issue Type: Sub-task
>  Components: Protobufs
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
>
> This is needed for HBASE-16859 since we need the COS#newInstance(ByteOutput) 
> version to be public to have ByteBuffByteOutput support.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711079#comment-15711079
 ] 

ramkrishna.s.vasudevan edited comment on HBASE-16859 at 12/1/16 6:56 AM:
-

bq.hat should be a miss! Raise a new issue to push this change into shaded PB 
of ours
Will do that. Probably there is one more thing.
In CIS the ByteIutputDecoder was directly working on the ByteInput.
Now in COS we have an AbstractBufferedEncoder extended by ByteOutputEncoder.
So apart from the ByteOutput we also have the byte[] inside 
AbstractBufferedEncoder .
So when we try to write the UINT part we write to this byte[] and that is in 
turn getting flushed later to the ByteOutput as a byte[] or buffer.
In ARrayEncoder and BBEncoder we don't do this way. We directly write to the 
byte[] and BB. It is not an issue but the byte[] we init is additional and we 
have two level copies.



was (Author: ram_krish):
bq.hat should be a miss! Raise a new issue to push this change into shaded PB 
of ours
Will do that. Probably there is one more thing.
In CIS the ByteIutputDecoder was directly working on the ByteInput.
Now in COS we have an AbstractBufferedEncoder extended by ByteOutputEncoder.
So apart from the ByteOutput we also have the byte[] inside 
AbstractBufferedEncoder .
So when we try to write the unit part we write to this byte[] and that is in 
turn getting flushed later to the ByteOutput as a byte[] or buffer.
In ARrayEncoder and BBEncoder we don't do this way. We directly write to the 
byte[] and BB. It is not an issue but the byte[] we init is additional and we 
have two level copies.


> Use Bytebuffer pool for non java clients specifically for scans/gets
> 
>
> Key: HBASE-16859
> URL: https://issues.apache.org/jira/browse/HBASE-16859
> Project: HBase
>  Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, 
> HBASE-16859_V2.patch
>
>
> In case of non java clients we still write the results and header into a on 
> demand  byte[]. This can be changed to use the BBPool (onheap or offheap 
> buffer?).
> But the basic problem is to identify if the response is for scans/gets. 
> - One easy way to do it is use the MethodDescriptor per Call and use the   
> name of the MethodDescriptor to identify it is a scan/get. But this will 
> pollute RpcServer by checking for scan/get type response.
> - Other way is always set the result to cellScanner but we know that 
> isClientCellBlockSupported is going to false for non PB clients. So ignore 
> the cellscanner and go ahead with the results in PB. But this is not clean
> - third one is that we already have a RpccallContext being passed to the RS. 
> In case of scan/gets/multiGets we already set a Rpccallback for shipped call. 
> So here on response we can check if the callback is not null and check for 
> isclientBlockSupported. In this case we can get the BB from the pool and 
> write the result and header to that BB. May be this looks clean?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17223) Make COS#newInstance(ByteOutput, int) as public API.

[ramkrishna.s.vasudevan (JIRA)]

ramkrishna.s.vasudevan created HBASE-17223:
--

 Summary: Make COS#newInstance(ByteOutput, int) as public API.
 Key: HBASE-17223
 URL: https://issues.apache.org/jira/browse/HBASE-17223
 Project: HBase
  Issue Type: Sub-task
  Components: Protobufs
Affects Versions: 2.0.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan


This is needed for HBASE-16859 since we need the COS#newInstance(ByteOutput) 
version to be public to have ByteBuffByteOutput support.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying

[Xiang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiang Li updated HBASE-14882:
-
Attachment: HBASE-14882.master.004.patch

> Provide a Put API that adds the provided family, qualifier, value without 
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch, 
> HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, 
> HBASE-14882.master.003.patch, HBASE-14882.master.004.patch
>
>
> In the Put API, we have addImmutable()
> {code}
>  /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
>   public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being 
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family, 
> qualifier and value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying

[Xiang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiang Li updated HBASE-14882:
-
Hadoop Flags:   (was: Reviewed)
  Status: Patch Available  (was: Open)

> Provide a Put API that adds the provided family, qualifier, value without 
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch, 
> HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, 
> HBASE-14882.master.003.patch, HBASE-14882.master.004.patch
>
>
> In the Put API, we have addImmutable()
> {code}
>  /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
>   public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being 
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family, 
> qualifier and value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711097#comment-15711097
 ] 

Anoop Sam John commented on HBASE-16859:


I saw it now.  So the ByteOuput is treated the same was as an OS.  Am not sure 
then why we need a new interface.  Not sure abt the intent of adding this by PB 
team..  In that case, one can go with an impl of OS itself no need for a new 
Byteouput.  The ByteInput is created with below criteria
{code}
/**
 * An input for raw bytes. This is similar to an InputStream but it is offset 
addressable. All the
 * read APIs are relative.
 */
@ExperimentalApi
public abstract class ByteInput {
{code}
See we say it has to be byte addressable/accessible.  Means it is some what 
like BB or byte[].. Its use is specifically when the input is a collection of 
these kind of data structures. Then also once can make an IS around that and 
use but the extra byte[] and in btw copy to there will happen.  When things are 
offset accessible, it is not needed.  Seems like ByteOuput is designed to allow 
lazy writes only
{code}
 Lazy write operations: where the caller guarantees that it will 
never modify the
 * provided buffer and it can therefore be considered immutable. The target 
method is free to
 * maintain a reference to the buffer beyond the scope of the method call (e.g. 
until the write
 * operation completes).
{code}


> Use Bytebuffer pool for non java clients specifically for scans/gets
> 
>
> Key: HBASE-16859
> URL: https://issues.apache.org/jira/browse/HBASE-16859
> Project: HBase
>  Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, 
> HBASE-16859_V2.patch
>
>
> In case of non java clients we still write the results and header into a on 
> demand  byte[]. This can be changed to use the BBPool (onheap or offheap 
> buffer?).
> But the basic problem is to identify if the response is for scans/gets. 
> - One easy way to do it is use the MethodDescriptor per Call and use the   
> name of the MethodDescriptor to identify it is a scan/get. But this will 
> pollute RpcServer by checking for scan/get type response.
> - Other way is always set the result to cellScanner but we know that 
> isClientCellBlockSupported is going to false for non PB clients. So ignore 
> the cellscanner and go ahead with the results in PB. But this is not clean
> - third one is that we already have a RpccallContext being passed to the RS. 
> In case of scan/gets/multiGets we already set a Rpccallback for shipped call. 
> So here on response we can check if the callback is not null and check for 
> isclientBlockSupported. In this case we can get the BB from the pool and 
> write the result and header to that BB. May be this looks clean?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711093#comment-15711093
 ] 

ramkrishna.s.vasudevan commented on HBASE-16859:


TestHRegion seems to pass locally.

> Use Bytebuffer pool for non java clients specifically for scans/gets
> 
>
> Key: HBASE-16859
> URL: https://issues.apache.org/jira/browse/HBASE-16859
> Project: HBase
>  Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, 
> HBASE-16859_V2.patch
>
>
> In case of non java clients we still write the results and header into a on 
> demand  byte[]. This can be changed to use the BBPool (onheap or offheap 
> buffer?).
> But the basic problem is to identify if the response is for scans/gets. 
> - One easy way to do it is use the MethodDescriptor per Call and use the   
> name of the MethodDescriptor to identify it is a scan/get. But this will 
> pollute RpcServer by checking for scan/get type response.
> - Other way is always set the result to cellScanner but we know that 
> isClientCellBlockSupported is going to false for non PB clients. So ignore 
> the cellscanner and go ahead with the results in PB. But this is not clean
> - third one is that we already have a RpccallContext being passed to the RS. 
> In case of scan/gets/multiGets we already set a Rpccallback for shipped call. 
> So here on response we can check if the callback is not null and check for 
> isclientBlockSupported. In this case we can get the BB from the pool and 
> write the result and header to that BB. May be this looks clean?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711089#comment-15711089
 ] 

Yu Li commented on HBASE-17212:
---

Thanks for review sir.

> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.addendum.patch, HBASE-17212.addendum.patch, 
> HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}
> For RegionServerCallable, we're lacking of some null checker when using table 
> name, such as in {{RegionServerCallable#prepare}}:
> {code}
>public void prepare(final boolean reload) throws IOException {
>  // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
>throw new TableNotEnabledException(tableName.getNameAsString() + " is 
> disabled.");
>  }
> {code}
> It will throw NPE if tableName is null and invoking {{tableName.equals}}. 
> We'll add null checker in such places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17114:
--
Attachment: HBASE-17114.branch-1.patch

Uploading patch for branch-1

> Add an option to set special retry pause when encountering 
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.branch-1.patch, HBASE-17114.patch, 
> HBASE-17114.v2.patch, HBASE-17114.v3.patch, HBASE-17114.v3.patch, 
> HBASE-17114.v4.patch, HBASE-17114.v5.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} 
> instead of dead-wait. This is good for performance for most cases but might 
> cause a side-effect that if too many clients connect to the busy RS, that the 
> retry requests may come over and over again and RS never got the chance for 
> recovering, and the issue will become especially critical when the target 
> region is META.
> So here in this JIRA we propose to add a new property in name of 
> {{hbase.client.pause.cqtbe}} to make it possible to set a special-longer 
> pause for CallQueueTooBigException, and by default it will use the setting of 
> {{hbase.client.pause}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16859) Use Bytebuffer pool for non java clients specifically for scans/gets

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711079#comment-15711079
 ] 

ramkrishna.s.vasudevan commented on HBASE-16859:


bq.hat should be a miss! Raise a new issue to push this change into shaded PB 
of ours
Will do that. Probably there is one more thing.
In CIS the ByteIutputDecoder was directly working on the ByteInput.
Now in COS we have an AbstractBufferedEncoder extended by ByteOutputEncoder.
So apart from the ByteOutput we also have the byte[] inside 
AbstractBufferedEncoder .
So when we try to write the unit part we write to this byte[] and that is in 
turn getting flushed later to the ByteOutput as a byte[] or buffer.
In ARrayEncoder and BBEncoder we don't do this way. We directly write to the 
byte[] and BB. It is not an issue but the byte[] we init is additional and we 
have two level copies.


> Use Bytebuffer pool for non java clients specifically for scans/gets
> 
>
> Key: HBASE-16859
> URL: https://issues.apache.org/jira/browse/HBASE-16859
> Project: HBase
>  Issue Type: Sub-task
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-16859_V1.patch, HBASE-16859_V2.patch, 
> HBASE-16859_V2.patch
>
>
> In case of non java clients we still write the results and header into a on 
> demand  byte[]. This can be changed to use the BBPool (onheap or offheap 
> buffer?).
> But the basic problem is to identify if the response is for scans/gets. 
> - One easy way to do it is use the MethodDescriptor per Call and use the   
> name of the MethodDescriptor to identify it is a scan/get. But this will 
> pollute RpcServer by checking for scan/get type response.
> - Other way is always set the result to cellScanner but we know that 
> isClientCellBlockSupported is going to false for non PB clients. So ignore 
> the cellscanner and go ahead with the results in PB. But this is not clean
> - third one is that we already have a RpccallContext being passed to the RS. 
> In case of scan/gets/multiGets we already set a Rpccallback for shipped call. 
> So here on response we can check if the callback is not null and check for 
> isclientBlockSupported. In this case we can get the BB from the pool and 
> write the result and header to that BB. May be this looks clean?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17206) FSHLog may roll a new writer successfully with unflushed entries

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711055#comment-15711055
 ] 

Duo Zhang commented on HBASE-17206:
---

Pushed to branch-1.1+ except branch-1.3. Leave this issue open for a while to 
wait for the response of [~mantonov].

> FSHLog may roll a new writer successfully with unflushed entries
> 
>
> Key: HBASE-17206
> URL: https://issues.apache.org/jira/browse/HBASE-17206
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17206.patch
>
>
> Found it when debugging the flakey TestFailedAppendAndSync.
> The problem is in waitSafePoint.
> {code}
>   while (true) {
> if (this.safePointAttainedLatch.await(1, TimeUnit.MILLISECONDS)) {
>   break;
> }
> if (syncFuture.isThrowable()) {
>   throw new 
> FailedSyncBeforeLogCloseException(syncFuture.getThrowable());
> }
>   }
>   return syncFuture;
> {code}
> If we attach the safe point quick enough then we will bypass the 
> syncFuture.isThrowable check and will not throw 
> FailedSyncBeforeLogCloseException.
> This may cause incosistency between memstore and wal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711045#comment-15711045
 ] 

Hadoop QA commented on HBASE-16398:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 0s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 46s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 125m 4s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestHTableMultiplexer |
|   | hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat |
|   | hadoop.hbase.mapred.TestTableSnapshotInputFormat |
|   | hadoop.hbase.client.TestHTableMultiplexerFlushCache |
|   | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841207/HBASE-16398_v2.patch |
| JIRA Issue | HBASE-16398 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 2a548bbc4530 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / b6f5d5b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4732/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4732/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4732/testReport/ |
| modules | C: 

[jira] [Commented] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711031#comment-15711031
 ] 

stack commented on HBASE-17212:
---

+1

> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.addendum.patch, HBASE-17212.addendum.patch, 
> HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}
> For RegionServerCallable, we're lacking of some null checker when using table 
> name, such as in {{RegionServerCallable#prepare}}:
> {code}
>public void prepare(final boolean reload) throws IOException {
>  // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
>throw new TableNotEnabledException(tableName.getNameAsString() + " is 
> disabled.");
>  }
> {code}
> It will throw NPE if tableName is null and invoking {{tableName.equals}}. 
> We'll add null checker in such places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17216) A Few Fields Can Be Safely Made Static

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-17216:
--
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master branch. Thanks for the patch [~jleach] (Test failures are 
unrelated)

> A Few Fields Can Be Safely Made Static
> --
>
> Key: HBASE-17216
> URL: https://issues.apache.org/jira/browse/HBASE-17216
> Project: HBase
>  Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
> Fix For: 2.0.0
>
> Attachments: HBASE-17216.patch
>
>
> Automated Test...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17221) Abstract out an interface for RpcServer.Call

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711017#comment-15711017
 ] 

stack commented on HBASE-17221:
---

I was expecting the Interface to be Call and the implementation CallImpl.

> Abstract out an interface for RpcServer.Call
> 
>
> Key: HBASE-17221
> URL: https://issues.apache.org/jira/browse/HBASE-17221
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-17221.patch
>
>
> RpcServer.Call is a concrete class, but it is marked as:
> {noformat}
> @InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC, 
> HBaseInterfaceAudience.PHOENIX})
> {noformat}
> Let's abstract out an interface out of it for potential consumers that want 
> to pass it around.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17205) Add a metric for the duration of region in transition

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-17205:
---
Attachment: HBASE-17205-v1.patch

There were no precommit job run for v1. Attach again.

> Add a metric for the duration of region in transition
> -
>
> Key: HBASE-17205
> URL: https://issues.apache.org/jira/browse/HBASE-17205
> Project: HBase
>  Issue Type: Improvement
>  Components: Region Assignment
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17205-branch-1.patch, HBASE-17205-v1.patch, 
> HBASE-17205-v1.patch, HBASE-17205.patch
>
>
> When work for HBASE-17178, I found there are not a metric for the overall 
> duration of region in transition. When move a region form A to B, the 
> transformation of region state is PENDING_CLOSE => CLOSING => CLOSED => 
> PENDING_OPEN => OPENING => OPENED. When transform old region state to new 
> region state, it update the time stamp to current time. So we can't get the 
> overall transformation's duration of region in transition. Add a rit duration 
> to RegionState for accumulating this metric.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710985#comment-15710985
 ] 

Yu Li commented on HBASE-17114:
---

Pushed into master branch. Thanks all for review [~ghelmling] [~anoop.hbase] 
[~tedyu] [~zghaobac]

Will prepare a patch for branch-1 soon.

> Add an option to set special retry pause when encountering 
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch, HBASE-17114.v2.patch, 
> HBASE-17114.v3.patch, HBASE-17114.v3.patch, HBASE-17114.v4.patch, 
> HBASE-17114.v5.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} 
> instead of dead-wait. This is good for performance for most cases but might 
> cause a side-effect that if too many clients connect to the busy RS, that the 
> retry requests may come over and over again and RS never got the chance for 
> recovering, and the issue will become especially critical when the target 
> region is META.
> So here in this JIRA we propose to add a new property in name of 
> {{hbase.client.pause.cqtbe}} to make it possible to set a special-longer 
> pause for CallQueueTooBigException, and by default it will use the setting of 
> {{hbase.client.pause}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710979#comment-15710979
 ] 

Yu Li commented on HBASE-17114:
---

Thanks for the confirmation sir, committing now (smile).

> Add an option to set special retry pause when encountering 
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch, HBASE-17114.v2.patch, 
> HBASE-17114.v3.patch, HBASE-17114.v3.patch, HBASE-17114.v4.patch, 
> HBASE-17114.v5.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} 
> instead of dead-wait. This is good for performance for most cases but might 
> cause a side-effect that if too many clients connect to the busy RS, that the 
> retry requests may come over and over again and RS never got the chance for 
> recovering, and the issue will become especially critical when the target 
> region is META.
> So here in this JIRA we propose to add a new property in name of 
> {{hbase.client.pause.cqtbe}} to make it possible to set a special-longer 
> pause for CallQueueTooBigException, and by default it will use the setting of 
> {{hbase.client.pause}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17212:
--
Comment: was deleted

(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color} 
| {color:red} HBASE-17212 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841224/HBASE-17212.addendum.patch
 |
| JIRA Issue | HBASE-17212 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4738/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.

)

> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.addendum.patch, HBASE-17212.addendum.patch, 
> HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}
> For RegionServerCallable, we're lacking of some null checker when using table 
> name, such as in {{RegionServerCallable#prepare}}:
> {code}
>public void prepare(final boolean reload) throws IOException {
>  // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
>throw new TableNotEnabledException(tableName.getNameAsString() + " is 
> disabled.");
>  }
> {code}
> It will throw NPE if tableName is null and invoking {{tableName.equals}}. 
> We'll add null checker in such places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17212:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Addendum pushed into master and branch-1.

> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.addendum.patch, HBASE-17212.addendum.patch, 
> HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}
> For RegionServerCallable, we're lacking of some null checker when using table 
> name, such as in {{RegionServerCallable#prepare}}:
> {code}
>public void prepare(final boolean reload) throws IOException {
>  // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
>throw new TableNotEnabledException(tableName.getNameAsString() + " is 
> disabled.");
>  }
> {code}
> It will throw NPE if tableName is null and invoking {{tableName.equals}}. 
> We'll add null checker in such places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710960#comment-15710960
 ] 

Anoop Sam John commented on HBASE-17114:


Apart from those 2 I was +1 :-)   Go for commit.

> Add an option to set special retry pause when encountering 
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch, HBASE-17114.v2.patch, 
> HBASE-17114.v3.patch, HBASE-17114.v3.patch, HBASE-17114.v4.patch, 
> HBASE-17114.v5.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} 
> instead of dead-wait. This is good for performance for most cases but might 
> cause a side-effect that if too many clients connect to the busy RS, that the 
> retry requests may come over and over again and RS never got the chance for 
> recovering, and the issue will become especially critical when the target 
> region is META.
> So here in this JIRA we propose to add a new property in name of 
> {{hbase.client.pause.cqtbe}} to make it possible to set a special-longer 
> pause for CallQueueTooBigException, and by default it will use the setting of 
> {{hbase.client.pause}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710958#comment-15710958
 ] 

Hadoop QA commented on HBASE-17212:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color} 
| {color:red} HBASE-17212 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841224/HBASE-17212.addendum.patch
 |
| JIRA Issue | HBASE-17212 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4738/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.addendum.patch, HBASE-17212.addendum.patch, 
> HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}
> For RegionServerCallable, we're lacking of some null checker when using table 
> name, such as in {{RegionServerCallable#prepare}}:
> {code}
>public void prepare(final boolean reload) throws IOException {
>  // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
>throw new TableNotEnabledException(tableName.getNameAsString() + " is 
> disabled.");
>  }
> {code}
> It will throw NPE if tableName is null and invoking {{tableName.equals}}. 
> We'll add null checker in such places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17187) DoNotRetryExceptions from coprocessors should bubble up to the application

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710950#comment-15710950
 ] 

Anoop Sam John commented on HBASE-17187:


See RegionScannerImpl#next
if (this.filterClosed) {
throw new UnknownScannerException("Scanner was closed (timed out?) " +
"after we renewed it. Could be caused by a very slow scanner " +
"or a lengthy garbage collection");
  }
So here we throw UnknownScannerException and RsRpcServices catch IOE and the 
new code is having a DNRIOE check and now we will throw back 
UnknownScannerException  not ScannerResetException.  I think that is ok.  Or 
not?
Another place is this
public boolean nextRaw(List outResults, ScannerContext scannerContext)
throws IOException {
  if (storeHeap == null) {
// scanner is closed
throw new UnknownScannerException("Scanner was closed");
  }

Just trying to understand the impact of the change fully. Tks.

> DoNotRetryExceptions from coprocessors should bubble up to the application
> --
>
> Key: HBASE-17187
> URL: https://issues.apache.org/jira/browse/HBASE-17187
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: hbase-17187_v1.patch
>
>
> In HBASE-16604, we fixed a case where scanner retries was causing the scan to 
> miss some data in case the scanner is left with a dirty state (like a 
> half-seeked KVHeap). 
> The patch introduced a minor compatibility issue, because now if a 
> coprocessor throws DNRIOE, we still retry the ClientScanner indefinitely. 
> The test {{ServerExceptionIT}} in Phoenix is failing because of this with 
> HBASE-16604. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710942#comment-15710942
 ] 

Hadoop QA commented on HBASE-17212:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m 19s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 50s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
31m 37s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 39s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 36s 
{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
21s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 10s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.10.1 Server=1.10.1 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841213/HBASE-17212.addendum.patch
 |
| JIRA Issue | HBASE-17212 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 34aa245ec51e 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 
24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / b208687 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4736/testReport/ |
| modules | C: hbase-client hbase-endpoint U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4736/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Should add null checker on 

[jira] [Updated] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17212:
--
Attachment: HBASE-17212.addendum.patch

> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.addendum.patch, HBASE-17212.addendum.patch, 
> HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}
> For RegionServerCallable, we're lacking of some null checker when using table 
> name, such as in {{RegionServerCallable#prepare}}:
> {code}
>public void prepare(final boolean reload) throws IOException {
>  // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
>throw new TableNotEnabledException(tableName.getNameAsString() + " is 
> disabled.");
>  }
> {code}
> It will throw NPE if tableName is null and invoking {{tableName.equals}}. 
> We'll add null checker in such places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17205) Add a metric for the duration of region in transition

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710926#comment-15710926
 ] 

Guanghao Zhang commented on HBASE-17205:


Failed ut are related to HBASE-17212.

> Add a metric for the duration of region in transition
> -
>
> Key: HBASE-17205
> URL: https://issues.apache.org/jira/browse/HBASE-17205
> Project: HBase
>  Issue Type: Improvement
>  Components: Region Assignment
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17205-branch-1.patch, HBASE-17205-v1.patch, 
> HBASE-17205.patch
>
>
> When work for HBASE-17178, I found there are not a metric for the overall 
> duration of region in transition. When move a region form A to B, the 
> transformation of region state is PENDING_CLOSE => CLOSING => CLOSED => 
> PENDING_OPEN => OPENING => OPENED. When transform old region state to new 
> region state, it update the time stamp to current time. So we can't get the 
> overall transformation's duration of region in transition. Add a rit duration 
> to RegionState for accumulating this metric.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17205) Add a metric for the duration of region in transition

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710878#comment-15710878
 ] 

Hadoop QA commented on HBASE-17205:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 32s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 16s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
14s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
56s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
57s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 57s 
{color} | {color:red} hbase-server in branch-1 has 2 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s 
{color} | {color:green} branch-1 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
14m 9s {color} | {color:green} The patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s 
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s 
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 41s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
48s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 33s 

[jira] [Commented] (HBASE-17210) Set timeout on trying rowlock according to client's RPC timeout

[Phil Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710817#comment-15710817
 ] 

Phil Yang commented on HBASE-17210:
---

Thanks for your review.

{quote}
Should timeout be set to rowLockWaitDuration if (call.getDeadline() - 
System.currentTimeMillis()) <= 0 ?
{quote}
No, it means the previous logic (eg. waiting in request queue and maybe a full 
gc) spend too much time and this request has been regarded as timeout at 
client, we can drop it.

{quote}
Please use EnvironmentEdgeManager.currentTime()
{quote}
It seems that after HBASE-16256 we are advised not to inject any 
EnvironmentEdge so we can use System.currentTimeMillis() directly?

> Set timeout on trying rowlock according to client's RPC timeout
> ---
>
> Key: HBASE-17210
> URL: https://issues.apache.org/jira/browse/HBASE-17210
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-17120.v1.patch
>
>
> Now when we want to get a row lock, the timeout is fixed and default is 30s. 
> But the requests from client have different RPC timeout setting. We can use 
> the client's deadline to set timeout on tryLock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17110) Improve SimpleLoadBalancer to always take server-level balance into account

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710807#comment-15710807
 ] 

Yu Li commented on HBASE-17110:
---

Pushed into master branch, could not directly apply to branch-1 so will wait 
for the branch-1 patch before closing this JIRA.

Thanks all for review.

> Improve SimpleLoadBalancer to always take server-level balance into account
> ---
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Fix For: 2.0.0
>
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch, 
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110-V6.patch, 
> HBASE-17110-V7.patch, HBASE-17110-V8.patch, HBASE-17110.patch
>
>
> Currently with bytable strategy there might still be server-level imbalance 
> and we will improve this in this JIRA.
> Some more background:
> When operating large scale clusters(our case), some companies still prefer to 
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan 
> generation, etc. Current SimpleLoadBalancer has two modes: 
> 1. byTable, which only guarantees that the regions of one table could be 
> uniformly distributed. 
> 2. byCluster, which ignores the distribution within tables and balance the 
> regions all together.
> If the pressures on different tables are different, the first byTable option 
> is the preferable one in most case. Yet, this choice sacrifice the cluster 
> level balance and would cause some servers to have significantly higher load, 
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case,  a cluster has 3 tables and 4 servers:
> {noformat}
>   server A has 3 regions: table1:1, table2:1, table3:1
>   server B has 3 regions: table1:2, table2:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been 
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
>   server A has 2 regions: table2:1, table3:1
>   server B has 2 regions: table1:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1, 
> table2 and table3 still keep balanced. And this is the goal this JIRA tries 
> to achieve.
> Two UTs will be added as well with the last one demonstrating advantage of 
> the new strategy. Also, a onConfigurationChange method will be implemented to 
> hot control the "slop" variable.
> We have been using the strategy on our largest cluster for several months, so 
> the effect could be assured to some extent.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16648) [JDK8] Use computeIfAbsent instead of get and putIfAbsent

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-16648:
--
Attachment: HBASE-16648-v5.patch

Remove debug code. Will commit after the pre commit job finish.

> [JDK8] Use computeIfAbsent instead of get and putIfAbsent
> -
>
> Key: HBASE-16648
> URL: https://issues.apache.org/jira/browse/HBASE-16648
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16648-v1.patch, HBASE-16648-v2.patch, 
> HBASE-16648-v3.patch, HBASE-16648-v4.patch, HBASE-16648-v5.patch, 
> HBASE-16648.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17110) Improve SimpleLoadBalancer to always take server-level balance into account

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17110:
--
 Hadoop Flags: Reviewed
Fix Version/s: 2.0.0
 Release Note: After HBASE-17110 the bytable strategy for 
SimpleLoadBalancer will also take server level balance into account
  Description: 
Currently with bytable strategy there might still be server-level imbalance and 
we will improve this in this JIRA.

Some more background:
When operating large scale clusters(our case), some companies still prefer to 
use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan 
generation, etc. Current SimpleLoadBalancer has two modes: 
1. byTable, which only guarantees that the regions of one table could be 
uniformly distributed. 
2. byCluster, which ignores the distribution within tables and balance the 
regions all together.
If the pressures on different tables are different, the first byTable option is 
the preferable one in most case. Yet, this choice sacrifice the cluster level 
balance and would cause some servers to have significantly higher load, e.g. 
242 regions on server A but 417 regions on server B.(real world stats)
Consider this case,  a cluster has 3 tables and 4 servers:
{noformat}
  server A has 3 regions: table1:1, table2:1, table3:1
  server B has 3 regions: table1:2, table2:2, table3:2
  server C has 3 regions: table1:3, table2:3, table3:3
  server D has 0 regions.
{noformat}
>From the byTable strategy's perspective, the cluster has already been 
>perfectly balanced on table level. But a perfect status should be like:
{noformat}
  server A has 2 regions: table2:1, table3:1
  server B has 2 regions: table1:2, table3:2
  server C has 3 regions: table1:3, table2:3, table3:3
  server D has 2 regions: table1:1, table2:2
{noformat}
We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1, 
table2 and table3 still keep balanced. And this is the goal this JIRA tries to 
achieve.

Two UTs will be added as well with the last one demonstrating advantage of the 
new strategy. Also, a onConfigurationChange method will be implemented to hot 
control the "slop" variable.

We have been using the strategy on our largest cluster for several months, so 
the effect could be assured to some extent.



 

  was:
This jira is about an enhancement of simpleLoadBalancer. Here we introduce a 
new strategy: "bytableOverall" which could be controlled by adding:
{noformat}

  hbase.master.loadbalance.bytableOverall
  true

{noformat}
We have been using the strategy on our largest cluster for several months. it's 
proven to be very helpful and stable, especially, the result is quite visible 
to the users.

Here is the reason why it's helpful:
When operating large scale clusters(our case), some companies still prefer to 
use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan 
generation, etc. Current SimpleLoadBalancer has two modes: 
1. byTable, which only guarantees that the regions of one table could be 
uniformly distributed. 
2. byCluster, which ignores the distribution within tables and balance the 
regions all together.
If the pressures on different tables are different, the first byTable option is 
the preferable one in most case. Yet, this choice sacrifice the cluster level 
balance and would cause some servers to have significantly higher load, e.g. 
242 regions on server A but 417 regions on server B.(real world stats)
Consider this case,  a cluster has 3 tables and 4 servers:
{noformat}
  server A has 3 regions: table1:1, table2:1, table3:1
  server B has 3 regions: table1:2, table2:2, table3:2
  server C has 3 regions: table1:3, table2:3, table3:3
  server D has 0 regions.
{noformat}
>From the byTable strategy's perspective, the cluster has already been 
>perfectly balanced on table level. But a perfect status should be like:
{noformat}
  server A has 2 regions: table2:1, table3:1
  server B has 2 regions: table1:2, table3:2
  server C has 3 regions: table1:3, table2:3, table3:3
  server D has 2 regions: table1:1, table2:2
{noformat}
We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1, 
table2 and table3 still keep balanced.   
And this is what the new mode "byTableOverall" can achieve.

Two UTs have been added as well and the last one demonstrates the advantage of 
the new strategy.
Also, a onConfigurationChange method has been implemented to hot control the 
"slop" variable.



 

  Summary: Improve SimpleLoadBalancer to always take server-level 
balance into account  (was: Improve SimpleLoadBalancer to consider server level 
balance)

Update the description and add release note

> Improve SimpleLoadBalancer to always take server-level balance into account
> ---
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
>   

[jira] [Updated] (HBASE-17110) Improve SimpleLoadBalancer to consider server level balance

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17110:
--
Summary: Improve SimpleLoadBalancer to consider server level balance  (was: 
Add an "Overall Strategy" option(balanced both on table level and server level) 
to SimpleLoadBalancer)

> Improve SimpleLoadBalancer to consider server level balance
> ---
>
> Key: HBASE-17110
> URL: https://issues.apache.org/jira/browse/HBASE-17110
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer
>Affects Versions: 2.0.0, 1.2.4
>Reporter: Charlie Qiangeng Xu
>Assignee: Charlie Qiangeng Xu
> Attachments: HBASE-17110-V2.patch, HBASE-17110-V3.patch, 
> HBASE-17110-V4.patch, HBASE-17110-V5.patch, HBASE-17110-V6.patch, 
> HBASE-17110-V7.patch, HBASE-17110-V8.patch, HBASE-17110.patch
>
>
> This jira is about an enhancement of simpleLoadBalancer. Here we introduce a 
> new strategy: "bytableOverall" which could be controlled by adding:
> {noformat}
> 
>   hbase.master.loadbalance.bytableOverall
>   true
> 
> {noformat}
> We have been using the strategy on our largest cluster for several months. 
> it's proven to be very helpful and stable, especially, the result is quite 
> visible to the users.
> Here is the reason why it's helpful:
> When operating large scale clusters(our case), some companies still prefer to 
> use {{SimpleLoadBalancer}} due to its simplicity, quick balance plan 
> generation, etc. Current SimpleLoadBalancer has two modes: 
> 1. byTable, which only guarantees that the regions of one table could be 
> uniformly distributed. 
> 2. byCluster, which ignores the distribution within tables and balance the 
> regions all together.
> If the pressures on different tables are different, the first byTable option 
> is the preferable one in most case. Yet, this choice sacrifice the cluster 
> level balance and would cause some servers to have significantly higher load, 
> e.g. 242 regions on server A but 417 regions on server B.(real world stats)
> Consider this case,  a cluster has 3 tables and 4 servers:
> {noformat}
>   server A has 3 regions: table1:1, table2:1, table3:1
>   server B has 3 regions: table1:2, table2:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 0 regions.
> {noformat}
> From the byTable strategy's perspective, the cluster has already been 
> perfectly balanced on table level. But a perfect status should be like:
> {noformat}
>   server A has 2 regions: table2:1, table3:1
>   server B has 2 regions: table1:2, table3:2
>   server C has 3 regions: table1:3, table2:3, table3:3
>   server D has 2 regions: table1:1, table2:2
> {noformat}
> We can see the server loads change from 3,3,3,0 to 2,2,3,2, while the table1, 
> table2 and table3 still keep balanced.   
> And this is what the new mode "byTableOverall" can achieve.
> Two UTs have been added as well and the last one demonstrates the advantage 
> of the new strategy.
> Also, a onConfigurationChange method has been implemented to hot control the 
> "slop" variable.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17114) Add an option to set special retry pause when encountering CallQueueTooBigException

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710770#comment-15710770
 ] 

Yu Li commented on HBASE-17114:
---

Ok, let me commit this one if no more comments/objections (smile).

> Add an option to set special retry pause when encountering 
> CallQueueTooBigException
> ---
>
> Key: HBASE-17114
> URL: https://issues.apache.org/jira/browse/HBASE-17114
> Project: HBase
>  Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-17114.patch, HBASE-17114.v2.patch, 
> HBASE-17114.v3.patch, HBASE-17114.v3.patch, HBASE-17114.v4.patch, 
> HBASE-17114.v5.patch
>
>
> As titled, after HBASE-15146 we will throw {{CallQueueTooBigException}} 
> instead of dead-wait. This is good for performance for most cases but might 
> cause a side-effect that if too many clients connect to the busy RS, that the 
> retry requests may come over and over again and RS never got the chance for 
> recovering, and the issue will become especially critical when the target 
> region is META.
> So here in this JIRA we propose to add a new property in name of 
> {{hbase.client.pause.cqtbe}} to make it possible to set a special-longer 
> pause for CallQueueTooBigException, and by default it will use the setting of 
> {{hbase.client.pause}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17208) Manual Array Copy Cleanup: Automated

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710763#comment-15710763
 ] 

Duo Zhang commented on HBASE-17208:
---

I think we can use Arrays.copyOf and Arrays.copyOfRange for some of the cases 
in the patch? It can save one extra new array line in our code base.

Thanks.

> Manual Array Copy Cleanup: Automated
> 
>
> Key: HBASE-17208
> URL: https://issues.apache.org/jira/browse/HBASE-17208
> Project: HBase
>  Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: HBASE-17208.patch
>
>
> Remove Manual Array Copies: Automated



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17181) Let HBase thrift2 support TThreadedSelectorServer

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710759#comment-15710759
 ] 

Duo Zhang commented on HBASE-17181:
---

You can generate a patch from your inhouse repo first, and then apply it to the 
offical 1.2 branch(the name is branch-1.2). Fix conflicts if any, and then 
generate the patch with git format-patch(I do not know whether the 
submit-patch.py works on branch-1.2, you can try).

Thanks for your patience.

> Let HBase thrift2 support TThreadedSelectorServer
> -
>
> Key: HBASE-17181
> URL: https://issues.apache.org/jira/browse/HBASE-17181
> Project: HBase
>  Issue Type: New Feature
>  Components: Thrift
>Affects Versions: 2.0.0, 1.4.0, 1.2.4
>Reporter: Jian Yi
>Assignee: Jian Yi
>Priority: Minor
>  Labels: features
> Fix For: 2.0.0, 1.4.0, 1.2.5
>
> Attachments: HBASE-17181-V1.patch, HBASE-17181-V2.patch, 
> HBASE-17181-V3.patch, HBASE-17181-V4.patch, HBASE-17181-V5.patch, 
> HBASE-17181-V6.patch, ThriftServer.java
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Add TThreadedSelectorServer for HBase Thrift2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17212:
--
Status: Patch Available  (was: Reopened)

Try HadoopQA

> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.addendum.patch, HBASE-17212.patch, 
> HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}
> For RegionServerCallable, we're lacking of some null checker when using table 
> name, such as in {{RegionServerCallable#prepare}}:
> {code}
>public void prepare(final boolean reload) throws IOException {
>  // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
>throw new TableNotEnabledException(tableName.getNameAsString() + " is 
> disabled.");
>  }
> {code}
> It will throw NPE if tableName is null and invoking {{tableName.equals}}. 
> We'll add null checker in such places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17212:
--
Attachment: HBASE-17212.addendum.patch

Uploading the addendum patch, have manually checked and confirmed 
{{TestHTableMultiplexer}} and {{TestHTableMultiplexerFlushCache}} could pass 
with it.

Since the change in addendum is straight forward and harmless, plan to commit 
it after HadoopQA to avoid blocking post commit.

> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.addendum.patch, HBASE-17212.patch, 
> HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}
> For RegionServerCallable, we're lacking of some null checker when using table 
> name, such as in {{RegionServerCallable#prepare}}:
> {code}
>public void prepare(final boolean reload) throws IOException {
>  // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
>throw new TableNotEnabledException(tableName.getNameAsString() + " is 
> disabled.");
>  }
> {code}
> It will throw NPE if tableName is null and invoking {{tableName.equals}}. 
> We'll add null checker in such places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17212:
--
Description: 
If we run below codes:
{code}
Table table = connection.getTable(null);
{code}
we will see below exception:
{noformat}
java.lang.NullPointerException
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
at 
org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
at 
org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
{noformat}

And in this JIRA we will add a null checker and throw a more graceful 
{{IllegalArgumentException}}

For RegionServerCallable, we're lacking of some null checker when using table 
name, such as in {{RegionServerCallable#prepare}}:
{code}
   public void prepare(final boolean reload) throws IOException {
 // check table state if this is a retry
if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
getConnection().isTableDisabled(tableName)) {
   throw new TableNotEnabledException(tableName.getNameAsString() + " is 
disabled.");
 }
{code}
It will throw NPE if tableName is null and invoking {{tableName.equals}}. We'll 
add null checker in such places.

  was:
If we run below codes:
{code}
Table table = connection.getTable(null);
{code}
we will see below exception:
{noformat}
java.lang.NullPointerException
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
at 
org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
at 
org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
{noformat}

And in this JIRA we will add a null checker and throw a more graceful 
{{IllegalArgumentException}}


> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}
> For RegionServerCallable, we're lacking of some null checker when using table 
> name, such as in {{RegionServerCallable#prepare}}:
> {code}
>public void prepare(final boolean reload) throws IOException {
>  // check table state if this is a retry
> if (reload && !tableName.equals(TableName.META_TABLE_NAME) &&
> getConnection().isTableDisabled(tableName)) {
>throw new TableNotEnabledException(tableName.getNameAsString() + " is 
> disabled.");
>  }
> {code}
> It will throw NPE if tableName is null and invoking {{tableName.equals}}. 
> We'll add null checker in such places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17057) Minor compactions should also drop page cache behind reads

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-17057:
--
Attachment: HBASE-17057.V1.patch

V1: Adds config for dropping page cache behind major and minor compactions. 
Both are enabled by default.

> Minor compactions should also drop page cache behind reads
> --
>
> Key: HBASE-17057
> URL: https://issues.apache.org/jira/browse/HBASE-17057
> Project: HBase
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Attachments: HBASE-17057.V1.patch
>
>
> Long compactions currently drop cache behind reads/writes so that they don't 
> pollute the page cache but short compactions don't do that. The bulk of the 
> data is actually read during minor compactions instead of major compactions,  
> and thrashes the page cache since it's mostly not needed. 
> We should drop page cache behind minor compactions too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17057) Minor compactions should also drop page cache behind reads

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-17057:
--
Status: Patch Available  (was: Open)

> Minor compactions should also drop page cache behind reads
> --
>
> Key: HBASE-17057
> URL: https://issues.apache.org/jira/browse/HBASE-17057
> Project: HBase
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
> Attachments: HBASE-17057.V1.patch
>
>
> Long compactions currently drop cache behind reads/writes so that they don't 
> pollute the page cache but short compactions don't do that. The bulk of the 
> data is actually read during minor compactions instead of major compactions,  
> and thrashes the page cache since it's mostly not needed. 
> We should drop page cache behind minor compactions too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-17212) Should add null checker on table name in HTable and RegionServerCallable constructor

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li reopened HBASE-17212:
---

Reopening to address post commit UT failure.

> Should add null checker on table name in HTable and RegionServerCallable 
> constructor
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17212) Should add null checker on table name in HTable constructor and RegionServerCallable

[Yu Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Li updated HBASE-17212:
--
Summary: Should add null checker on table name in HTable constructor and 
RegionServerCallable  (was: Should add null checker on table name in HTable and 
RegionServerCallable constructor)

> Should add null checker on table name in HTable constructor and 
> RegionServerCallable
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17212) Should add null checker on table name in HTable and RegionServerCallable constructor

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710701#comment-15710701
 ] 

Yu Li commented on HBASE-17212:
---

Checking the post commit UT report, the change here has introduced a regression 
failure on TestHTableMultiplexer (thanks [~tedyu] for the reminder), and we 
could see below exception in the UT log:
{noformat}
2016-11-30 22:53:00,301 ERROR [htable-pool20-t1] 
client.AsyncRequestFutureImpl$SingleServerRequestRunnable(259): Internal 
AsyncProcess #24 error for null processing for 
asf902.gq1.ygridcore.net,44046,1480546369625
java.lang.IllegalArgumentException: Given tableName is null
at 
org.apache.hadoop.hbase.client.RegionServerCallable.(RegionServerCallable.java:82)
at 
org.apache.hadoop.hbase.client.ClientServiceCallable.(ClientServiceCallable.java:38)
at 
org.apache.hadoop.hbase.client.CancellableRegionServerCallable.(CancellableRegionServerCallable.java:44)
at 
org.apache.hadoop.hbase.client.MultiServerCallable.(MultiServerCallable.java:59)
at 
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.createCallable(AsyncRequestFutureImpl.java:1298)
at 
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.access$1000(AsyncRequestFutureImpl.java:70)
at 
org.apache.hadoop.hbase.client.AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncRequestFutureImpl.java:223)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
{noformat}

After investigation, there's some case we allow cross-table Rpc call through 
{{AsyncProcess#submitMultiActions}}, just like it does in 
{{HTableMultiplexer}}, which will group the puts from different tables against 
the same RS and send in a batch. And we could also see special handling for 
such cross-table call in {{AsyncRequestFutureImpl#receiveGlobalFailure}}:
{code}
if (tableName == null && ClientExceptionsUtil.isMetaClearingException(t)) {
  // tableName is null when we made a cross-table RPC call.
  asyncProcess.connection.clearCaches(server);
}
{code}

So instead of forbidding table name to be null in RegionServerCallable 
constructor, we should add null checker on table name wherever it's used inside 
RegionServerCallable.

Will make an addendum soon.

> Should add null checker on table name in HTable and RegionServerCallable 
> constructor
> 
>
> Key: HBASE-17212
> URL: https://issues.apache.org/jira/browse/HBASE-17212
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-17212.patch, HBASE-17212.v2.patch
>
>
> If we run below codes:
> {code}
> Table table = connection.getTable(null);
> {code}
> we will see below exception:
> {noformat}
> java.lang.NullPointerException
>   at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:221)
>   at org.apache.hadoop.hbase.client.HTable.(HTable.java:182)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:298)
>   at 
> org.apache.hadoop.hbase.client.ConnectionImplementation.getTable(ConnectionImplementation.java:293)
> {noformat}
> And in this JIRA we will add a null checker and throw a more graceful 
> {{IllegalArgumentException}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16119) Procedure v2 - Reimplement merge

[Stephen Yuan Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Yuan Jiang updated HBASE-16119:
---
Status: Patch Available  (was: Open)

> Procedure v2 - Reimplement merge
> 
>
> Key: HBASE-16119
> URL: https://issues.apache.org/jira/browse/HBASE-16119
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Stephen Yuan Jiang
> Fix For: 2.0.0
>
> Attachments: HBASE-16119.v1-master.patch
>
>
> use the proc-v2 state machine for merge. also update the logic to have a 
> single meta-writer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution

[binlijin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710679#comment-15710679
 ] 

binlijin commented on HBASE-16398:
--

Most of the code copy from you.

> optimize HRegion computeHDFSBlocksDistribution
> --
>
> Key: HBASE-16398
> URL: https://issues.apache.org/jira/browse/HBASE-16398
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0
>
> Attachments: HBASE-16398.patch, HBASE-16398_v2.patch, 
> LocatedBlockStatusComparison.java
>
>
> First i assume there is no reference and link in a region family's directory. 
> Without the patch to computeHDFSBlocksDistribution for a region family, there 
> is 1+2*N rpc call, N is hfile numbers, The first rpc call is to 
> DistributedFileSystem#listStatus to get hfiles, for every hfile there is two 
> rpc call DistributedFileSystem#getFileStatus(path) and then 
> DistributedFileSystem#getFileBlockLocations(status, start, length).
> With the patch to computeHDFSBlocksDistribution for a region family, there is 
> 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and  
> DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter 
> filter).
> So if there is at least one hfile, with the patch, the rpc call will less.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17221) Abstract out an interface for RpcServer.Call

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710680#comment-15710680
 ] 

Hadoop QA commented on HBASE-17221:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 7s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 12 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 58s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 18s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
14s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 55s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestHTableMultiplexerFlushCache |
|   | hadoop.hbase.client.TestHTableMultiplexer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841178/HBASE-17221.patch |
| JIRA Issue | HBASE-17221 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux dfaf765323eb 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / ad857d1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4728/artifact/patchprocess/whitespace-eol.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4728/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4728/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4728/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console 

[jira] [Commented] (HBASE-16261) MultiHFileOutputFormat Enhancement

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710673#comment-15710673
 ] 

Hadoop QA commented on HBASE-16261:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
24m 26s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 13s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
11s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 125m 14s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestHTableMultiplexer |
|   | hadoop.hbase.client.TestHTableMultiplexerFlushCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841183/HBASE-16261-V5.patch |
| JIRA Issue | HBASE-16261 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 71969d35360c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / ad857d1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4730/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4730/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4730/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4730/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



>  MultiHFileOutputFormat Enhancement 
> 
>
>  

[jira] [Commented] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution

[binlijin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710675#comment-15710675
 ] 

binlijin commented on HBASE-16398:
--

[~thiruvel] what do you think of the new patch HBASE-16398_v2.patch? 

> optimize HRegion computeHDFSBlocksDistribution
> --
>
> Key: HBASE-16398
> URL: https://issues.apache.org/jira/browse/HBASE-16398
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0
>
> Attachments: HBASE-16398.patch, HBASE-16398_v2.patch, 
> LocatedBlockStatusComparison.java
>
>
> First i assume there is no reference and link in a region family's directory. 
> Without the patch to computeHDFSBlocksDistribution for a region family, there 
> is 1+2*N rpc call, N is hfile numbers, The first rpc call is to 
> DistributedFileSystem#listStatus to get hfiles, for every hfile there is two 
> rpc call DistributedFileSystem#getFileStatus(path) and then 
> DistributedFileSystem#getFileBlockLocations(status, start, length).
> With the patch to computeHDFSBlocksDistribution for a region family, there is 
> 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and  
> DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter 
> filter).
> So if there is at least one hfile, with the patch, the rpc call will less.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution

[binlijin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

binlijin updated HBASE-16398:
-
Status: Patch Available  (was: Open)

> optimize HRegion computeHDFSBlocksDistribution
> --
>
> Key: HBASE-16398
> URL: https://issues.apache.org/jira/browse/HBASE-16398
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0
>
> Attachments: HBASE-16398.patch, HBASE-16398_v2.patch, 
> LocatedBlockStatusComparison.java
>
>
> First i assume there is no reference and link in a region family's directory. 
> Without the patch to computeHDFSBlocksDistribution for a region family, there 
> is 1+2*N rpc call, N is hfile numbers, The first rpc call is to 
> DistributedFileSystem#listStatus to get hfiles, for every hfile there is two 
> rpc call DistributedFileSystem#getFileStatus(path) and then 
> DistributedFileSystem#getFileBlockLocations(status, start, length).
> With the patch to computeHDFSBlocksDistribution for a region family, there is 
> 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and  
> DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter 
> filter).
> So if there is at least one hfile, with the patch, the rpc call will less.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17194) Assign the new region to the idle server after splitting

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17194:
--
Status: Patch Available  (was: Open)

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17194) Assign the new region to the idle server after splitting

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17194:
--
Attachment: HBASE-17194.v1.patch

Apply the feedback from [~stack] and [~jinghe] in the v1.patch.

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution

[binlijin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

binlijin updated HBASE-16398:
-
Affects Version/s: 2.0.0
Fix Version/s: 2.0.0
  Component/s: regionserver

> optimize HRegion computeHDFSBlocksDistribution
> --
>
> Key: HBASE-16398
> URL: https://issues.apache.org/jira/browse/HBASE-16398
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0
>
> Attachments: HBASE-16398.patch, HBASE-16398_v2.patch, 
> LocatedBlockStatusComparison.java
>
>
> First i assume there is no reference and link in a region family's directory. 
> Without the patch to computeHDFSBlocksDistribution for a region family, there 
> is 1+2*N rpc call, N is hfile numbers, The first rpc call is to 
> DistributedFileSystem#listStatus to get hfiles, for every hfile there is two 
> rpc call DistributedFileSystem#getFileStatus(path) and then 
> DistributedFileSystem#getFileBlockLocations(status, start, length).
> With the patch to computeHDFSBlocksDistribution for a region family, there is 
> 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and  
> DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter 
> filter).
> So if there is at least one hfile, with the patch, the rpc call will less.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16398) optimize HRegion computeHDFSBlocksDistribution

[binlijin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

binlijin updated HBASE-16398:
-
Attachment: HBASE-16398_v2.patch

> optimize HRegion computeHDFSBlocksDistribution
> --
>
> Key: HBASE-16398
> URL: https://issues.apache.org/jira/browse/HBASE-16398
> Project: HBase
>  Issue Type: Sub-task
>Reporter: binlijin
>Assignee: binlijin
> Attachments: HBASE-16398.patch, HBASE-16398_v2.patch, 
> LocatedBlockStatusComparison.java
>
>
> First i assume there is no reference and link in a region family's directory. 
> Without the patch to computeHDFSBlocksDistribution for a region family, there 
> is 1+2*N rpc call, N is hfile numbers, The first rpc call is to 
> DistributedFileSystem#listStatus to get hfiles, for every hfile there is two 
> rpc call DistributedFileSystem#getFileStatus(path) and then 
> DistributedFileSystem#getFileBlockLocations(status, start, length).
> With the patch to computeHDFSBlocksDistribution for a region family, there is 
> 2 rpc call, they are DistributedFileSystem#getFileStatus(path) and  
> DistributedFileSystem#listLocatedStatus(final Path p, final PathFilter 
> filter).
> So if there is at least one hfile, with the patch, the rpc call will less.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17215) Separate small/large file delete threads in HFileCleaner to accelerate hfile cleanup speed

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710649#comment-15710649
 ] 

Yu Li commented on HBASE-17215:
---

There will be a property supporting online configuration change as the 
threshold. Will upload a patch soon.

> Separate small/large file delete threads in HFileCleaner to accelerate hfile 
> cleanup speed
> --
>
> Key: HBASE-17215
> URL: https://issues.apache.org/jira/browse/HBASE-17215
> Project: HBase
>  Issue Type: Improvement
>Reporter: Yu Li
>Assignee: Yu Li
>
> When using PCIe-SSD the flush speed will be really quick, and although we 
> have per CF flush, we still have the 
> {{hbase.regionserver.optionalcacheflushinterval}} setting and some other 
> mechanism to avoid data kept in memory for too long to flush small hfiles. In 
> our online environment we found the single thread cleaner kept cleaning 
> earlier flushed small files while large files got no chance, which caused 
> disk full then many other problems.
> Deleting hfiles in parallel with too many threads will also increase the 
> workload of namenode, so here we propose to separate large/small hfile 
> cleaner threads just like we do for compaction, and it turned out to work 
> well in our cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17205) Add a metric for the duration of region in transition

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-17205:
---
Affects Version/s: 1.4.0
   2.0.0

> Add a metric for the duration of region in transition
> -
>
> Key: HBASE-17205
> URL: https://issues.apache.org/jira/browse/HBASE-17205
> Project: HBase
>  Issue Type: Improvement
>  Components: Region Assignment
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17205-branch-1.patch, HBASE-17205-v1.patch, 
> HBASE-17205.patch
>
>
> When work for HBASE-17178, I found there are not a metric for the overall 
> duration of region in transition. When move a region form A to B, the 
> transformation of region state is PENDING_CLOSE => CLOSING => CLOSED => 
> PENDING_OPEN => OPENING => OPENED. When transform old region state to new 
> region state, it update the time stamp to current time. So we can't get the 
> overall transformation's duration of region in transition. Add a rit duration 
> to RegionState for accumulating this metric.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17205) Add a metric for the duration of region in transition

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-17205:
---
Attachment: HBASE-17205-branch-1.patch

> Add a metric for the duration of region in transition
> -
>
> Key: HBASE-17205
> URL: https://issues.apache.org/jira/browse/HBASE-17205
> Project: HBase
>  Issue Type: Improvement
>  Components: Region Assignment
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17205-branch-1.patch, HBASE-17205-v1.patch, 
> HBASE-17205.patch
>
>
> When work for HBASE-17178, I found there are not a metric for the overall 
> duration of region in transition. When move a region form A to B, the 
> transformation of region state is PENDING_CLOSE => CLOSING => CLOSED => 
> PENDING_OPEN => OPENING => OPENED. When transform old region state to new 
> region state, it update the time stamp to current time. So we can't get the 
> overall transformation's duration of region in transition. Add a rit duration 
> to RegionState for accumulating this metric.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17218) [C++] Use Google Style guide and cpplint

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-17218:
--
Attachment: HBASE-17218_v1.patch

[~xiaobingo], [~sudeeps] can you guys please take a look. 

bin/format-code.sh formats the code in-place. 
{{make lint}} runs the cpplint utility. 

> [C++] Use Google Style guide and cpplint
> 
>
> Key: HBASE-17218
> URL: https://issues.apache.org/jira/browse/HBASE-17218
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-17218_v1.patch
>
>
> This was discussed elsewhere in other sub jiras. Let's use the Google's Style 
> guide going forward (instead of LLVM one). 
> There is an Eclipse formatter, a cpplint script and clang-format integration 
> https://github.com/google/styleguide. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17151) New API to create HFile.Reader without instantiating block cache

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-17151:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

I've pushed this. Thanks Vladimir. 

> New API to create HFile.Reader without instantiating block cache 
> -
>
> Key: HBASE-17151
> URL: https://issues.apache.org/jira/browse/HBASE-17151
> Project: HBase
>  Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-17151-v1.patch, HBASE-17151-v2.patch, 
> HBASE-17151-v3.patch, HBASE-17151-v4.patch
>
>
> Currently, to create HFile.Reader instance, the CacheConfig instance is 
> required (which instantiates block cache). We need API for reading HFile w/o 
> block cache being involved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15902) Scan Object

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-15902:
--
Attachment: (was: HBASE-15902.HBASE-14850.v3.patch)

> Scan Object
> ---
>
> Key: HBASE-15902
> URL: https://issues.apache.org/jira/browse/HBASE-15902
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-15902.HBASE-14850.patch, 
> HBASE-15902.HBASE-14850.v2.patch
>
>
> Patch for creating Scan objects. Scan objects thus created can be used by 
> Table implementation to fetch results for a given row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15902) Scan Object

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-15902:
--
Attachment: HBASE-15902.HBASE-14850.v3.patch

Hi,
This patch consists of the foll:-
# Sources for Scan API.
# Changing  raw pointers to smart pointers in Get API.
# Removal of advanced API features from Get and Scan API's in the first cut.

> Scan Object
> ---
>
> Key: HBASE-15902
> URL: https://issues.apache.org/jira/browse/HBASE-15902
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-15902.HBASE-14850.patch, 
> HBASE-15902.HBASE-14850.v2.patch, HBASE-15902.HBASE-14850.v3.patch
>
>
> Patch for creating Scan objects. Scan objects thus created can be used by 
> Table implementation to fetch results for a given row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17205) Add a metric for the duration of region in transition

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710497#comment-15710497
 ] 

Guanghao Zhang commented on HBASE-17205:


bq.  with the new AM we have the actual time of assign and unassign operation 
for each region and the time of the region in failed open or those kind of 
states.
Look forward to the new AM in 2.0. :)


> Add a metric for the duration of region in transition
> -
>
> Key: HBASE-17205
> URL: https://issues.apache.org/jira/browse/HBASE-17205
> Project: HBase
>  Issue Type: Improvement
>  Components: Region Assignment
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17205-v1.patch, HBASE-17205.patch
>
>
> When work for HBASE-17178, I found there are not a metric for the overall 
> duration of region in transition. When move a region form A to B, the 
> transformation of region state is PENDING_CLOSE => CLOSING => CLOSED => 
> PENDING_OPEN => OPENING => OPENED. When transform old region state to new 
> region state, it update the time stamp to current time. So we can't get the 
> overall transformation's duration of region in transition. Add a rit duration 
> to RegionState for accumulating this metric.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17205) Add a metric for the duration of region in transition

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-17205:
---
Attachment: HBASE-17205-v1.patch

Attach v1 patch addressed review comments.

> Add a metric for the duration of region in transition
> -
>
> Key: HBASE-17205
> URL: https://issues.apache.org/jira/browse/HBASE-17205
> Project: HBase
>  Issue Type: Improvement
>  Components: Region Assignment
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17205-v1.patch, HBASE-17205.patch
>
>
> When work for HBASE-17178, I found there are not a metric for the overall 
> duration of region in transition. When move a region form A to B, the 
> transformation of region state is PENDING_CLOSE => CLOSING => CLOSED => 
> PENDING_OPEN => OPENING => OPENED. When transform old region state to new 
> region state, it update the time stamp to current time. So we can't get the 
> overall transformation's duration of region in transition. Add a rit duration 
> to RegionState for accumulating this metric.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710487#comment-15710487
 ] 

Aaron Tokhy commented on HBASE-15314:
-

I created a github pull request with my patch applied as I am unable to attach 
my patch here:
https://github.com/apache/hbase/pull/42

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710483#comment-15710483
 ] 

Aaron Tokhy commented on HBASE-15314:
-

I created a github pull request with my patch applied as I am unable to attach 
my patch here:

https://github.com/apache/hbase/pull/42

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HBASE-15314) Allow more than one backing file in bucketcache

[Aaron Tokhy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron Tokhy updated HBASE-15314:

Comment: was deleted

(was: I created a github pull request with my patch applied as I am unable to 
attach my patch here:

https://github.com/apache/hbase/pull/42)

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710459#comment-15710459
 ] 

Hadoop QA commented on HBASE-17051:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 6s 
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} HBASE-14850 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 0s 
{color} | {color:green} HBASE-14850 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 2s 
{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 2s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 
3s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 9s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
2s {color} | {color:green} the patch passed {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 42s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:date2016-12-01 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841184/HBASE-17051-HBASE-14850.003.patch
 |
| JIRA Issue | HBASE-17051 |
| Optional Tests |  cc  compile  shellcheck  shelldocs  |
| uname | Linux 6fd43402cd2d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | nobuild |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | HBASE-14850 / 78c8bc7 |
| shellcheck | v0.4.5 |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4729/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch, 
> HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch, 
> HBASE-17051-HBASE-14850.003.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17206) FSHLog may roll a new writer successfully with unflushed entries

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710422#comment-15710422
 ] 

stack commented on HBASE-17206:
---

Ugly. Good fix [~Apache9] +1

> FSHLog may roll a new writer successfully with unflushed entries
> 
>
> Key: HBASE-17206
> URL: https://issues.apache.org/jira/browse/HBASE-17206
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17206.patch
>
>
> Found it when debugging the flakey TestFailedAppendAndSync.
> The problem is in waitSafePoint.
> {code}
>   while (true) {
> if (this.safePointAttainedLatch.await(1, TimeUnit.MILLISECONDS)) {
>   break;
> }
> if (syncFuture.isThrowable()) {
>   throw new 
> FailedSyncBeforeLogCloseException(syncFuture.getThrowable());
> }
>   }
>   return syncFuture;
> {code}
> If we attach the safe point quick enough then we will bypass the 
> syncFuture.isThrowable check and will not throw 
> FailedSyncBeforeLogCloseException.
> This may cause incosistency between memstore and wal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17181) Let HBase thrift2 support TThreadedSelectorServer

[Jian Yi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710392#comment-15710392
 ] 

Jian Yi edited comment on HBASE-17181 at 12/1/16 1:03 AM:
--

How to create patch with 1.2.3 if sources are not cloned from git downloaded 
directly.


was (Author: eyj...@gmail.com):
OK, with 1.2.3 being used.

> Let HBase thrift2 support TThreadedSelectorServer
> -
>
> Key: HBASE-17181
> URL: https://issues.apache.org/jira/browse/HBASE-17181
> Project: HBase
>  Issue Type: New Feature
>  Components: Thrift
>Affects Versions: 2.0.0, 1.4.0, 1.2.4
>Reporter: Jian Yi
>Assignee: Jian Yi
>Priority: Minor
>  Labels: features
> Fix For: 2.0.0, 1.4.0, 1.2.5
>
> Attachments: HBASE-17181-V1.patch, HBASE-17181-V2.patch, 
> HBASE-17181-V3.patch, HBASE-17181-V4.patch, HBASE-17181-V5.patch, 
> HBASE-17181-V6.patch, ThriftServer.java
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Add TThreadedSelectorServer for HBase Thrift2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17210) Set timeout on trying rowlock according to client's RPC timeout

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710399#comment-15710399
 ] 

Ted Yu commented on HBASE-17210:


{code}
5343  if (call != null && call.getDeadline() < Long.MAX_VALUE) {
5344timeout = (int)(call.getDeadline() - 
System.currentTimeMillis());
{code}
Should timeout be set to rowLockWaitDuration if (call.getDeadline() - 
System.currentTimeMillis()) <= 0 ?

Please use EnvironmentEdgeManager.currentTime()

> Set timeout on trying rowlock according to client's RPC timeout
> ---
>
> Key: HBASE-17210
> URL: https://issues.apache.org/jira/browse/HBASE-17210
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-17120.v1.patch
>
>
> Now when we want to get a row lock, the timeout is fixed and default is 30s. 
> But the requests from client have different RPC timeout setting. We can use 
> the client's deadline to set timeout on tryLock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17181) Let HBase thrift2 support TThreadedSelectorServer

[Jian Yi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710392#comment-15710392
 ] 

Jian Yi commented on HBASE-17181:
-

OK, with 1.2.3 being used.

> Let HBase thrift2 support TThreadedSelectorServer
> -
>
> Key: HBASE-17181
> URL: https://issues.apache.org/jira/browse/HBASE-17181
> Project: HBase
>  Issue Type: New Feature
>  Components: Thrift
>Affects Versions: 2.0.0, 1.4.0, 1.2.4
>Reporter: Jian Yi
>Assignee: Jian Yi
>Priority: Minor
>  Labels: features
> Fix For: 2.0.0, 1.4.0, 1.2.5
>
> Attachments: HBASE-17181-V1.patch, HBASE-17181-V2.patch, 
> HBASE-17181-V3.patch, HBASE-17181-V4.patch, HBASE-17181-V5.patch, 
> HBASE-17181-V6.patch, ThriftServer.java
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Add TThreadedSelectorServer for HBase Thrift2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17181) Let HBase thrift2 support TThreadedSelectorServer

[Jian Yi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710387#comment-15710387
 ] 

Jian Yi commented on HBASE-17181:
-

I test it on 1.2.3 online service

> Let HBase thrift2 support TThreadedSelectorServer
> -
>
> Key: HBASE-17181
> URL: https://issues.apache.org/jira/browse/HBASE-17181
> Project: HBase
>  Issue Type: New Feature
>  Components: Thrift
>Affects Versions: 2.0.0, 1.4.0, 1.2.4
>Reporter: Jian Yi
>Assignee: Jian Yi
>Priority: Minor
>  Labels: features
> Fix For: 2.0.0, 1.4.0, 1.2.5
>
> Attachments: HBASE-17181-V1.patch, HBASE-17181-V2.patch, 
> HBASE-17181-V3.patch, HBASE-17181-V4.patch, HBASE-17181-V5.patch, 
> HBASE-17181-V6.patch, ThriftServer.java
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Add TThreadedSelectorServer for HBase Thrift2



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710383#comment-15710383
 ] 

Xiaobing Zhou commented on HBASE-17051:
---

v003 fixed some header files reference issue.

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch, 
> HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch, 
> HBASE-17051-HBASE-14850.003.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710383#comment-15710383
 ] 

Xiaobing Zhou edited comment on HBASE-17051 at 12/1/16 12:51 AM:
-

v003 fixed some header files reference issues.


was (Author: xiaobingo):
v003 fixed some header files reference issue.

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch, 
> HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch, 
> HBASE-17051-HBASE-14850.003.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17051) libhbase++: implement RPC client and connection management

[Xiaobing Zhou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaobing Zhou updated HBASE-17051:
--
Attachment: HBASE-17051-HBASE-14850.003.patch

> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch, 
> HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch, 
> HBASE-17051-HBASE-14850.003.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17206) FSHLog may roll a new writer successfully with unflushed entries

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710350#comment-15710350
 ] 

Duo Zhang commented on HBASE-17206:
---

[~stack] Are you OK with this fix?

 [~mantonov] Is it OK to get this into 1.3?

Thanks.

> FSHLog may roll a new writer successfully with unflushed entries
> 
>
> Key: HBASE-17206
> URL: https://issues.apache.org/jira/browse/HBASE-17206
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17206.patch
>
>
> Found it when debugging the flakey TestFailedAppendAndSync.
> The problem is in waitSafePoint.
> {code}
>   while (true) {
> if (this.safePointAttainedLatch.await(1, TimeUnit.MILLISECONDS)) {
>   break;
> }
> if (syncFuture.isThrowable()) {
>   throw new 
> FailedSyncBeforeLogCloseException(syncFuture.getThrowable());
> }
>   }
>   return syncFuture;
> {code}
> If we attach the safe point quick enough then we will bypass the 
> syncFuture.isThrowable check and will not throw 
> FailedSyncBeforeLogCloseException.
> This may cause incosistency between memstore and wal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16261) MultiHFileOutputFormat Enhancement

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710325#comment-15710325
 ] 

Yi Liang commented on HBASE-16261:
--

patch v5 fix findbugs

>  MultiHFileOutputFormat Enhancement 
> 
>
> Key: HBASE-16261
> URL: https://issues.apache.org/jira/browse/HBASE-16261
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase, mapreduce
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16261-V1.patch, HBASE-16261-V2.patch, 
> HBASE-16261-V3.patch, HBASE-16261-V4.patch, HBASE-16261-V5.patch
>
>
> MultiHFileOutputFormat follow HFileOutputFormat2
> (1) HFileOutputFormat2 can read one table's region split keys. and then 
> output multiple hfiles for one family, and each hfile map to one region. We 
> can add partitioner in MultiHFileOutputFormat to make it support this feature.
> (2) HFileOutputFormat2 support Customized Compression algorithm for column 
> family and BloomFilter, also support customized DataBlockEncoding for the 
> output hfiles. We can also make MultiHFileOutputFormat to support these 
> features.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16261) MultiHFileOutputFormat Enhancement

[Yi Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liang updated HBASE-16261:
-
Attachment: HBASE-16261-V5.patch

>  MultiHFileOutputFormat Enhancement 
> 
>
> Key: HBASE-16261
> URL: https://issues.apache.org/jira/browse/HBASE-16261
> Project: HBase
>  Issue Type: Sub-task
>  Components: hbase, mapreduce
>Affects Versions: 2.0.0
>Reporter: Yi Liang
>Assignee: Yi Liang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16261-V1.patch, HBASE-16261-V2.patch, 
> HBASE-16261-V3.patch, HBASE-16261-V4.patch, HBASE-16261-V5.patch
>
>
> MultiHFileOutputFormat follow HFileOutputFormat2
> (1) HFileOutputFormat2 can read one table's region split keys. and then 
> output multiple hfiles for one family, and each hfile map to one region. We 
> can add partitioner in MultiHFileOutputFormat to make it support this feature.
> (2) HFileOutputFormat2 support Customized Compression algorithm for column 
> family and BloomFilter, also support customized DataBlockEncoding for the 
> output hfiles. We can also make MultiHFileOutputFormat to support these 
> features.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15437) Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710314#comment-15710314
 ] 

Jerry He commented on HBASE-15437:
--

Filed HBASE-17221 to see if I can abstract out an interface for RpcServer.Call.

> Response size calculated in RPCServer for warning tooLarge responses does NOT 
> count CellScanner payload
> ---
>
> Key: HBASE-15437
> URL: https://issues.apache.org/jira/browse/HBASE-15437
> Project: HBase
>  Issue Type: Bug
>  Components: IPC/RPC
>Reporter: deepankar
>Assignee: deepankar
> Attachments: HBASE-15437-v1.patch, HBASE-15437-v2.patch, 
> HBASE-15437.patch
>
>
> After HBASE-13158 where we respond back to RPCs with cells in the payload , 
> the protobuf response will just have the count the cells to read from 
> payload, but there are set of features where we log warn in RPCServer 
> whenever the response is tooLarge, but this size now is not considering the 
> sizes of the cells in the PayloadCellScanner. Code form RPCServer
> {code}
>   long responseSize = result.getSerializedSize();
>   // log any RPC responses that are slower than the configured warn
>   // response time or larger than configured warning size
>   boolean tooSlow = (processingTime > warnResponseTime && 
> warnResponseTime > -1);
>   boolean tooLarge = (responseSize > warnResponseSize && warnResponseSize 
> > -1);
>   if (tooSlow || tooLarge) {
> // when tagging, we let TooLarge trump TooSmall to keep output simple
> // note that large responses will often also be slow.
> logResponse(new Object[]{param},
> md.getName(), md.getName() + "(" + param.getClass().getName() + 
> ")",
> (tooLarge ? "TooLarge" : "TooSlow"),
> status.getClient(), startTime, processingTime, qTime,
> responseSize);
>   }
> {code}
> Should this feature be not supported any more or should we add a method to 
> CellScanner or a new interface which returns the serialized size (but this 
> might not include the compression codecs which might be used during response 
> ?) Any other Idea this could be fixed ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16261) MultiHFileOutputFormat Enhancement

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710310#comment-15710310
 ] 

Hadoop QA commented on HBASE-16261:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
24m 46s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 48s 
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1 
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 29m 53s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
11s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 19s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
|  |  Invocation of toString on 
org.apache.hadoop.hbase.HConstants.EMPTY_BYTE_ARRAY in 
org.apache.hadoop.hbase.mapreduce.MultiHFileOutputFormat$MultiHFilePartitioner.getPartition(ImmutableBytesWritable,
 Cell, int)  At MultiHFileOutputFormat.java:in 
org.apache.hadoop.hbase.mapreduce.MultiHFileOutputFormat$MultiHFilePartitioner.getPartition(ImmutableBytesWritable,
 Cell, int)  At MultiHFileOutputFormat.java:[line 440] |
| Timed out junit tests | org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841152/HBASE-16261-V4.patch |
| JIRA Issue | HBASE-16261 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 796b2001fb46 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / ad857d1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4726/artifact/patchprocess/new-findbugs-hbase-server.html
 |
| unit | 

[jira] [Updated] (HBASE-17221) Abstract out an interface for RpcServer.Call

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17221:
-
Status: Patch Available  (was: Open)

Initial patch.

> Abstract out an interface for RpcServer.Call
> 
>
> Key: HBASE-17221
> URL: https://issues.apache.org/jira/browse/HBASE-17221
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-17221.patch
>
>
> RpcServer.Call is a concrete class, but it is marked as:
> {noformat}
> @InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC, 
> HBaseInterfaceAudience.PHOENIX})
> {noformat}
> Let's abstract out an interface out of it for potential consumers that want 
> to pass it around.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17221) Abstract out an interface for RpcServer.Call

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-17221:
-
Attachment: HBASE-17221.patch

> Abstract out an interface for RpcServer.Call
> 
>
> Key: HBASE-17221
> URL: https://issues.apache.org/jira/browse/HBASE-17221
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0
>
> Attachments: HBASE-17221.patch
>
>
> RpcServer.Call is a concrete class, but it is marked as:
> {noformat}
> @InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.COPROC, 
> HBaseInterfaceAudience.PHOENIX})
> {noformat}
> Let's abstract out an interface out of it for potential consumers that want 
> to pass it around.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17051) libhbase++: implement RPC client and connection management

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710283#comment-15710283
 ] 

Hadoop QA commented on HBASE-17051:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 7s 
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} HBASE-14850 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} HBASE-14850 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 1s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 15s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m 
3s {color} | {color:green} the patch passed {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 46s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:date2016-12-01 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841157/HBASE-17051-HBASE-14850.002.patch
 |
| JIRA Issue | HBASE-17051 |
| Optional Tests |  cc  compile  shellcheck  shelldocs  |
| uname | Linux e0cbe21a221a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | nobuild |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | HBASE-14850 / 78c8bc7 |
| shellcheck | v0.4.5 |
| modules | C: . U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4727/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> libhbase++: implement RPC client and connection management
> --
>
> Key: HBASE-17051
> URL: https://issues.apache.org/jira/browse/HBASE-17051
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Xiaobing Zhou
>Assignee: Xiaobing Zhou
> Attachments: HBASE-17051-HBASE-14850.000.patch, 
> HBASE-17051-HBASE-14850.001.patch, HBASE-17051-HBASE-14850.002.patch
>
>
> This proposes building RPC client and connection management layer, which 
> supports the equivalent functions resides in RpcClient.java and 
> RpcConnection.java. Specifically, handler/pipeline concepts will be used for 
> implementation, similar to NettyRpcClient and NettyRpcConnection in java side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17222) [C++] include hbase-default.xml from code rather than under config dir

[Enis Soztutar (JIRA)]

Enis Soztutar created HBASE-17222:
-

 Summary: [C++] include hbase-default.xml from code rather than 
under config dir
 Key: HBASE-17222
 URL: https://issues.apache.org/jira/browse/HBASE-17222
 Project: HBase
  Issue Type: Sub-task
Reporter: Enis Soztutar


In HBASE-16489 we are sourcing hbase-site.xml and hbase-default.xml from given 
directories or HBASE_CONF_DIR env. 

However, the {{hbase-default.xml}} is not there in the conf directories anymore 
in regular deployments. The actual source files is in 
{{hbase-common/src/main/resources}} and it gets packed in a hbase-common.jar. 
We should find a way to copy the file for the C++ build, and source the data 
from the C++ bundle, rather than expecting it coming from conf dirs. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17057) Minor compactions should also drop page cache behind reads

[Ashu Pachauri (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710244#comment-15710244
 ] 

Ashu Pachauri commented on HBASE-17057:
---

Talked to [~eclark] offline, it turns out that throttling compactions has 
nothing to do with dropping page cache, it was used as a hint to figure out the 
total size of the files involved in a compaction request. Since, in the old 
world, compactions piggybacked on the store file scanners that were already 
open, we considered it more efficient to not drop pages during compactions that 
were small enough rather than potentially dropping pages for storefiles that 
were probably already being read. However, since we use private readers for 
compactions by default, we should drop pages for minor compactions by default.
I'll add a patch that introduces a config to drop page cache for minor and 
major compactions. This config will be set to true by default, but someone who 
is not using private readers can choose to turn it off (though I doubt turning 
it off will be any positive impact especially in large clusters.)
For master branch, this jira will address correctly passing the drop cache 
hint; I'll open a separate issue (or find one if it already exists) that makes 
sure we honor the hint in the compaction path.

> Minor compactions should also drop page cache behind reads
> --
>
> Key: HBASE-17057
> URL: https://issues.apache.org/jira/browse/HBASE-17057
> Project: HBase
>  Issue Type: Improvement
>  Components: Compaction
>Reporter: Ashu Pachauri
>Assignee: Ashu Pachauri
>
> Long compactions currently drop cache behind reads/writes so that they don't 
> pollute the page cache but short compactions don't do that. The bulk of the 
> data is actually read during minor compactions instead of major compactions,  
> and thrashes the page cache since it's mostly not needed. 
> We should drop page cache behind minor compactions too. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)