[jira] [Commented] (HBASE-16225) Refactor ScanQueryMatcher

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395062#comment-15395062
 ] 

Duo Zhang commented on HBASE-16225:
---

The timed out UTs can pass locally.

> Refactor ScanQueryMatcher
> -
>
> Key: HBASE-16225
> URL: https://issues.apache.org/jira/browse/HBASE-16225
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-16225-v1.patch, HBASE-16225-v2.patch, 
> HBASE-16225-v3.patch, HBASE-16225.patch
>
>
> As said in HBASE-16223, the code of {{ScanQueryMatcher}} is too complicated. 
> I suggest that we can abstract an interface and implement several sub classes 
> which separate different logic into different implementations. For example, 
> the requirements of compaction and user scan are different, now we also need 
> to consider the logic of user scan even if we only want to add a logic for 
> compaction. And at least, the raw scan does not need a query matcher... we 
> can implement a dummy query matcher for it.
> Suggestions are welcomed. Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16287) BlockCache size should not exceed acceptableSize too many

[Yu Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395059#comment-15395059
 ] 

Yu Li commented on HBASE-16287:
---

This issue happens in our online system, and I believe we should add some 
protection mechanism for such blockcache overload. Please upload a patch so 
others could better understand the issue by checking the code, thanks. [~haoran]

> BlockCache size should not exceed acceptableSize too many
> -
>
> Key: HBASE-16287
> URL: https://issues.apache.org/jira/browse/HBASE-16287
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache
>Reporter: Yu Sun
>
> Our regionserver has a configuation as bellow：
>   -Xmn4g -Xms32g -Xmx32g -XX:SurvriorRatio=2 -XX:+UseConcMarkSweepGC 
> also we only use blockcache,and set hfile.block.cache.size = 0.3 in 
> hbase_site.xml,so under this configuration, the lru block cache size will 
> be(32g-1g)*0.3=9.3g. but in some scenarios，some of the rs will occur 
> continuous FullGC  for hours and most importantly, after FullGC most of the 
> object in old will not be GCed. so we dump the heap and analyse with MAT and 
> we observed a obvious memory leak in LruBlockCache, which occpy about 16g 
> memory, then we set set class LruBlockCache log level to TRACE and observed 
> this in log:
> {quote}
> 2016-07-22 12:17:58,158 INFO  [LruBlockCacheStatsExecutor] 
> hfile.LruBlockCache: totalSize=15.29 GB, freeSize=-5.99 GB, max=9.30 GB, 
> blockCount=628182, accesses=101799469125, hits=93517800259, hitRatio=91.86%, 
> , cachingAccesses=99462650031, cachingHits=93468334621, 
> cachingHitsRatio=93.97%, evictions=238199, evicted=4776350518, 
> evictedPerRun=20051.93359375{quote}
> we can see blockcache size has exceeded acceptableSize too many, which will 
> cause the FullGC more seriously. 
> Afterfter some investigations, I found in this function:
> {code:borderStyle=solid}
>   public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean 
> inMemory,
>   final boolean cacheDataInL1) {
> {code}
> No matter the blockcache size has been used, just put the block into it. but 
> if the evict thread is not fast enough, blockcache size will increament 
> significantly.
> So here I think we should have a check, for example, if the blockcache size > 
> 1.2 * acceptableSize(), just return and dont put into it until the blockcache 
> size if under watrmark. if this is reasonable, I can make a small patch for 
> this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16225) Refactor ScanQueryMatcher

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395052#comment-15395052
 ] 

Hadoop QA commented on HBASE-16225:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 18 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
40m 54s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 123m 10s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
34s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 195m 13s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| Timed out junit tests | 
org.apache.hadoop.hbase.security.access.TestCellACLWithMultipleVersions |
|   | org.apache.hadoop.hbase.security.access.TestScanEarlyTermination |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820347/HBASE-16225-v3.patch |
| JIRA Issue | HBASE-16225 |
| 

[jira] [Commented] (HBASE-16284) Unauthorized client can shutdown the cluster

[DeokWoo Han (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395042#comment-15395042
 ] 

DeokWoo Han commented on HBASE-16284:
-

The failed tests seem to be unrelated to the patch. They passed locally.

> Unauthorized client can shutdown the cluster
> 
>
> Key: HBASE-16284
> URL: https://issues.apache.org/jira/browse/HBASE-16284
> Project: HBase
>  Issue Type: Bug
>Reporter: DeokWoo Han
> Attachments: HBASE-16284.patch
>
>
> An unauthorized client can shutdown the cluster as {{AccessDeniedException}} 
> is ignored during {{Admin.stopMaster}} and {{Admin.shutdown}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16283) Batch Append/Increment will always fail if set ReturnResults to false

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395039#comment-15395039
 ] 

Anoop Sam John commented on HBASE-16283:


Ya there are no tests present in code with ur mentioned case. Else that would 
have failed before. Better add that case(s) as new test case in an existing 
test class.

> Batch Append/Increment will always fail if set ReturnResults to false
> -
>
> Key: HBASE-16283
> URL: https://issues.apache.org/jira/browse/HBASE-16283
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 2.0.0, 1.1.5, 1.2.2
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: FailedCase.java, HBASE-16283.patch, HBASE-16283v2.patch
>
>
> If set Append/Increment's ReturnResult attribute to false, and batch the 
> appends/increments to server. The batch operation will always return false.
> The reason is that, since return result is set to false, append/increment 
> will return null instead of Result object. But in 
> ResponseConverter#getResults, there is some check code 
> {code}
> if (requestRegionActionCount != responseRegionActionResultCount) {
>   throw new IllegalStateException("Request mutation count=" + 
> requestRegionActionCount +
>   " does not match response mutation result count=" + 
> responseRegionActionResultCount);
> }
> {code}
> That means if the result count is not meet with request mutation count, it 
> will fail the request.
> The solution is simple, instead of returning a null result, returning a empty 
> result if ReturnResult set to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16287) BlockCache size should not exceed acceptableSize too many

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395004#comment-15395004
 ] 

Heng Chen commented on HBASE-16287:
---

Currently, we check the block cache size after put block into cache,  i think 
we should do it firstly.   wdyt? 

> BlockCache size should not exceed acceptableSize too many
> -
>
> Key: HBASE-16287
> URL: https://issues.apache.org/jira/browse/HBASE-16287
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache
>Reporter: Yu Sun
>
> Our regionserver has a configuation as bellow：
>   -Xmn4g -Xms32g -Xmx32g -XX:SurvriorRatio=2 -XX:+UseConcMarkSweepGC 
> also we only use blockcache,and set hfile.block.cache.size = 0.3 in 
> hbase_site.xml,so under this configuration, the lru block cache size will 
> be(32g-1g)*0.3=9.3g. but in some scenarios，some of the rs will occur 
> continuous FullGC  for hours and most importantly, after FullGC most of the 
> object in old will not be GCed. so we dump the heap and analyse with MAT and 
> we observed a obvious memory leak in LruBlockCache, which occpy about 16g 
> memory, then we set set class LruBlockCache log level to TRACE and observed 
> this in log:
> {quote}
> 2016-07-22 12:17:58,158 INFO  [LruBlockCacheStatsExecutor] 
> hfile.LruBlockCache: totalSize=15.29 GB, freeSize=-5.99 GB, max=9.30 GB, 
> blockCount=628182, accesses=101799469125, hits=93517800259, hitRatio=91.86%, 
> , cachingAccesses=99462650031, cachingHits=93468334621, 
> cachingHitsRatio=93.97%, evictions=238199, evicted=4776350518, 
> evictedPerRun=20051.93359375{quote}
> we can see blockcache size has exceeded acceptableSize too many, which will 
> cause the FullGC more seriously. 
> Afterfter some investigations, I found in this function:
> {code:borderStyle=solid}
>   public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean 
> inMemory,
>   final boolean cacheDataInL1) {
> {code}
> No matter the blockcache size has been used, just put the block into it. but 
> if the evict thread is not fast enough, blockcache size will increament 
> significantly.
> So here I think we should have a check, for example, if the blockcache size > 
> 1.2 * acceptableSize(), just return and dont put into it until the blockcache 
> size if under watrmark. if this is reasonable, I can make a small patch for 
> this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16234) Expect and handle nulls when assigning replicas

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394984#comment-15394984
 ] 

Heng Chen commented on HBASE-16234:
---

In patch v2,  i think it will be better to deal with NPE as patch v1 did,  
please do not use Exception instead of IOException

> Expect and handle nulls when assigning replicas
> ---
>
> Key: HBASE-16234
> URL: https://issues.apache.org/jira/browse/HBASE-16234
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Yi Liang
> Attachments: HBASE-16234-V1.patch, HBASE-16234-V1.patch, 
> HBASE-16234-V2.patch
>
>
> Observed this on a cluster:
> {code}
> FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting 
> shutdown. 
> java.lang.NullPointerException 
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.replicaRegionsNotRecordedInMeta(AssignmentManager.java:2799)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:2778)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:638)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:485)
>  
> at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:723)
>  
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) 
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1481) 
> at java.lang.Thread.run(Thread.java:745) 
> {code}
> It looks like {{FSTableDescriptors#get(…)}} can be expected to return null in 
> some cases, but {{AssignmentManager.replicaRegionsNotRecordedInMeta(…)}} does 
> not currently have any handling for such a possibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16225) Refactor ScanQueryMatcher

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394932#comment-15394932
 ] 

Duo Zhang commented on HBASE-16225:
---

[~apurtell] [~stack] Let's finish this issue if you guys do not have much 
concerns? This is only a refactoring, not a new feature. As said above, lots of 
new stuffs need this issue to be resolved.

Thanks.

> Refactor ScanQueryMatcher
> -
>
> Key: HBASE-16225
> URL: https://issues.apache.org/jira/browse/HBASE-16225
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-16225-v1.patch, HBASE-16225-v2.patch, 
> HBASE-16225-v3.patch, HBASE-16225.patch
>
>
> As said in HBASE-16223, the code of {{ScanQueryMatcher}} is too complicated. 
> I suggest that we can abstract an interface and implement several sub classes 
> which separate different logic into different implementations. For example, 
> the requirements of compaction and user scan are different, now we also need 
> to consider the logic of user scan even if we only want to add a logic for 
> compaction. And at least, the raw scan does not need a query matcher... we 
> can implement a dummy query matcher for it.
> Suggestions are welcomed. Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16225) Refactor ScanQueryMatcher

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-16225:
--
Attachment: HBASE-16225-v3.patch

Fix javadoc warnings.

> Refactor ScanQueryMatcher
> -
>
> Key: HBASE-16225
> URL: https://issues.apache.org/jira/browse/HBASE-16225
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Attachments: HBASE-16225-v1.patch, HBASE-16225-v2.patch, 
> HBASE-16225-v3.patch, HBASE-16225.patch
>
>
> As said in HBASE-16223, the code of {{ScanQueryMatcher}} is too complicated. 
> I suggest that we can abstract an interface and implement several sub classes 
> which separate different logic into different implementations. For example, 
> the requirements of compaction and user scan are different, now we also need 
> to consider the logic of user scan even if we only want to add a logic for 
> compaction. And at least, the raw scan does not need a query matcher... we 
> can implement a dummy query matcher for it.
> Suggestions are welcomed. Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16234) Expect and handle nulls when assigning replicas

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394780#comment-15394780
 ] 

Yi Liang commented on HBASE-16234:
--

Hi [~chenheng], 
thanks for reviewing, i just read through the AssignmentManager.java file. 
I found there are same problem at other places in this file, some place handle 
the exception, some place not (nullpointer not belong to IOException). So in 
V2, I fix all the similar problems and make all the fix consistency. Could you 
take a look, thanks.   

> Expect and handle nulls when assigning replicas
> ---
>
> Key: HBASE-16234
> URL: https://issues.apache.org/jira/browse/HBASE-16234
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Yi Liang
> Attachments: HBASE-16234-V1.patch, HBASE-16234-V1.patch, 
> HBASE-16234-V2.patch
>
>
> Observed this on a cluster:
> {code}
> FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting 
> shutdown. 
> java.lang.NullPointerException 
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.replicaRegionsNotRecordedInMeta(AssignmentManager.java:2799)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:2778)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:638)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:485)
>  
> at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:723)
>  
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) 
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1481) 
> at java.lang.Thread.run(Thread.java:745) 
> {code}
> It looks like {{FSTableDescriptors#get(…)}} can be expected to return null in 
> some cases, but {{AssignmentManager.replicaRegionsNotRecordedInMeta(…)}} does 
> not currently have any handling for such a possibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16234) Expect and handle nulls when assigning replicas

[Yi Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yi Liang updated HBASE-16234:
-
Attachment: HBASE-16234-V2.patch

> Expect and handle nulls when assigning replicas
> ---
>
> Key: HBASE-16234
> URL: https://issues.apache.org/jira/browse/HBASE-16234
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Yi Liang
> Attachments: HBASE-16234-V1.patch, HBASE-16234-V1.patch, 
> HBASE-16234-V2.patch
>
>
> Observed this on a cluster:
> {code}
> FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting 
> shutdown. 
> java.lang.NullPointerException 
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.replicaRegionsNotRecordedInMeta(AssignmentManager.java:2799)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:2778)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:638)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:485)
>  
> at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:723)
>  
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) 
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1481) 
> at java.lang.Thread.run(Thread.java:745) 
> {code}
> It looks like {{FSTableDescriptors#get(…)}} can be expected to return null in 
> some cases, but {{AssignmentManager.replicaRegionsNotRecordedInMeta(…)}} does 
> not currently have any handling for such a possibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394765#comment-15394765
 ] 

Ted Yu commented on HBASE-16275:


lgtm

Do the failed tests pass locally ?

> Change ServerManager#onlineServers from ConcurrentHashMap to 
> ConcurrentSkipListMap
> --
>
> Key: HBASE-16275
> URL: https://issues.apache.org/jira/browse/HBASE-16275
> Project: HBase
>  Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, 
> TestServerManagerPerf.java
>
>
> In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In 
> findServerWithSameHostnamePortWithLock(), it has to do a loop to find if 
> there is a ServerName with same host:port pair. If replaced with 
> ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be 
> replaced with a O(logN) implementation. 
> I run some performance comparison(test the function only), it seems that 
> there is no difference if there are 1000 servers. With more servers, 
> ConcurrentSkipListMap implementation is going to win big.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes

[Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph updated HBASE-16096:
---
Status: Patch Available  (was: Open)

> Replication keeps accumulating znodes
> -
>
> Key: HBASE-16096
> URL: https://issues.apache.org/jira/browse/HBASE-16096
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.2.0, 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Joseph
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16096-branch-1.patch, HBASE-16096.patch
>
>
> If there is an error while creating the replication source on adding the 
> peer, the source if not added to the in memory list of sources but the 
> replication peer is. 
> However, in such a scenario, when you remove the peer, it is deleted from 
> zookeeper successfully but for removing the in memory list of peers, we wait 
> for the corresponding sources to get deleted (which as we said don't exist 
> because of error creating the source). 
> The problem here is the ordering of operations for adding/removing source and 
> peer. 
> Modifying the code to always remove queues from the underlying storage, even 
> if there exists no sources also requires a small refactoring of 
> TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty 
> queue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes

[Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph updated HBASE-16096:
---
Status: Open  (was: Patch Available)

> Replication keeps accumulating znodes
> -
>
> Key: HBASE-16096
> URL: https://issues.apache.org/jira/browse/HBASE-16096
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.2.0, 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Joseph
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16096-branch-1.patch, HBASE-16096.patch
>
>
> If there is an error while creating the replication source on adding the 
> peer, the source if not added to the in memory list of sources but the 
> replication peer is. 
> However, in such a scenario, when you remove the peer, it is deleted from 
> zookeeper successfully but for removing the in memory list of peers, we wait 
> for the corresponding sources to get deleted (which as we said don't exist 
> because of error creating the source). 
> The problem here is the ordering of operations for adding/removing source and 
> peer. 
> Modifying the code to always remove queues from the underlying storage, even 
> if there exists no sources also requires a small refactoring of 
> TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty 
> queue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394727#comment-15394727
 ] 

Hadoop QA commented on HBASE-15638:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 73 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 43s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 14s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m 
54s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 20s 
{color} | {color:red} hbase-protocol in master has 93 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 40s 
{color} | {color:red} hbase-rsgroup in master has 6 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 49s 
{color} | {color:red} hbase-rest in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 20s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 0s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 26s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 17s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 17s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 54s 
{color} | {color:red} hbase-common-jdk1.7.0_80 with JDK v1.7.0_80 generated 1 
new + 25 unchanged - 1 fixed = 26 total (was 26) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 17s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
16s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
41m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m 
16s {color} | {color:green} the patch passed {color} |
| 

[jira] [Commented] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394723#comment-15394723
 ] 

Hadoop QA commented on HBASE-16288:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 6s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m 50s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 140m 53s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820270/hbase-16288_v3.patch |
| JIRA Issue | HBASE-16288 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / bcf409e |
| Default Java | 1.7.0_80 |
| Multi-JDK versions |  /home/jenkins/tools/java/jdk1.8.0:1.8.0 
/home/jenkins/jenkins-slave/tools/hudson.model.JDK/JDK_1.7_latest_:1.7.0_80 |
| findbugs | v3.0.0 |
|  Test Results | 

[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394646#comment-15394646
 ] 

Hadoop QA commented on HBASE-16209:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
32m 5s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 122m 32s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
33s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 182m 17s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820234/HBASE-16209-v2.patch |
| JIRA Issue | HBASE-16209 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / bcf409e |
| Default Java | 1.7.0_80 |
| Multi-JDK versions |  /usr/local/jenkins/java/jdk1.8.0:1.8.0 
/home/jenkins/jenkins-slave/tools/hudson.model.JDK/JDK_1.7_latest_:1.7.0_80 |

[jira] [Commented] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394492#comment-15394492
 ] 

Enis Soztutar commented on HBASE-16288:
---

Nice catch. I've stolen some of the test code from 
TestHFileInlineToRootChunkConversion. Should have read the bottom part first. 

> HFile intermediate block level indexes might recurse forever creating multi 
> TB files
> 
>
> Key: HBASE-16288
> URL: https://issues.apache.org/jira/browse/HBASE-16288
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: hbase-16288_v1.patch, hbase-16288_v2.patch, 
> hbase-16288_v3.patch
>
>
> Mighty [~elserj] was debugging an opentsdb cluster where some region 
> directory ended up having 5TB+ files under /.tmp/ 
> Further debugging and analysis, we were able to reproduce the problem locally 
> where we never we recursing in this code path for writing intermediate level 
> indices: 
> {code:title=HFileBlockIndex.java}
> if (curInlineChunk != null) {
> while (rootChunk.getRootSize() > maxChunkSize) {
>   rootChunk = writeIntermediateLevel(out, rootChunk);
>   numLevels += 1;
> }
>   }
> {code}
> The problem happens if we end up with a very large rowKey (larger than 
> "hfile.index.block.max.size" being the first key in the block, then moving 
> all the way to the root-level index building. We will keep writing and 
> building the next level of intermediate level indices with a single 
> very-large key. This can happen in flush / compaction / region recovery 
> causing cluster inoperability due to ever-growing files. 
> Seems the issue was also reported earlier, with a temporary workaround: 
> https://github.com/OpenTSDB/opentsdb/issues/490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394473#comment-15394473
 ] 

Matteo Bertozzi commented on HBASE-16288:
-

the last part of test seems to not execute the seekTo(). no one is adding to 
"keys".
other than that looks ok to me.

> HFile intermediate block level indexes might recurse forever creating multi 
> TB files
> 
>
> Key: HBASE-16288
> URL: https://issues.apache.org/jira/browse/HBASE-16288
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: hbase-16288_v1.patch, hbase-16288_v2.patch
>
>
> Mighty [~elserj] was debugging an opentsdb cluster where some region 
> directory ended up having 5TB+ files under /.tmp/ 
> Further debugging and analysis, we were able to reproduce the problem locally 
> where we never we recursing in this code path for writing intermediate level 
> indices: 
> {code:title=HFileBlockIndex.java}
> if (curInlineChunk != null) {
> while (rootChunk.getRootSize() > maxChunkSize) {
>   rootChunk = writeIntermediateLevel(out, rootChunk);
>   numLevels += 1;
> }
>   }
> {code}
> The problem happens if we end up with a very large rowKey (larger than 
> "hfile.index.block.max.size" being the first key in the block, then moving 
> all the way to the root-level index building. We will keep writing and 
> building the next level of intermediate level indices with a single 
> very-large key. This can happen in flush / compaction / region recovery 
> causing cluster inoperability due to ever-growing files. 
> Seems the issue was also reported earlier, with a temporary workaround: 
> https://github.com/OpenTSDB/opentsdb/issues/490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394466#comment-15394466
 ] 

Hadoop QA commented on HBASE-15638:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 73 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 59s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 37s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 30m 
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
49s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 30s 
{color} | {color:red} hbase-protocol in master has 93 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 45s 
{color} | {color:red} hbase-rsgroup in master has 6 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 56s 
{color} | {color:red} hbase-rest in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 7s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 21s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 20s 
{color} | {color:red} hbase-examples in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 36s 
{color} | {color:red} hbase-examples in the patch failed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s {color} 
| {color:red} hbase-examples in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 21s 
{color} | {color:red} hbase-examples in the patch failed with JDK v1.7.0_80. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 20m 33s 
{color} | {color:red} hbase-common-jdk1.7.0_80 with JDK v1.7.0_80 generated 1 
new + 25 unchanged - 1 fixed = 26 total (was 26) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s {color} 
| {color:red} hbase-examples in the patch failed with JDK v1.7.0_80. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 32m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m 
22s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 3s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 53s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 55s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 12m 5s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 15m 16s 
{color} | {color:red} Patch causes 83 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} 

[jira] [Commented] (HBASE-16096) Replication keeps accumulating znodes

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394439#comment-15394439
 ] 

Hadoop QA commented on HBASE-16096:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
14s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
59s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
21s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
57s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s 
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
18m 22s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 21s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 5s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestFailedAppendAndSync |
|   | hadoop.hbase.replication.TestReplicationSmallTests |
|   | hadoop.hbase.replication.multiwal.TestReplicationEndpointWithMultipleWAL |
|   | 
hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleWAL
 |
| Timed out junit tests | org.apache.hadoop.hbase.regionserver.TestClusterId |
|   | org.apache.hadoop.hbase.replication.TestMasterReplication |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820204/HBASE-16096-branch-1.patch
 |
| JIRA Issue | HBASE-16096 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 

[jira] [Updated] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-16275:
-
Status: Patch Available  (was: In Progress)

> Change ServerManager#onlineServers from ConcurrentHashMap to 
> ConcurrentSkipListMap
> --
>
> Key: HBASE-16275
> URL: https://issues.apache.org/jira/browse/HBASE-16275
> Project: HBase
>  Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, 
> TestServerManagerPerf.java
>
>
> In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In 
> findServerWithSameHostnamePortWithLock(), it has to do a loop to find if 
> there is a ServerName with same host:port pair. If replaced with 
> ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be 
> replaced with a O(logN) implementation. 
> I run some performance comparison(test the function only), it seems that 
> there is no difference if there are 1000 servers. With more servers, 
> ConcurrentSkipListMap implementation is going to win big.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394424#comment-15394424
 ] 

huaxiang sun edited comment on HBASE-16275 at 7/26/16 7:46 PM:
---

Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100   200 5001000  2000  
30005000 
WOPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WithPatch 8(102)11(202)16(341) 23(548)34(823)   
 47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.


was (Author: huaxiang):
Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100   200 5001000  2000  
30005000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.

> Change ServerManager#onlineServers from ConcurrentHashMap to 
> ConcurrentSkipListMap
> --
>
> Key: HBASE-16275
> URL: https://issues.apache.org/jira/browse/HBASE-16275
> Project: HBase
>  Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, 
> TestServerManagerPerf.java
>
>
> In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In 
> findServerWithSameHostnamePortWithLock(), it has to do a loop to find if 
> there is a ServerName with same host:port pair. If replaced with 
> ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be 
> replaced with a O(logN) implementation. 
> I run some performance comparison(test the function only), it seems that 
> there is no difference if there are 1000 servers. With more servers, 
> ConcurrentSkipListMap implementation is going to win big.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394424#comment-15394424
 ] 

huaxiang sun edited comment on HBASE-16275 at 7/26/16 7:45 PM:
---

Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100   200 5001000  2000  
30005000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.


was (Author: huaxiang):
Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100   200 5001000  2000 
3000  5000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.

> Change ServerManager#onlineServers from ConcurrentHashMap to 
> ConcurrentSkipListMap
> --
>
> Key: HBASE-16275
> URL: https://issues.apache.org/jira/browse/HBASE-16275
> Project: HBase
>  Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, 
> TestServerManagerPerf.java
>
>
> In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In 
> findServerWithSameHostnamePortWithLock(), it has to do a loop to find if 
> there is a ServerName with same host:port pair. If replaced with 
> ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be 
> replaced with a O(logN) implementation. 
> I run some performance comparison(test the function only), it seems that 
> there is no difference if there are 1000 servers. With more servers, 
> ConcurrentSkipListMap implementation is going to win big.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394424#comment-15394424
 ] 

huaxiang sun edited comment on HBASE-16275 at 7/26/16 7:45 PM:
---

Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100   200 5001000  2000 
3000  5000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.


was (Author: huaxiang):
Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100   200 5001000   2000 3000   
   5000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.

> Change ServerManager#onlineServers from ConcurrentHashMap to 
> ConcurrentSkipListMap
> --
>
> Key: HBASE-16275
> URL: https://issues.apache.org/jira/browse/HBASE-16275
> Project: HBase
>  Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, 
> TestServerManagerPerf.java
>
>
> In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In 
> findServerWithSameHostnamePortWithLock(), it has to do a loop to find if 
> there is a ServerName with same host:port pair. If replaced with 
> ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be 
> replaced with a O(logN) implementation. 
> I run some performance comparison(test the function only), it seems that 
> there is no difference if there are 1000 servers. With more servers, 
> ConcurrentSkipListMap implementation is going to win big.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394424#comment-15394424
 ] 

huaxiang sun edited comment on HBASE-16275 at 7/26/16 7:44 PM:
---

Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100   200 5001000   2000 3000   
   5000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.


was (Author: huaxiang):
Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100 200500 1000   2000  
 3000  5000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.

> Change ServerManager#onlineServers from ConcurrentHashMap to 
> ConcurrentSkipListMap
> --
>
> Key: HBASE-16275
> URL: https://issues.apache.org/jira/browse/HBASE-16275
> Project: HBase
>  Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, 
> TestServerManagerPerf.java
>
>
> In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In 
> findServerWithSameHostnamePortWithLock(), it has to do a loop to find if 
> there is a ServerName with same host:port pair. If replaced with 
> ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be 
> replaced with a O(logN) implementation. 
> I run some performance comparison(test the function only), it seems that 
> there is no difference if there are 1000 servers. With more servers, 
> ConcurrentSkipListMap implementation is going to win big.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394424#comment-15394424
 ] 

huaxiang sun commented on HBASE-16275:
--

Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100 200500 1000   2000  
 3000  5000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.

> Change ServerManager#onlineServers from ConcurrentHashMap to 
> ConcurrentSkipListMap
> --
>
> Key: HBASE-16275
> URL: https://issues.apache.org/jira/browse/HBASE-16275
> Project: HBase
>  Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, 
> TestServerManagerPerf.java
>
>
> In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In 
> findServerWithSameHostnamePortWithLock(), it has to do a loop to find if 
> there is a ServerName with same host:port pair. If replaced with 
> ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be 
> replaced with a O(logN) implementation. 
> I run some performance comparison(test the function only), it seems that 
> there is no difference if there are 1000 servers. With more servers, 
> ConcurrentSkipListMap implementation is going to win big.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16142) Trigger JFR session when under duress -- e.g. backed-up request queue count -- and dump the recording to log dir

[Konstantin Ryakhovskiy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Ryakhovskiy updated HBASE-16142:
---
Status: Patch Available  (was: Open)

> Trigger JFR session when under duress -- e.g. backed-up request queue count 
> -- and dump the recording to log dir
> 
>
> Key: HBASE-16142
> URL: https://issues.apache.org/jira/browse/HBASE-16142
> Project: HBase
>  Issue Type: Task
>  Components: Operability
>Reporter: stack
>Assignee: Konstantin Ryakhovskiy
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-16142.master.001.patch, 
> HBASE-16142.master.002.patch, HBASE-16142.master.003.patch, 
> HBASE-16142.master.004.patch
>
>
> Chatting today w/ a mighty hbase operator on how to figure what is happening 
> during transitory latency spike or any other transitory 'weirdness' in a 
> server, the idea came up that a java flight recording during a spike would 
> include a pretty good picture of what is going on during the time of duress 
> (more ideal would be a trace of the explicit slow queries showing call stack 
> with timings dumped to a sink for later review; i.e. trigger an htrace when a 
> query is slow...).
> Taking a look, programmatically triggering a JFR recording seems doable, if 
> awkward (MBean invocations). There is even a means of specifying 'triggers' 
> based off any published mbean emission -- e.g. a query queue count threshold 
> -- which looks nice. See 
> https://community.oracle.com/thread/3676275?start=0=0 and 
> https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/run.htm#JFRUH184
> This feature could start out as a blog post describing how to do it for one 
> server. A plugin on Canary that looks at mbean values and if over a 
> configured threshold, triggers a recording remotely could be next. Finally 
> could integrate a couple of triggers that fire when issue via the trigger 
> mechanism.
> Marking as beginner feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394424#comment-15394424
 ] 

huaxiang sun edited comment on HBASE-16275 at 7/26/16 7:44 PM:
---

Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100 200500 1000   2000  
 3000  5000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.


was (Author: huaxiang):
Some numbers to share, tested with number of unique servers. The test program 
is attached.

{code}
Server count100 200500 1000   2000  
 3000  5000 
WithPatch8(52) 13(228)22(498) 47(1176) 150(3518) 
309(7970)  855(20756)
WoPatch 8(102)11(202)16(341) 23(548)34(823)
47(1114)65(2192) 
{code}

the format is like N(M), N is the max time spent by one thread in Milliseconds, 
M is the total time spent by all threads. It gives rough idea.

> Change ServerManager#onlineServers from ConcurrentHashMap to 
> ConcurrentSkipListMap
> --
>
> Key: HBASE-16275
> URL: https://issues.apache.org/jira/browse/HBASE-16275
> Project: HBase
>  Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, 
> TestServerManagerPerf.java
>
>
> In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In 
> findServerWithSameHostnamePortWithLock(), it has to do a loop to find if 
> there is a ServerName with same host:port pair. If replaced with 
> ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be 
> replaced with a O(logN) implementation. 
> I run some performance comparison(test the function only), it seems that 
> there is no difference if there are 1000 servers. With more servers, 
> ConcurrentSkipListMap implementation is going to win big.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16142) Trigger JFR session when under duress -- e.g. backed-up request queue count -- and dump the recording to log dir

[Konstantin Ryakhovskiy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Ryakhovskiy updated HBASE-16142:
---
Attachment: HBASE-16142.master.004.patch

- add more comments with links to JMC and JFR
- modify log-directory initialization, use environment variable HBASE_LOG_DIR
- add more info to the printUsage() method


> Trigger JFR session when under duress -- e.g. backed-up request queue count 
> -- and dump the recording to log dir
> 
>
> Key: HBASE-16142
> URL: https://issues.apache.org/jira/browse/HBASE-16142
> Project: HBase
>  Issue Type: Task
>  Components: Operability
>Reporter: stack
>Assignee: Konstantin Ryakhovskiy
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-16142.master.001.patch, 
> HBASE-16142.master.002.patch, HBASE-16142.master.003.patch, 
> HBASE-16142.master.004.patch
>
>
> Chatting today w/ a mighty hbase operator on how to figure what is happening 
> during transitory latency spike or any other transitory 'weirdness' in a 
> server, the idea came up that a java flight recording during a spike would 
> include a pretty good picture of what is going on during the time of duress 
> (more ideal would be a trace of the explicit slow queries showing call stack 
> with timings dumped to a sink for later review; i.e. trigger an htrace when a 
> query is slow...).
> Taking a look, programmatically triggering a JFR recording seems doable, if 
> awkward (MBean invocations). There is even a means of specifying 'triggers' 
> based off any published mbean emission -- e.g. a query queue count threshold 
> -- which looks nice. See 
> https://community.oracle.com/thread/3676275?start=0=0 and 
> https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/run.htm#JFRUH184
> This feature could start out as a blog post describing how to do it for one 
> server. A plugin on Canary that looks at mbean values and if over a 
> configured threshold, triggers a recording remotely could be next. Finally 
> could integrate a couple of triggers that fire when issue via the trigger 
> mechanism.
> Marking as beginner feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16142) Trigger JFR session when under duress -- e.g. backed-up request queue count -- and dump the recording to log dir

[Konstantin Ryakhovskiy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Ryakhovskiy updated HBASE-16142:
---
Status: Open  (was: Patch Available)

> Trigger JFR session when under duress -- e.g. backed-up request queue count 
> -- and dump the recording to log dir
> 
>
> Key: HBASE-16142
> URL: https://issues.apache.org/jira/browse/HBASE-16142
> Project: HBase
>  Issue Type: Task
>  Components: Operability
>Reporter: stack
>Assignee: Konstantin Ryakhovskiy
>Priority: Minor
>  Labels: beginner
> Attachments: HBASE-16142.master.001.patch, 
> HBASE-16142.master.002.patch, HBASE-16142.master.003.patch
>
>
> Chatting today w/ a mighty hbase operator on how to figure what is happening 
> during transitory latency spike or any other transitory 'weirdness' in a 
> server, the idea came up that a java flight recording during a spike would 
> include a pretty good picture of what is going on during the time of duress 
> (more ideal would be a trace of the explicit slow queries showing call stack 
> with timings dumped to a sink for later review; i.e. trigger an htrace when a 
> query is slow...).
> Taking a look, programmatically triggering a JFR recording seems doable, if 
> awkward (MBean invocations). There is even a means of specifying 'triggers' 
> based off any published mbean emission -- e.g. a query queue count threshold 
> -- which looks nice. See 
> https://community.oracle.com/thread/3676275?start=0=0 and 
> https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/run.htm#JFRUH184
> This feature could start out as a blog post describing how to do it for one 
> server. A plugin on Canary that looks at mbean values and if over a 
> configured threshold, triggers a recording remotely could be next. Finally 
> could integrate a couple of triggers that fire when issue via the trigger 
> mechanism.
> Marking as beginner feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15625) Make minimum values configurable and smaller

[Konstantin Ryakhovskiy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394410#comment-15394410
 ] 

Konstantin Ryakhovskiy commented on HBASE-15625:


[~asher] are you working on this issue?
If not, can I re-assign and submit the patch?
Thanks

> Make minimum values configurable and smaller
> 
>
> Key: HBASE-15625
> URL: https://issues.apache.org/jira/browse/HBASE-15625
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jean-Marc Spaggiari
>Assignee: Asher Bartch
>Priority: Minor
>  Labels: beginner
>
> When we start a RS, we check 
> HConstants.HBASE_CLUSTER_MINIMUM_MEMORY_THRESHOLD to make sure we always keep 
> 20% of the heap for HBase (See below). In the past maximum heap size was 
> about 20GB, which means 4GB for HBase.
> Today, with huge heaps and GC1, 20% gives a lot to HBase. Like with 80GB 
> heap, it gives 16GB, which I think it not required.
> We need to make HBASE_CLUSTER_MINIMUM_MEMORY_THRESHOLD configurable and lower 
> it's default value to 10%. It will not make any difference on any HBase 
> configuration but will allow admins to be more flexible.
> Same thing for the minimum memstore and blockcache sizes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-16288:
--
Fix Version/s: 1.2.3
   1.1.6
   1.3.0

> HFile intermediate block level indexes might recurse forever creating multi 
> TB files
> 
>
> Key: HBASE-16288
> URL: https://issues.apache.org/jira/browse/HBASE-16288
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.6, 1.2.3
>
> Attachments: hbase-16288_v1.patch, hbase-16288_v2.patch
>
>
> Mighty [~elserj] was debugging an opentsdb cluster where some region 
> directory ended up having 5TB+ files under /.tmp/ 
> Further debugging and analysis, we were able to reproduce the problem locally 
> where we never we recursing in this code path for writing intermediate level 
> indices: 
> {code:title=HFileBlockIndex.java}
> if (curInlineChunk != null) {
> while (rootChunk.getRootSize() > maxChunkSize) {
>   rootChunk = writeIntermediateLevel(out, rootChunk);
>   numLevels += 1;
> }
>   }
> {code}
> The problem happens if we end up with a very large rowKey (larger than 
> "hfile.index.block.max.size" being the first key in the block, then moving 
> all the way to the root-level index building. We will keep writing and 
> building the next level of intermediate level indices with a single 
> very-large key. This can happen in flush / compaction / region recovery 
> causing cluster inoperability due to ever-growing files. 
> Seems the issue was also reported earlier, with a temporary workaround: 
> https://github.com/OpenTSDB/opentsdb/issues/490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-16288:
--
Fix Version/s: 1.4.0
   2.0.0
   Status: Patch Available  (was: Open)

> HFile intermediate block level indexes might recurse forever creating multi 
> TB files
> 
>
> Key: HBASE-16288
> URL: https://issues.apache.org/jira/browse/HBASE-16288
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: hbase-16288_v1.patch, hbase-16288_v2.patch
>
>
> Mighty [~elserj] was debugging an opentsdb cluster where some region 
> directory ended up having 5TB+ files under /.tmp/ 
> Further debugging and analysis, we were able to reproduce the problem locally 
> where we never we recursing in this code path for writing intermediate level 
> indices: 
> {code:title=HFileBlockIndex.java}
> if (curInlineChunk != null) {
> while (rootChunk.getRootSize() > maxChunkSize) {
>   rootChunk = writeIntermediateLevel(out, rootChunk);
>   numLevels += 1;
> }
>   }
> {code}
> The problem happens if we end up with a very large rowKey (larger than 
> "hfile.index.block.max.size" being the first key in the block, then moving 
> all the way to the root-level index building. We will keep writing and 
> building the next level of intermediate level indices with a single 
> very-large key. This can happen in flush / compaction / region recovery 
> causing cluster inoperability due to ever-growing files. 
> Seems the issue was also reported earlier, with a temporary workaround: 
> https://github.com/OpenTSDB/opentsdb/issues/490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-16288:
--
Attachment: hbase-16288_v2.patch

I'm glad that I have written a unit test for this. There is one more case it 
seems, not just the first key in a block, but we have to make sure that every 
block in an intermediate level index contains at least 2 keys. Otherwise, the 
recursion never stops still. The UT creates 100 blocks of such where each key 
is larger than the max chunk size. 

> HFile intermediate block level indexes might recurse forever creating multi 
> TB files
> 
>
> Key: HBASE-16288
> URL: https://issues.apache.org/jira/browse/HBASE-16288
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Attachments: hbase-16288_v1.patch, hbase-16288_v2.patch
>
>
> Mighty [~elserj] was debugging an opentsdb cluster where some region 
> directory ended up having 5TB+ files under /.tmp/ 
> Further debugging and analysis, we were able to reproduce the problem locally 
> where we never we recursing in this code path for writing intermediate level 
> indices: 
> {code:title=HFileBlockIndex.java}
> if (curInlineChunk != null) {
> while (rootChunk.getRootSize() > maxChunkSize) {
>   rootChunk = writeIntermediateLevel(out, rootChunk);
>   numLevels += 1;
> }
>   }
> {code}
> The problem happens if we end up with a very large rowKey (larger than 
> "hfile.index.block.max.size" being the first key in the block, then moving 
> all the way to the root-level index building. We will keep writing and 
> building the next level of intermediate level indices with a single 
> very-large key. This can happen in flush / compaction / region recovery 
> causing cluster inoperability due to ever-growing files. 
> Seems the issue was also reported earlier, with a temporary workaround: 
> https://github.com/OpenTSDB/opentsdb/issues/490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16275) Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-16275:
-
Attachment: HBASE-16275-v002.patch
TestServerManagerPerf.java

Upload the latest patch and a test program. 

> Change ServerManager#onlineServers from ConcurrentHashMap to 
> ConcurrentSkipListMap
> --
>
> Key: HBASE-16275
> URL: https://issues.apache.org/jira/browse/HBASE-16275
> Project: HBase
>  Issue Type: Improvement
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-16275-v001.patch, HBASE-16275-v002.patch, 
> TestServerManagerPerf.java
>
>
> In Class ServerManager, onlineServers is declared as ConcurrentHashMap. In 
> findServerWithSameHostnamePortWithLock(), it has to do a loop to find if 
> there is a ServerName with same host:port pair. If replaced with 
> ConcurrentSkipListMap, findServerWithSameHostnamePortWithLock() can be 
> replaced with a O(logN) implementation. 
> I run some performance comparison(test the function only), it seems that 
> there is no difference if there are 1000 servers. With more servers, 
> ConcurrentSkipListMap implementation is going to win big.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests

[Joseph (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394352#comment-15394352
 ] 

Joseph commented on HBASE-16209:


Sorry, I forgot to mock a method inside of the UnitTest. It should be passing 
now. I also modified the max sleep time to be the same as the initial sleep 
time interval, defaulting to a constant sleep time strategy if the user only 
sets an initial sleep time.

> Provide an ExponentialBackOffPolicy sleep between failed region open requests
> -
>
> Key: HBASE-16209
> URL: https://issues.apache.org/jira/browse/HBASE-16209
> Project: HBase
>  Issue Type: Bug
>Reporter: Joseph
>Assignee: Joseph
> Attachments: HBASE-16209-v2.patch, HBASE-16209.patch
>
>
> Related to HBASE-16138. As of now we currently have no pause between retrying 
> failed region open requests. And with a low maximumAttempt default, we can 
> quickly use up all our regionOpen retries if the server is in a bad state. I 
> added in a ExponentialBackOffPolicy so that we spread out the timing of our 
> open region retries in AssignmentManager. Review board at 
> https://reviews.apache.org/r/50011/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests

[Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph updated HBASE-16209:
---
Status: Patch Available  (was: Open)

> Provide an ExponentialBackOffPolicy sleep between failed region open requests
> -
>
> Key: HBASE-16209
> URL: https://issues.apache.org/jira/browse/HBASE-16209
> Project: HBase
>  Issue Type: Bug
>Reporter: Joseph
>Assignee: Joseph
> Attachments: HBASE-16209-v2.patch, HBASE-16209.patch
>
>
> Related to HBASE-16138. As of now we currently have no pause between retrying 
> failed region open requests. And with a low maximumAttempt default, we can 
> quickly use up all our regionOpen retries if the server is in a bad state. I 
> added in a ExponentialBackOffPolicy so that we spread out the timing of our 
> open region retries in AssignmentManager. Review board at 
> https://reviews.apache.org/r/50011/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests

[Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph updated HBASE-16209:
---
Status: Open  (was: Patch Available)

> Provide an ExponentialBackOffPolicy sleep between failed region open requests
> -
>
> Key: HBASE-16209
> URL: https://issues.apache.org/jira/browse/HBASE-16209
> Project: HBase
>  Issue Type: Bug
>Reporter: Joseph
>Assignee: Joseph
> Attachments: HBASE-16209-v2.patch, HBASE-16209.patch
>
>
> Related to HBASE-16138. As of now we currently have no pause between retrying 
> failed region open requests. And with a low maximumAttempt default, we can 
> quickly use up all our regionOpen retries if the server is in a bad state. I 
> added in a ExponentialBackOffPolicy so that we spread out the timing of our 
> open region retries in AssignmentManager. Review board at 
> https://reviews.apache.org/r/50011/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16209) Provide an ExponentialBackOffPolicy sleep between failed region open requests

[Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph updated HBASE-16209:
---
Attachment: HBASE-16209-v2.patch

> Provide an ExponentialBackOffPolicy sleep between failed region open requests
> -
>
> Key: HBASE-16209
> URL: https://issues.apache.org/jira/browse/HBASE-16209
> Project: HBase
>  Issue Type: Bug
>Reporter: Joseph
>Assignee: Joseph
> Attachments: HBASE-16209-v2.patch, HBASE-16209.patch
>
>
> Related to HBASE-16138. As of now we currently have no pause between retrying 
> failed region open requests. And with a low maximumAttempt default, we can 
> quickly use up all our regionOpen retries if the server is in a bad state. I 
> added in a ExponentialBackOffPolicy so that we spread out the timing of our 
> open region retries in AssignmentManager. Review board at 
> https://reviews.apache.org/r/50011/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16260) Audit dependencies for Category-X

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394219#comment-15394219
 ] 

Andrew Purtell commented on HBASE-16260:


Inscrutable velocity errors and some on list or JIRA churn figuring out the 
problem would be superior to releasing something including category X. 

> Audit dependencies for Category-X
> -
>
> Key: HBASE-16260
> URL: https://issues.apache.org/jira/browse/HBASE-16260
> Project: HBase
>  Issue Type: Task
>  Components: community, dependencies
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 1.1.5, 1.2.2
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.0.0, 1.1.6, 1.2.3
>
>
> Make sure we do not have category x dependencies.
> right now we atleast have an LGPL for xom:xom (thanks to PHOENIX-3103 for the 
> catch)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-16288:
--
Attachment: hbase-16288_v1.patch

This fixes the problem (I've manually verified with real data). Let's see 
whether I can come up with a test. 

> HFile intermediate block level indexes might recurse forever creating multi 
> TB files
> 
>
> Key: HBASE-16288
> URL: https://issues.apache.org/jira/browse/HBASE-16288
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Attachments: hbase-16288_v1.patch
>
>
> Mighty [~elserj] was debugging an opentsdb cluster where some region 
> directory ended up having 5TB+ files under /.tmp/ 
> Further debugging and analysis, we were able to reproduce the problem locally 
> where we never we recursing in this code path for writing intermediate level 
> indices: 
> {code:title=HFileBlockIndex.java}
> if (curInlineChunk != null) {
> while (rootChunk.getRootSize() > maxChunkSize) {
>   rootChunk = writeIntermediateLevel(out, rootChunk);
>   numLevels += 1;
> }
>   }
> {code}
> The problem happens if we end up with a very large rowKey (larger than 
> "hfile.index.block.max.size" being the first key in the block, then moving 
> all the way to the root-level index building. We will keep writing and 
> building the next level of intermediate level indices with a single 
> very-large key. This can happen in flush / compaction / region recovery 
> causing cluster inoperability due to ever-growing files. 
> Seems the issue was also reported earlier, with a temporary workaround: 
> https://github.com/OpenTSDB/opentsdb/issues/490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files

[Enis Soztutar (JIRA)]

Enis Soztutar created HBASE-16288:
-

 Summary: HFile intermediate block level indexes might recurse 
forever creating multi TB files
 Key: HBASE-16288
 URL: https://issues.apache.org/jira/browse/HBASE-16288
 Project: HBase
  Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar


Mighty [~elserj] was debugging an opentsdb cluster where some region directory 
ended up having 5TB+ files under /.tmp/ 

Further debugging and analysis, we were able to reproduce the problem locally 
where we never we recursing in this code path for writing intermediate level 
indices: 

{code:title=HFileBlockIndex.java}
if (curInlineChunk != null) {
while (rootChunk.getRootSize() > maxChunkSize) {
  rootChunk = writeIntermediateLevel(out, rootChunk);
  numLevels += 1;
}
  }
{code}

The problem happens if we end up with a very large rowKey (larger than 
"hfile.index.block.max.size" being the first key in the block, then moving all 
the way to the root-level index building. We will keep writing and building the 
next level of intermediate level indices with a single very-large key. This can 
happen in flush / compaction / region recovery causing cluster inoperability 
due to ever-growing files. 

Seems the issue was also reported earlier, with a temporary workaround: 
https://github.com/OpenTSDB/opentsdb/issues/490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16288) HFile intermediate block level indexes might recurse forever creating multi TB files

[Enis Soztutar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Enis Soztutar updated HBASE-16288:
--
Priority: Critical  (was: Major)

> HFile intermediate block level indexes might recurse forever creating multi 
> TB files
> 
>
> Key: HBASE-16288
> URL: https://issues.apache.org/jira/browse/HBASE-16288
> Project: HBase
>  Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
>
> Mighty [~elserj] was debugging an opentsdb cluster where some region 
> directory ended up having 5TB+ files under /.tmp/ 
> Further debugging and analysis, we were able to reproduce the problem locally 
> where we never we recursing in this code path for writing intermediate level 
> indices: 
> {code:title=HFileBlockIndex.java}
> if (curInlineChunk != null) {
> while (rootChunk.getRootSize() > maxChunkSize) {
>   rootChunk = writeIntermediateLevel(out, rootChunk);
>   numLevels += 1;
> }
>   }
> {code}
> The problem happens if we end up with a very large rowKey (larger than 
> "hfile.index.block.max.size" being the first key in the block, then moving 
> all the way to the root-level index building. We will keep writing and 
> building the next level of intermediate level indices with a single 
> very-large key. This can happen in flush / compaction / region recovery 
> causing cluster inoperability due to ever-growing files. 
> Seems the issue was also reported earlier, with a temporary workaround: 
> https://github.com/OpenTSDB/opentsdb/issues/490



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys

[Mikhail Antonov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394195#comment-15394195
 ] 

Mikhail Antonov commented on HBASE-16232:
-

With enough regions in the cluster overall (up to high tens thousands, though 
most were just empty) I got one failed run with lost keys on 1.2.1.

> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
>  Issue Type: Bug
>  Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix 
> for fake keys showing up in the scans) with increased number of regions per 
> regionserver and seeing the following.
> {quote} 
> $Verify​$Counts   
> REFERENCED0   4,999,999,994   4,999,999,994
> UNDEFINED 0   3   3
> UNREFERENCED  0   3   3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef 
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F  0   1   1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE30   1   1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A  0   1   1
> unref 
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0   1   1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC  0   1   1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool 
> reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE35   0   5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A  4   0   4
> CELL_WITH_MISSING_ROW 15  0   15
> {quote}
> Will post more as I dig into.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16281) TestMasterReplication is flaky

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394192#comment-15394192
 ] 

Hudson commented on HBASE-16281:


ABORTED: Integrated in HBase-0.98-matrix #377 (See 
[https://builds.apache.org/job/HBase-0.98-matrix/377/])
HBASE-16281 TestMasterReplication is flaky (zhangduo: rev 
f982f50f53ac96f796cf1905b62ecfd0168bdb66)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


> TestMasterReplication is flaky
> --
>
> Key: HBASE-16281
> URL: https://issues.apache.org/jira/browse/HBASE-16281
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.2, 0.98.20, 1.1.6
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21, 1.2.3, 1.1.7
>
> Attachments: HBASE-16281-v1.patch, HBASE-16281-v1.patch, 
> HBASE-16281-v1.patch
>
>
> In TestMasterReplication we put some mutations and wait until we can read the 
> data from slave cluster. However the waiting time is too short. Replication 
> service in slave cluster may not be initialized and ready to handle 
> replication RPC requests in several seconds. 
> We should wait for more time.
> {quote}
> 2016-07-25 11:47:03,156 WARN  [Time-limited 
> test-EventThread.replicationSource,1.replicationSource.10.235.114.28%2C56313%2C1469418386448,1]
>  regionserver.HBaseInterClusterReplicationEndpoint(310): Can't replicate 
> because of a local or network error: 
> java.io.IOException: java.io.IOException: Replication services are not 
> initialized yet
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2263)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> Caused by: com.google.protobuf.ServiceException: Replication services are not 
> initialized yet
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replicateWALEntry(RSRpcServices.java:1935)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22751)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2212)
>   ... 3 more
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (HBASE-16249) Remove direct use of Hadoop Path/FileSysem 1/5

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HBASE-16249 started by Sean Busbey.
---
> Remove direct use of Hadoop Path/FileSysem 1/5
> --
>
> Key: HBASE-16249
> URL: https://issues.apache.org/jira/browse/HBASE-16249
> Project: HBase
>  Issue Type: Sub-task
>  Components: Filesystem Integration
>Reporter: Sean Busbey
>Assignee: Sean Busbey
> Fix For: 2.0.0
>
>
> Remove or document exceptions for direct use of Hadoop FileSystem APIs in
> {code}
> org/apache/hadoop/hbase/  
> org/apache/hadoop/hbase/backup/   
> org/apache/hadoop/hbase/backup/example/   
> org/apache/hadoop/hbase/client/   
> org/apache/hadoop/hbase/client/coprocessor/   
> org/apache/hadoop/hbase/coordination/ 
> org/apache/hadoop/hbase/coprocessor/  
> org/apache/hadoop/hbase/fs/   
> org/apache/hadoop/hbase/io/   
> org/apache/hadoop/hbase/io/asyncfs/   
> org/apache/hadoop/hbase/io/encoding/  
> org/apache/hadoop/hbase/mapred/   
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-16096:

Fix Version/s: 1.4.0

> Replication keeps accumulating znodes
> -
>
> Key: HBASE-16096
> URL: https://issues.apache.org/jira/browse/HBASE-16096
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Joseph
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16096-branch-1.patch, HBASE-16096.patch
>
>
> If there is an error while creating the replication source on adding the 
> peer, the source if not added to the in memory list of sources but the 
> replication peer is. 
> However, in such a scenario, when you remove the peer, it is deleted from 
> zookeeper successfully but for removing the in memory list of peers, we wait 
> for the corresponding sources to get deleted (which as we said don't exist 
> because of error creating the source). 
> The problem here is the ordering of operations for adding/removing source and 
> peer. 
> Modifying the code to always remove queues from the underlying storage, even 
> if there exists no sources also requires a small refactoring of 
> TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty 
> queue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HBASE-16096) Replication keeps accumulating znodes

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey reopened HBASE-16096:
-

reopening so we can get a precommit check on branch-1 patch.

> Replication keeps accumulating znodes
> -
>
> Key: HBASE-16096
> URL: https://issues.apache.org/jira/browse/HBASE-16096
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Joseph
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16096-branch-1.patch, HBASE-16096.patch
>
>
> If there is an error while creating the replication source on adding the 
> peer, the source if not added to the in memory list of sources but the 
> replication peer is. 
> However, in such a scenario, when you remove the peer, it is deleted from 
> zookeeper successfully but for removing the in memory list of peers, we wait 
> for the corresponding sources to get deleted (which as we said don't exist 
> because of error creating the source). 
> The problem here is the ordering of operations for adding/removing source and 
> peer. 
> Modifying the code to always remove queues from the underlying storage, even 
> if there exists no sources also requires a small refactoring of 
> TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty 
> queue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey updated HBASE-16096:

Status: Patch Available  (was: Reopened)

> Replication keeps accumulating znodes
> -
>
> Key: HBASE-16096
> URL: https://issues.apache.org/jira/browse/HBASE-16096
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 1.2.0, 2.0.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Joseph
> Fix For: 2.0.0
>
> Attachments: HBASE-16096-branch-1.patch, HBASE-16096.patch
>
>
> If there is an error while creating the replication source on adding the 
> peer, the source if not added to the in memory list of sources but the 
> replication peer is. 
> However, in such a scenario, when you remove the peer, it is deleted from 
> zookeeper successfully but for removing the in memory list of peers, we wait 
> for the corresponding sources to get deleted (which as we said don't exist 
> because of error creating the source). 
> The problem here is the ordering of operations for adding/removing source and 
> peer. 
> Modifying the code to always remove queues from the underlying storage, even 
> if there exists no sources also requires a small refactoring of 
> TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty 
> queue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16096) Replication keeps accumulating znodes

[Joseph (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394172#comment-15394172
 ] 

Joseph commented on HBASE-16096:


Just attached a patch rebased on branch-1

> Replication keeps accumulating znodes
> -
>
> Key: HBASE-16096
> URL: https://issues.apache.org/jira/browse/HBASE-16096
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Joseph
> Fix For: 2.0.0
>
> Attachments: HBASE-16096-branch-1.patch, HBASE-16096.patch
>
>
> If there is an error while creating the replication source on adding the 
> peer, the source if not added to the in memory list of sources but the 
> replication peer is. 
> However, in such a scenario, when you remove the peer, it is deleted from 
> zookeeper successfully but for removing the in memory list of peers, we wait 
> for the corresponding sources to get deleted (which as we said don't exist 
> because of error creating the source). 
> The problem here is the ordering of operations for adding/removing source and 
> peer. 
> Modifying the code to always remove queues from the underlying storage, even 
> if there exists no sources also requires a small refactoring of 
> TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty 
> queue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16096) Replication keeps accumulating znodes

[Joseph (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph updated HBASE-16096:
---
Attachment: HBASE-16096-branch-1.patch

> Replication keeps accumulating znodes
> -
>
> Key: HBASE-16096
> URL: https://issues.apache.org/jira/browse/HBASE-16096
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Affects Versions: 2.0.0, 1.2.0, 1.3.0
>Reporter: Ashu Pachauri
>Assignee: Joseph
> Fix For: 2.0.0
>
> Attachments: HBASE-16096-branch-1.patch, HBASE-16096.patch
>
>
> If there is an error while creating the replication source on adding the 
> peer, the source if not added to the in memory list of sources but the 
> replication peer is. 
> However, in such a scenario, when you remove the peer, it is deleted from 
> zookeeper successfully but for removing the in memory list of peers, we wait 
> for the corresponding sources to get deleted (which as we said don't exist 
> because of error creating the source). 
> The problem here is the ordering of operations for adding/removing source and 
> peer. 
> Modifying the code to always remove queues from the underlying storage, even 
> if there exists no sources also requires a small refactoring of 
> TableBasedReplicationQueuesImpl to not abort on removeQueues() of an empty 
> queue



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16176) Bug fixes/improvements on HBASE-15650 Remove TimeRangeTracker as point of contention when many threads reading a StoreFile

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-16176:
--
Attachment: (was: HBASE-16176.master.002.patch)

> Bug fixes/improvements on HBASE-15650 Remove TimeRangeTracker as point of 
> contention when many threads reading a StoreFile
> --
>
> Key: HBASE-16176
> URL: https://issues.apache.org/jira/browse/HBASE-16176
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 1.3.0, 0.98.20
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16176.branch-1.3.001.patch, 
> HBASE-16176.branch-1.3.002.patch, HBASE-16176.branch-1.3.002.patch, 
> HBASE-16176.branch-1.3.003.patch, HBASE-16176.master.001.patch
>
>
> Debugging the parent issue, came up with some improvements on old HBASE-15650 
> "Remove TimeRangeTracker as point of contention when many threads reading a 
> StoreFile". Lets get them in. Here are the changes:
> {code}
>   6  Change HFile Writer constructor so we pass in the TimeRangeTracker, 
> if one,
>   7  on construction rather than set later (the flag and reference were 
> not
>   8  volatile so could have made for issues in concurrent case) 2. Make 
> sure the
>   9  construction of a TimeRange from a TimeRangeTracer on open of an 
> HFile Reader
>  10  never makes a bad minimum value, one that would preclude us reading 
> any
>  11  values from a file (add a log and set min to 0)
>  12 M hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
>  13  Call through to next constructor (if minStamp was 0, we'd skip 
> setting allTime=true)
>  14 M 
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
>  15  Add constructor override that takes a TimeRangeTracker (set when 
> flushing but
>  16  not when compacting)
>  17 M 
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
>  18  Add override creating an HFile in tmp that takes a TimeRangeTracker
>  19 M 
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
>  20  Add override for HFile Writer that takes a TimeRangeTracker
>  21  Take it on construction instead of having it passed by a setter 
> later (flags
>  22  and reference set by the setter were not volatile... could have been 
> prob
>  23  in concurrent case)
>  24 M 
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
>  25  Log WARN if bad initial TimeRange value (and then 'fix' it)
>  26 M 
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTimeRangeTracker.java
>  27  A few tests to prove serialization works as expected and that we'll 
> get a bad min if
>  28  not constructed properly.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.006.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394133#comment-15394133
 ] 

stack commented on HBASE-15638:
---

Ok. v6 should be better. Includes a commit message describing content too. 
Chasing down all pb references and getting all under shade is a bit of cat 
herding. BulkDelete.proto seems to have gone missing too in our recent history 
but this patch does restore. Lets see how it does.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16176) Bug fixes/improvements on HBASE-15650 Remove TimeRangeTracker as point of contention when many threads reading a StoreFile

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-16176:
--
Attachment: HBASE-16176.master.002.patch

> Bug fixes/improvements on HBASE-15650 Remove TimeRangeTracker as point of 
> contention when many threads reading a StoreFile
> --
>
> Key: HBASE-16176
> URL: https://issues.apache.org/jira/browse/HBASE-16176
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Affects Versions: 1.3.0, 0.98.20
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16176.branch-1.3.001.patch, 
> HBASE-16176.branch-1.3.002.patch, HBASE-16176.branch-1.3.002.patch, 
> HBASE-16176.branch-1.3.003.patch, HBASE-16176.master.001.patch, 
> HBASE-16176.master.002.patch
>
>
> Debugging the parent issue, came up with some improvements on old HBASE-15650 
> "Remove TimeRangeTracker as point of contention when many threads reading a 
> StoreFile". Lets get them in. Here are the changes:
> {code}
>   6  Change HFile Writer constructor so we pass in the TimeRangeTracker, 
> if one,
>   7  on construction rather than set later (the flag and reference were 
> not
>   8  volatile so could have made for issues in concurrent case) 2. Make 
> sure the
>   9  construction of a TimeRange from a TimeRangeTracer on open of an 
> HFile Reader
>  10  never makes a bad minimum value, one that would preclude us reading 
> any
>  11  values from a file (add a log and set min to 0)
>  12 M hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
>  13  Call through to next constructor (if minStamp was 0, we'd skip 
> setting allTime=true)
>  14 M 
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
>  15  Add constructor override that takes a TimeRangeTracker (set when 
> flushing but
>  16  not when compacting)
>  17 M 
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
>  18  Add override creating an HFile in tmp that takes a TimeRangeTracker
>  19 M 
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
>  20  Add override for HFile Writer that takes a TimeRangeTracker
>  21  Take it on construction instead of having it passed by a setter 
> later (flags
>  22  and reference set by the setter were not volatile... could have been 
> prob
>  23  in concurrent case)
>  24 M 
> hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
>  25  Log WARN if bad initial TimeRange value (and then 'fix' it)
>  26 M 
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTimeRangeTracker.java
>  27  A few tests to prove serialization works as expected and that we'll 
> get a bad min if
>  28  not constructed properly.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16281) TestMasterReplication is flaky

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394124#comment-15394124
 ] 

Hudson commented on HBASE-16281:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1249 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1249/])
HBASE-16281 TestMasterReplication is flaky (zhangduo: rev 
f982f50f53ac96f796cf1905b62ecfd0168bdb66)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


> TestMasterReplication is flaky
> --
>
> Key: HBASE-16281
> URL: https://issues.apache.org/jira/browse/HBASE-16281
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.2, 0.98.20, 1.1.6
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21, 1.2.3, 1.1.7
>
> Attachments: HBASE-16281-v1.patch, HBASE-16281-v1.patch, 
> HBASE-16281-v1.patch
>
>
> In TestMasterReplication we put some mutations and wait until we can read the 
> data from slave cluster. However the waiting time is too short. Replication 
> service in slave cluster may not be initialized and ready to handle 
> replication RPC requests in several seconds. 
> We should wait for more time.
> {quote}
> 2016-07-25 11:47:03,156 WARN  [Time-limited 
> test-EventThread.replicationSource,1.replicationSource.10.235.114.28%2C56313%2C1469418386448,1]
>  regionserver.HBaseInterClusterReplicationEndpoint(310): Can't replicate 
> because of a local or network error: 
> java.io.IOException: java.io.IOException: Replication services are not 
> initialized yet
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2263)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> Caused by: com.google.protobuf.ServiceException: Replication services are not 
> initialized yet
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replicateWALEntry(RSRpcServices.java:1935)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22751)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2212)
>   ... 3 more
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-14921) Memory optimizations

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394006#comment-15394006
 ] 

stack edited comment on HBASE-14921 at 7/26/16 4:11 PM:


Let me do some more review of the posted patch. I promise not to expand scope 
(other than as new related JIRAs). May I have a bit of overview of what is in 
it [~anastas] currently. My latest reviews are messy as I'm bad on context 
(point at doc if current patch aligns still -- I'm not sure). Thanks.




was (Author: stack):
Let me do some more review of the posted patch. May I have a bit of overview of 
what is in it [~anastas] currently. My latest reviews are messy as I'm bad on 
context (point at doc if current patch aligns still -- I'm not sure). Thanks.



> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.005.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-14921) Memory optimizations

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394006#comment-15394006
 ] 

stack edited comment on HBASE-14921 at 7/26/16 4:08 PM:


Let me do some more review of the posted patch. May I have a bit of overview of 
what is in it [~anastas] currently. My latest reviews are messy as I'm bad on 
context (point at doc if current patch aligns still -- I'm not sure). Thanks.




was (Author: stack):
Let me do some more review of the posted patch. It is a bit messy.
St.Ack

On Tue, Jul 26, 2016 at 9:04 AM, Anoop Sam John (JIRA) 



> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394006#comment-15394006
 ] 

stack commented on HBASE-14921:
---

Let me do some more review of the posted patch. It is a bit messy.
St.Ack

On Tue, Jul 26, 2016 at 9:04 AM, Anoop Sam John (JIRA) 



> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394004#comment-15394004
 ] 

Anoop Sam John commented on HBASE-14921:


+1

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14921) Memory optimizations

[Edward Bortnikov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393993#comment-15393993
 ] 

Edward Bortnikov commented on HBASE-14921:
--

Thanks, all, for the last round ... Seems like we are converging. 

So, we seem to agree on the following:
1. At the end of the day, we want compaction (data de-duplication) where it 
helps, and don't want it where it doesn't. That might use a flag or some smart 
policy. Getting there might take 1-2 more (smaller) patches. 
2. We want to land the current patch because it's getting big. The mandatory 
condition for passing it is bug-freedom (via improved coverage), including the 
more stressful PE tests. Hopeful to finish this week or early next week. 
3. We want to release all the dependencies for [~anoop.hbase] and [~ram_krish] 
on their way to off-heap memory implementation. We'll switch to that as soon as 
the current patch is landed. We have a common goal (smile). 

Deal? 

> Memory optimizations
> 
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf, 
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, 
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, 
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, 
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap 
> allocations



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393830#comment-15393830
 ] 

Hadoop QA commented on HBASE-16285:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m 6s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 17s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12820156/HBASE-16285-v1.patch |
| JIRA Issue | HBASE-16285 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / bcf409e |
| Default Java | 1.7.0_80 |
| Multi-JDK versions |  /home/jenkins/tools/java/jdk1.8.0:1.8.0 
/home/jenkins/jenkins-slave/tools/hudson.model.JDK/JDK_1.7_latest_:1.7.0_80 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/2772/testReport/ 

[jira] [Commented] (HBASE-16286) When TagRewriteCell are not copied to MSLAB, deep clone it while adding to Memstore

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393816#comment-15393816
 ] 

Hadoop QA commented on HBASE-16286:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 36s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 24m 35s 
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 28m 3s 
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 31m 37s 
{color} | {color:red} Patch causes 11 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 35m 6s 
{color} | {color:red} Patch causes 11 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 4s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 128m 53s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
42s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 194m 41s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 

[jira] [Resolved] (HBASE-9876) Backport HBASE-4285 "partitions file created in user's home directory"

[Sean Busbey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-9876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Busbey resolved HBASE-9876.

   Resolution: Won't Fix
Fix Version/s: (was: 0.94.28)

closing this as stale, due to pending EOM for 0.94 branch. If anyone would like 
to pick it back up, please feel free to reopen.

> Backport HBASE-4285 "partitions file created in user's home directory"
> --
>
> Key: HBASE-9876
> URL: https://issues.apache.org/jira/browse/HBASE-9876
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Affects Versions: 0.94.12
>Reporter: Nick Dimiduk
>Priority: Minor
>
> The main benefit brought by HBASE-4285 was removing the dependency on a 
> symlink. Instead, it sets the partitions file path explicitly. Would be nice 
> to have on 0.94.
> (cc [~alexandre.normand])



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16287) BlockCache size should not exceed acceptableSize too many

[Yu Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Sun updated HBASE-16287:
---
Description: 
Our regionserver has a configuation as bellow：
  -Xmn4g -Xms32g -Xmx32g -XX:SurvriorRatio=2 -XX:+UseConcMarkSweepGC 
also we only use blockcache,and set hfile.block.cache.size = 0.3 in 
hbase_site.xml,so under this configuration, the lru block cache size will 
be(32g-1g)*0.3=9.3g. but in some scenarios，some of the rs will occur continuous 
FullGC  for hours and most importantly, after FullGC most of the object in old 
will not be GCed. so we dump the heap and analyse with MAT and we observed a 
obvious memory leak in LruBlockCache, which occpy about 16g memory, then we set 
set class LruBlockCache log level to TRACE and observed this in log:

{quote}
2016-07-22 12:17:58,158 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: 
totalSize=15.29 GB, freeSize=-5.99 GB, max=9.30 GB, blockCount=628182, 
accesses=101799469125, hits=93517800259, hitRatio=91.86%, , 
cachingAccesses=99462650031, cachingHits=93468334621, cachingHitsRatio=93.97%, 
evictions=238199, evicted=4776350518, evictedPerRun=20051.93359375{quote}

we can see blockcache size has exceeded acceptableSize too many, which will 
cause the FullGC more seriously. 
Afterfter some investigations, I found in this function:

{code:borderStyle=solid}
  public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean 
inMemory,
  final boolean cacheDataInL1) {
{code}


No matter the blockcache size has been used, just put the block into it. but if 
the evict thread is not fast enough, blockcache size will increament 
significantly.
So here I think we should have a check, for example, if the blockcache size > 
1.2 * acceptableSize(), just return and dont put into it until the blockcache 
size if under watrmark. if this is reasonable, I can make a small patch for 
this.

  was:
Our regionserver has a configuation as bellow：
  -Xmn4g -Xms32g -Xmx32g -XX:SurvriorRatio=2 -XX:+UseConcMarkSweepGC 
also we only use blockcache,and set hfile.block.cache.size = 0.3 in 
hbase_site.xml,so under this configuration, the lru block cache size will 
be(32g-1g)*0.3=9.3g. but in some scenarios，some of the rs will occur continuous 
FullGC  for hours and most importantly, after FullGC most of the object in old 
will not be GCed. so we dump the heap and analyse with MAT and we observed a 
obvious memory leak in LruBlockCache, which occpy about 16g memory, then we set 
set class LruBlockCache log level to TRACE and observed this in log:

2016-07-22 12:17:58,158 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: 
totalSize=15.29 GB, freeSize=-5.99 GB, max=9.30 GB, blockCount=628182, 
accesses=101799469125, hits=93517800259, hitRatio=91.86%, , 
cachingAccesses=99462650031, cachingHits=93468334621, cachingHitsRatio=93.97%, 
evictions=238199, evicted=4776350518, evictedPerRun=20051.93359375

we can see blockcache size has exceeded acceptableSize too many, which will 
cause the FullGC more seriously. 
Afterfter some investigations, I found in this function:

{code:borderStyle=solid}
  public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean 
inMemory,
  final boolean cacheDataInL1) {
{code}


No matter the blockcache size has been used, just put the block into it. but if 
the evict thread is not fast enough, blockcache size will increament 
significantly.
So here I think we should have a check, for example, if the blockcache size > 
1.2 * acceptableSize(), just return and dont put into it until the blockcache 
size if under watrmark. if this is reasonable, I can make a small patch for 
this.


> BlockCache size should not exceed acceptableSize too many
> -
>
> Key: HBASE-16287
> URL: https://issues.apache.org/jira/browse/HBASE-16287
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache
>Reporter: Yu Sun
>
> Our regionserver has a configuation as bellow：
>   -Xmn4g -Xms32g -Xmx32g -XX:SurvriorRatio=2 -XX:+UseConcMarkSweepGC 
> also we only use blockcache,and set hfile.block.cache.size = 0.3 in 
> hbase_site.xml,so under this configuration, the lru block cache size will 
> be(32g-1g)*0.3=9.3g. but in some scenarios，some of the rs will occur 
> continuous FullGC  for hours and most importantly, after FullGC most of the 
> object in old will not be GCed. so we dump the heap and analyse with MAT and 
> we observed a obvious memory leak in LruBlockCache, which occpy about 16g 
> memory, then we set set class LruBlockCache log level to TRACE and observed 
> this in log:
> {quote}
> 2016-07-22 12:17:58,158 INFO  [LruBlockCacheStatsExecutor] 
> hfile.LruBlockCache: totalSize=15.29 GB, freeSize=-5.99 GB, max=9.30 GB, 
> blockCount=628182, accesses=101799469125, hits=93517800259, hitRatio=91.86%, 
> , 

[jira] [Updated] (HBASE-16287) BlockCache size should not exceed acceptableSize too many

[Yu Sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yu Sun updated HBASE-16287:
---
Description: 
Our regionserver has a configuation as bellow：
  -Xmn4g -Xms32g -Xmx32g -XX:SurvriorRatio=2 -XX:+UseConcMarkSweepGC 
also we only use blockcache,and set hfile.block.cache.size = 0.3 in 
hbase_site.xml,so under this configuration, the lru block cache size will 
be(32g-1g)*0.3=9.3g. but in some scenarios，some of the rs will occur continuous 
FullGC  for hours and most importantly, after FullGC most of the object in old 
will not be GCed. so we dump the heap and analyse with MAT and we observed a 
obvious memory leak in LruBlockCache, which occpy about 16g memory, then we set 
set class LruBlockCache log level to TRACE and observed this in log:

2016-07-22 12:17:58,158 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: 
totalSize=15.29 GB, freeSize=-5.99 GB, max=9.30 GB, blockCount=628182, 
accesses=101799469125, hits=93517800259, hitRatio=91.86%, , 
cachingAccesses=99462650031, cachingHits=93468334621, cachingHitsRatio=93.97%, 
evictions=238199, evicted=4776350518, evictedPerRun=20051.93359375

we can see blockcache size has exceeded acceptableSize too many, which will 
cause the FullGC more seriously. 
Afterfter some investigations, I found in this function:

{code:borderStyle=solid}
  public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean 
inMemory,
  final boolean cacheDataInL1) {
{code}


No matter the blockcache size has been used, just put the block into it. but if 
the evict thread is not fast enough, blockcache size will increament 
significantly.
So here I think we should have a check, for example, if the blockcache size > 
1.2 * acceptableSize(), just return and dont put into it until the blockcache 
size if under watrmark. if this is reasonable, I can make a small patch for 
this.

  was:
Our regionserver has a configuation as bellow：
  -Xmn4g -Xms32g -Xmx32g -XX:SurvriorRatio=2 -XX:+UseConcMarkSweepGC 
also we only use blockcache,and set hfile.block.cache.size = 0.3 in 
hbase_site.xml,so under this configuration, the lru block cache size will 
be(32g-1g)*0.3=9.3g. but in some scenarios，some of the rs will occur continuous 
FullGC  for hours and most importantly, after FullGC most of the object in old 
will not be GCed. so we dump the heap and analyse with MAT and we observed a 
obvious memory leak in LruBlockCache, which occpy about 16g memory, then we set 
set class LruBlockCache log level to TRACE and observed this in log:

2016-07-22 12:17:58,158 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: 
totalSize=15.29 GB, freeSize=-5.99 GB, max=9.30 GB, blockCount=628182, 
accesses=101799469125, hits=93517800259, hitRatio=91.86%, , 
cachingAccesses=99462650031, cachingHits=93468334621, cachingHitsRatio=93.97%, 
evictions=238199, evicted=4776350518, evictedPerRun=20051.93359375

we can see blockcache size has exceeded acceptableSize too many, which will 
cause the FullGC more seriously. 
Afterfter some investigations, I found in this function:

  public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean 
inMemory,
  final boolean cacheDataInL1) {

No matter the blockcache size has been used, just put the block into it. but if 
the evict thread is not fast enough, blockcache size will increament 
significantly.
So here I think we should have a check, for example, if the blockcache size > 
1.2 * acceptableSize(), just return and dont put into it until the blockcache 
size if under watrmark. if this is reasonable, I can make a small patch for 
this.


> BlockCache size should not exceed acceptableSize too many
> -
>
> Key: HBASE-16287
> URL: https://issues.apache.org/jira/browse/HBASE-16287
> Project: HBase
>  Issue Type: Improvement
>  Components: BlockCache
>Reporter: Yu Sun
>
> Our regionserver has a configuation as bellow：
>   -Xmn4g -Xms32g -Xmx32g -XX:SurvriorRatio=2 -XX:+UseConcMarkSweepGC 
> also we only use blockcache,and set hfile.block.cache.size = 0.3 in 
> hbase_site.xml,so under this configuration, the lru block cache size will 
> be(32g-1g)*0.3=9.3g. but in some scenarios，some of the rs will occur 
> continuous FullGC  for hours and most importantly, after FullGC most of the 
> object in old will not be GCed. so we dump the heap and analyse with MAT and 
> we observed a obvious memory leak in LruBlockCache, which occpy about 16g 
> memory, then we set set class LruBlockCache log level to TRACE and observed 
> this in log:
> 2016-07-22 12:17:58,158 INFO  [LruBlockCacheStatsExecutor] 
> hfile.LruBlockCache: totalSize=15.29 GB, freeSize=-5.99 GB, max=9.30 GB, 
> blockCount=628182, accesses=101799469125, hits=93517800259, hitRatio=91.86%, 
> , cachingAccesses=99462650031, cachingHits=93468334621, 
> 

[jira] [Created] (HBASE-16287) BlockCache size should not exceed acceptableSize too many

[Yu Sun (JIRA)]

Yu Sun created HBASE-16287:
--

 Summary: BlockCache size should not exceed acceptableSize too many
 Key: HBASE-16287
 URL: https://issues.apache.org/jira/browse/HBASE-16287
 Project: HBase
  Issue Type: Improvement
  Components: BlockCache
Reporter: Yu Sun


Our regionserver has a configuation as bellow：
  -Xmn4g -Xms32g -Xmx32g -XX:SurvriorRatio=2 -XX:+UseConcMarkSweepGC 
also we only use blockcache,and set hfile.block.cache.size = 0.3 in 
hbase_site.xml,so under this configuration, the lru block cache size will 
be(32g-1g)*0.3=9.3g. but in some scenarios，some of the rs will occur continuous 
FullGC  for hours and most importantly, after FullGC most of the object in old 
will not be GCed. so we dump the heap and analyse with MAT and we observed a 
obvious memory leak in LruBlockCache, which occpy about 16g memory, then we set 
set class LruBlockCache log level to TRACE and observed this in log:

2016-07-22 12:17:58,158 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: 
totalSize=15.29 GB, freeSize=-5.99 GB, max=9.30 GB, blockCount=628182, 
accesses=101799469125, hits=93517800259, hitRatio=91.86%, , 
cachingAccesses=99462650031, cachingHits=93468334621, cachingHitsRatio=93.97%, 
evictions=238199, evicted=4776350518, evictedPerRun=20051.93359375

we can see blockcache size has exceeded acceptableSize too many, which will 
cause the FullGC more seriously. 
Afterfter some investigations, I found in this function:

  public void cacheBlock(BlockCacheKey cacheKey, Cacheable buf, boolean 
inMemory,
  final boolean cacheDataInL1) {

No matter the blockcache size has been used, just put the block into it. but if 
the evict thread is not fast enough, blockcache size will increament 
significantly.
So here I think we should have a check, for example, if the blockcache size > 
1.2 * acceptableSize(), just return and dont put into it until the blockcache 
size if under watrmark. if this is reasonable, I can make a small patch for 
this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16234) Expect and handle nulls when assigning replicas

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-16234:
--
Attachment: HBASE-16234-V1.patch

> Expect and handle nulls when assigning replicas
> ---
>
> Key: HBASE-16234
> URL: https://issues.apache.org/jira/browse/HBASE-16234
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>Affects Versions: 2.0.0
>Reporter: Harsh J
>Assignee: Yi Liang
> Attachments: HBASE-16234-V1.patch, HBASE-16234-V1.patch
>
>
> Observed this on a cluster:
> {code}
> FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting 
> shutdown. 
> java.lang.NullPointerException 
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.replicaRegionsNotRecordedInMeta(AssignmentManager.java:2799)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.assignAllUserRegions(AssignmentManager.java:2778)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:638)
>  
> at 
> org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:485)
>  
> at 
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:723)
>  
> at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) 
> at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1481) 
> at java.lang.Thread.run(Thread.java:745) 
> {code}
> It looks like {{FSTableDescriptors#get(…)}} can be expected to return null in 
> some cases, but {{AssignmentManager.replicaRegionsNotRecordedInMeta(…)}} does 
> not currently have any handling for such a possibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client

[Phil Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phil Yang updated HBASE-16285:
--
Description: 
After HBASE-15593, we have a timeout param in header of RPC requests. We can 
use it in more scenes.
A straightforward scene is to drop requests if it has waited so long in RPC 
queue and has been dropped by client. Even if we handle this request and send 
the response back, it will not be used any more. And client may have sent a 
retry. In an extreme case, if the server is slow, all requests may be timeout 
or queue-full-exception because we should handle previous requests which have 
been dropped by client and many resources at server are wasted.

  was:
After HBASE-15593, we have a timeout param in header of RPC requests. We can 
use it in more scenes.
A straightforward scene is to drop requests if it has waited so long in RPC 
queue and has been dropped by client. Even if we handle this request and send 
the response back, it will not be used any more. And client may have been sent 
a retry. In an extreme case, if the server is slow, all requests may be timeout 
or queue-full-exception because we should handle previous requests which have 
been dropped by client and many resources at server are wasted.


> Drop RPC requests if it must be considered as timeout at client
> ---
>
> Key: HBASE-16285
> URL: https://issues.apache.org/jira/browse/HBASE-16285
> Project: HBase
>  Issue Type: Improvement
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-16285-v1.patch
>
>
> After HBASE-15593, we have a timeout param in header of RPC requests. We can 
> use it in more scenes.
> A straightforward scene is to drop requests if it has waited so long in RPC 
> queue and has been dropped by client. Even if we handle this request and send 
> the response back, it will not be used any more. And client may have sent a 
> retry. In an extreme case, if the server is slow, all requests may be timeout 
> or queue-full-exception because we should handle previous requests which have 
> been dropped by client and many resources at server are wasted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client

[Phil Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phil Yang updated HBASE-16285:
--
Attachment: HBASE-16285-v1.patch

Upload a patch which drops timeout requests at the start of CallRunner.run(). I 
think it is a simple but useful approach. With this patch we can recover much 
faster when a RS has been stuck for several seconds.

And there can be more complex improvement which we can do later, for example, 
aborting a request in the path of reading/writing to reduce the waste of 
resource.

> Drop RPC requests if it must be considered as timeout at client
> ---
>
> Key: HBASE-16285
> URL: https://issues.apache.org/jira/browse/HBASE-16285
> Project: HBase
>  Issue Type: Improvement
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-16285-v1.patch
>
>
> After HBASE-15593, we have a timeout param in header of RPC requests. We can 
> use it in more scenes.
> A straightforward scene is to drop requests if it has waited so long in RPC 
> queue and has been dropped by client. Even if we handle this request and send 
> the response back, it will not be used any more. And client may have been 
> sent a retry. In an extreme case, if the server is slow, all requests may be 
> timeout or queue-full-exception because we should handle previous requests 
> which have been dropped by client and many resources at server are wasted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client

[Phil Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phil Yang updated HBASE-16285:
--
Status: Patch Available  (was: Open)

> Drop RPC requests if it must be considered as timeout at client
> ---
>
> Key: HBASE-16285
> URL: https://issues.apache.org/jira/browse/HBASE-16285
> Project: HBase
>  Issue Type: Improvement
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-16285-v1.patch
>
>
> After HBASE-15593, we have a timeout param in header of RPC requests. We can 
> use it in more scenes.
> A straightforward scene is to drop requests if it has waited so long in RPC 
> queue and has been dropped by client. Even if we handle this request and send 
> the response back, it will not be used any more. And client may have been 
> sent a retry. In an extreme case, if the server is slow, all requests may be 
> timeout or queue-full-exception because we should handle previous requests 
> which have been dropped by client and many resources at server are wasted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16260) Audit dependencies for Category-X

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393646#comment-15393646
 ] 

Sean Busbey commented on HBASE-16260:
-

{quote}
I suggest we move forward with the revert, downgrade this issue from blocker, 
and free up RM's. 
{quote}

+1. we'll need to have a good release note that calls out we're vulnerable to 
whatever web stuff was mitigated. also please JIRA(s) for getting mitigations 
in place without blacklisted dependencies.

{quote}
 I looked briefly at the rat module source code, it appears to be only designed 
to enforce the presence of approved headers in distributed files. There's 
nothing I can find about checking metadata on dependencies. Are we reduced to 
consuming the DEPENDENCIES report mentioned earlier? Maybe Sean Busbey knows 
more voodoo than I...
{quote}

The best I can think of is generating a dependency list of licenses via maven, 
preferably in a way that leverages the supplemental info we already track for 
our generated LICENSE/NOTICE files. I don't know if the DEPENDENCIES file does 
that, but it should be easy enough to check. I can think of how we could make 
the velocity template that makes LICENSE/NOTICE fail if there are only cat-x 
licenses, but I think we've seen how poor the error messaging out of that is.

> Audit dependencies for Category-X
> -
>
> Key: HBASE-16260
> URL: https://issues.apache.org/jira/browse/HBASE-16260
> Project: HBase
>  Issue Type: Task
>  Components: community, dependencies
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 1.1.5, 1.2.2
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Blocker
> Fix For: 2.0.0, 1.1.6, 1.2.3
>
>
> Make sure we do not have category x dependencies.
> right now we atleast have an LGPL for xom:xom (thanks to PHOENIX-3103 for the 
> catch)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16280) Use hash based map in SequenceIdAccounting

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393645#comment-15393645
 ] 

Hudson commented on HBASE-16280:


FAILURE: Integrated in HBase-Trunk_matrix #1298 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/1298/])
HBASE-16280 Use hash based map in SequenceIdAccounting (zhangduo: rev 
bcf409e11f081be077f1232c987d05fa78a1793c)
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/util/ImmutableByteArray.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceIdAccounting.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java


> Use hash based map in SequenceIdAccounting
> --
>
> Key: HBASE-16280
> URL: https://issues.apache.org/jira/browse/HBASE-16280
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>  Labels: performance
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16280-branch-1.patch, HBASE-16280-v1.patch, 
> HBASE-16280.patch
>
>
> Its update method is on the write path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16286) When TagRewriteCell are not copied to MSLAB, deep clone it while adding to Memstore

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393640#comment-15393640
 ] 

ramkrishna.s.vasudevan commented on HBASE-16286:


+1

> When TagRewriteCell are not copied to MSLAB, deep clone it while adding to 
> Memstore
> ---
>
> Key: HBASE-16286
> URL: https://issues.apache.org/jira/browse/HBASE-16286
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16286.patch
>
>
> HBASE-16205 missed to handle the case when we add TagRewriteCell to MSLAB. 
> Here we handle that case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16286) When TagRewriteCell are not copied to MSLAB, deep clone it while adding to Memstore

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393602#comment-15393602
 ] 

Anoop Sam John commented on HBASE-16286:


Ping [~ram_krish], [~carp84]

> When TagRewriteCell are not copied to MSLAB, deep clone it while adding to 
> Memstore
> ---
>
> Key: HBASE-16286
> URL: https://issues.apache.org/jira/browse/HBASE-16286
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16286.patch
>
>
> HBASE-16205 missed to handle the case when we add TagRewriteCell to MSLAB. 
> Here we handle that case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16286) When TagRewriteCell are not copied to MSLAB, deep clone it while adding to Memstore

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-16286:
---
Status: Patch Available  (was: Open)

> When TagRewriteCell are not copied to MSLAB, deep clone it while adding to 
> Memstore
> ---
>
> Key: HBASE-16286
> URL: https://issues.apache.org/jira/browse/HBASE-16286
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16286.patch
>
>
> HBASE-16205 missed to handle the case when we add TagRewriteCell to MSLAB. 
> Here we handle that case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16286) When TagRewriteCell are not copied to MSLAB, deep clone it while adding to Memstore

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-16286:
---
Attachment: HBASE-16286.patch

TagRewriteCell creation moved to CellUtil and also the class.  Added a 
ShareableMemory verison of TagRewriteCell .

> When TagRewriteCell are not copied to MSLAB, deep clone it while adding to 
> Memstore
> ---
>
> Key: HBASE-16286
> URL: https://issues.apache.org/jira/browse/HBASE-16286
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16286.patch
>
>
> HBASE-16205 missed to handle the case when we add TagRewriteCell to MSLAB. 
> Here we handle that case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-16286) When TagRewriteCell are not copied to MSLAB, deep clone it while adding to Memstore

[Anoop Sam John (JIRA)]

Anoop Sam John created HBASE-16286:
--

 Summary: When TagRewriteCell are not copied to MSLAB, deep clone 
it while adding to Memstore
 Key: HBASE-16286
 URL: https://issues.apache.org/jira/browse/HBASE-16286
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.0.0
Reporter: Anoop Sam John
Assignee: Anoop Sam John
 Fix For: 2.0.0


HBASE-16205 missed to handle the case when we add TagRewriteCell to MSLAB. Here 
we handle that case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16283) Batch Append/Increment will always fail if set ReturnResults to false

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393587#comment-15393587
 ] 

Hadoop QA commented on HBASE-16283:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 58m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 10s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
41m 9s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 16s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 
41s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 210m 31s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestImportTsv |
|   | hadoop.hbase.TestPartialResultsFromClientSide |
|   | hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
|   | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
|   | hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort |
|   | hadoop.hbase.regionserver.TestRegionReplicaFailover |
|   | hadoop.hbase.util.TestRegionMover |
|   | hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat |
|   | hadoop.hbase.mapreduce.TestHashTable |
|   | hadoop.hbase.namespace.TestNamespaceAuditor |
|   | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
|   | hadoop.hbase.regionserver.TestRegionServerMetrics |
|   | hadoop.hbase.wal.TestDefaultWALProviderWithHLogKey |
|   | hadoop.hbase.util.TestRegionSplitter |
|   | hadoop.hbase.util.TestMiniClusterLoadEncoded |
| 

[jira] [Updated] (HBASE-16283) Batch Append/Increment will always fail if set ReturnResults to false

[Allan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Yang updated HBASE-16283:
---
Attachment: FailedCase.java

> Batch Append/Increment will always fail if set ReturnResults to false
> -
>
> Key: HBASE-16283
> URL: https://issues.apache.org/jira/browse/HBASE-16283
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 2.0.0, 1.1.5, 1.2.2
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: FailedCase.java, HBASE-16283.patch, HBASE-16283v2.patch
>
>
> If set Append/Increment's ReturnResult attribute to false, and batch the 
> appends/increments to server. The batch operation will always return false.
> The reason is that, since return result is set to false, append/increment 
> will return null instead of Result object. But in 
> ResponseConverter#getResults, there is some check code 
> {code}
> if (requestRegionActionCount != responseRegionActionResultCount) {
>   throw new IllegalStateException("Request mutation count=" + 
> requestRegionActionCount +
>   " does not match response mutation result count=" + 
> responseRegionActionResultCount);
> }
> {code}
> That means if the result count is not meet with request mutation count, it 
> will fail the request.
> The solution is simple, instead of returning a null result, returning a empty 
> result if ReturnResult set to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16283) Batch Append/Increment will always fail if set ReturnResults to false

[Allan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Yang updated HBASE-16283:
---
Attachment: FailedCase.java

> Batch Append/Increment will always fail if set ReturnResults to false
> -
>
> Key: HBASE-16283
> URL: https://issues.apache.org/jira/browse/HBASE-16283
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 2.0.0, 1.1.5, 1.2.2
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: FailedCase.java, HBASE-16283.patch, HBASE-16283v2.patch
>
>
> If set Append/Increment's ReturnResult attribute to false, and batch the 
> appends/increments to server. The batch operation will always return false.
> The reason is that, since return result is set to false, append/increment 
> will return null instead of Result object. But in 
> ResponseConverter#getResults, there is some check code 
> {code}
> if (requestRegionActionCount != responseRegionActionResultCount) {
>   throw new IllegalStateException("Request mutation count=" + 
> requestRegionActionCount +
>   " does not match response mutation result count=" + 
> responseRegionActionResultCount);
> }
> {code}
> That means if the result count is not meet with request mutation count, it 
> will fail the request.
> The solution is simple, instead of returning a null result, returning a empty 
> result if ReturnResult set to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16283) Batch Append/Increment will always fail if set ReturnResults to false

[Allan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Yang updated HBASE-16283:
---
Attachment: (was: FailedCase.java)

> Batch Append/Increment will always fail if set ReturnResults to false
> -
>
> Key: HBASE-16283
> URL: https://issues.apache.org/jira/browse/HBASE-16283
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 2.0.0, 1.1.5, 1.2.2
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: FailedCase.java, HBASE-16283.patch, HBASE-16283v2.patch
>
>
> If set Append/Increment's ReturnResult attribute to false, and batch the 
> appends/increments to server. The batch operation will always return false.
> The reason is that, since return result is set to false, append/increment 
> will return null instead of Result object. But in 
> ResponseConverter#getResults, there is some check code 
> {code}
> if (requestRegionActionCount != responseRegionActionResultCount) {
>   throw new IllegalStateException("Request mutation count=" + 
> requestRegionActionCount +
>   " does not match response mutation result count=" + 
> responseRegionActionResultCount);
> }
> {code}
> That means if the result count is not meet with request mutation count, it 
> will fail the request.
> The solution is simple, instead of returning a null result, returning a empty 
> result if ReturnResult set to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16205) When Cells are not copied to MSLAB, deep clone it while adding to Memstore

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393575#comment-15393575
 ] 

Anoop Sam John commented on HBASE-16205:


Missed handling TagRewriteCell here as part of this jira.  This will come into 
pic when say there are visibility labels in coming Mutation.  At RPC codec 
layer, a ShareableMemory version of KeyValue will be created. Later on in VC, 
we will add tags to the incoming Cell and that in turn will make 
TagRewriteCell. The underlying cell here refers to big memory chunk where we 
read request in RPC layer. I will handle that as part of another jira as the 
patch is bit big and dont want to do that much big change as an addendum. You 
ok with that [~ram_krish], [~carp84]?

> When Cells are not copied to MSLAB, deep clone it while adding to Memstore
> --
>
> Key: HBASE-16205
> URL: https://issues.apache.org/jira/browse/HBASE-16205
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16205.patch, HBASE-16205_V2.patch, 
> HBASE-16205_V3.patch, HBASE-16205_V3.patch
>
>
> This is imp after HBASE-15180 optimization. After that we the cells flowing 
> in write path will be backed by the same byte[] where the RPC read the 
> request into. By default we have MSLAB On and so we have a copy operation 
> while adding Cells to memstore.  This copy might not be there if
> 1. MSLAB is turned OFF
> 2. Cell size is more than a configurable max size. This defaults to 256 KB
> 3. If the operation is Append/Increment. 
> In such cases, we should just clone the Cell into a new byte[] and then add 
> to memstore.  Or else we keep referring to the bigger byte[] chunk for longer 
> time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16283) Batch Append/Increment will always fail if set ReturnResults to false

[Allan Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Allan Yang updated HBASE-16283:
---
Attachment: (was: FailedCase.java)

> Batch Append/Increment will always fail if set ReturnResults to false
> -
>
> Key: HBASE-16283
> URL: https://issues.apache.org/jira/browse/HBASE-16283
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 2.0.0, 1.1.5, 1.2.2
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16283.patch, HBASE-16283v2.patch
>
>
> If set Append/Increment's ReturnResult attribute to false, and batch the 
> appends/increments to server. The batch operation will always return false.
> The reason is that, since return result is set to false, append/increment 
> will return null instead of Result object. But in 
> ResponseConverter#getResults, there is some check code 
> {code}
> if (requestRegionActionCount != responseRegionActionResultCount) {
>   throw new IllegalStateException("Request mutation count=" + 
> requestRegionActionCount +
>   " does not match response mutation result count=" + 
> responseRegionActionResultCount);
> }
> {code}
> That means if the result count is not meet with request mutation count, it 
> will fail the request.
> The solution is simple, instead of returning a null result, returning a empty 
> result if ReturnResult set to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16283) Batch Append/Increment will always fail if set ReturnResults to false

[Allan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393573#comment-15393573
 ] 

Allan Yang commented on HBASE-16283:


Those are two cases just to show the issue. I'm not intend to include these two 
test into the project. Do you think we need to add batch append/increment test 
case?

> Batch Append/Increment will always fail if set ReturnResults to false
> -
>
> Key: HBASE-16283
> URL: https://issues.apache.org/jira/browse/HBASE-16283
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 2.0.0, 1.1.5, 1.2.2
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: FailedCase.java, HBASE-16283.patch, HBASE-16283v2.patch
>
>
> If set Append/Increment's ReturnResult attribute to false, and batch the 
> appends/increments to server. The batch operation will always return false.
> The reason is that, since return result is set to false, append/increment 
> will return null instead of Result object. But in 
> ResponseConverter#getResults, there is some check code 
> {code}
> if (requestRegionActionCount != responseRegionActionResultCount) {
>   throw new IllegalStateException("Request mutation count=" + 
> requestRegionActionCount +
>   " does not match response mutation result count=" + 
> responseRegionActionResultCount);
> }
> {code}
> That means if the result count is not meet with request mutation count, it 
> will fail the request.
> The solution is simple, instead of returning a null result, returning a empty 
> result if ReturnResult set to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15638) Shade protobuf

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393542#comment-15393542
 ] 

Hadoop QA commented on HBASE-15638:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 72 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 3s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 37s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 38s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 33m 
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m 
41s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 29s 
{color} | {color:red} hbase-protocol in master has 93 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s 
{color} | {color:red} hbase-rsgroup in master has 6 extant Findbugs warnings. 
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 54s 
{color} | {color:red} hbase-rest in master has 1 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 7s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 21s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 40s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 21s 
{color} | {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 26s 
{color} | {color:red} hbase-it in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s 
{color} | {color:red} hbase-examples in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 57s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} 
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 37s 
{color} | {color:red} hbase-rsgroup in the patch failed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 40s 
{color} | {color:red} hbase-it in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 29s 
{color} | {color:red} hbase-examples in the patch failed with JDK v1.8.0. 
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 57s {color} 
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 37s {color} 
| {color:red} hbase-rsgroup in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 40s {color} 
| {color:red} hbase-it in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 29s {color} 
| {color:red} hbase-examples in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 47s 
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_80. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 26s 
{color} | {color:red} hbase-rsgroup in the patch failed with JDK v1.7.0_80. 
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 40s 
{color} | {color:red} hbase-it in the patch failed with JDK v1.7.0_80. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 24s 
{color} | {color:red} hbase-examples in 

[jira] [Created] (HBASE-16285) Drop RPC requests if it must be considered as timeout at client

[Phil Yang (JIRA)]

Phil Yang created HBASE-16285:
-

 Summary: Drop RPC requests if it must be considered as timeout at 
client
 Key: HBASE-16285
 URL: https://issues.apache.org/jira/browse/HBASE-16285
 Project: HBase
  Issue Type: Improvement
Reporter: Phil Yang
Assignee: Phil Yang


After HBASE-15593, we have a timeout param in header of RPC requests. We can 
use it in more scenes.
A straightforward scene is to drop requests if it has waited so long in RPC 
queue and has been dropped by client. Even if we handle this request and send 
the response back, it will not be used any more. And client may have been sent 
a retry. In an extreme case, if the server is slow, all requests may be timeout 
or queue-full-exception because we should handle previous requests which have 
been dropped by client and many resources at server are wasted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16229) Cleaning up size and heapSize calculation

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393534#comment-15393534
 ] 

Anastasia Braginsky commented on HBASE-16229:
-

Thank you very much!

> Cleaning up size and heapSize calculation
> -
>
> Key: HBASE-16229
> URL: https://issues.apache.org/jira/browse/HBASE-16229
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16229.patch, HBASE-16229_V2.patch, 
> HBASE-16229_V3.patch
>
>
> It is bit ugly now. For eg:
> AbstractMemStore
> {code}
> public final static long FIXED_OVERHEAD = ClassSize.align(
>   ClassSize.OBJECT +
>   (4 * ClassSize.REFERENCE) +
>   (2 * Bytes.SIZEOF_LONG));
>   public final static long DEEP_OVERHEAD = ClassSize.align(FIXED_OVERHEAD +
>   (ClassSize.ATOMIC_LONG + ClassSize.TIMERANGE_TRACKER +
>   ClassSize.CELL_SKIPLIST_SET + ClassSize.CONCURRENT_SKIPLISTMAP));
> {code}
> We include the heap overhead of Segment also here. It will be better the 
> Segment contains its overhead part and the Memstore impl uses the heap size 
> of all of its segments to calculate its size.
> Also this
> {code}
> public long heapSize() {
> return getActive().getSize();
>   }
> {code}
> HeapSize to consider all segment's size not just active's. I am not able to 
> see an override method in CompactingMemstore.
> This jira tries to solve some of these.
> When we create a Segment, we seems pass some initial heap size value to it. 
> Why?  The segment object internally has to know what is its heap size not 
> like some one else dictate it.
> More to add when doing this cleanup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16283) Batch Append/Increment will always fail if set ReturnResults to false

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393530#comment-15393530
 ] 

Ted Yu commented on HBASE-16283:


Can you fold TestWebUI into existing unit test ?

Remember adding proper license header.

> Batch Append/Increment will always fail if set ReturnResults to false
> -
>
> Key: HBASE-16283
> URL: https://issues.apache.org/jira/browse/HBASE-16283
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 2.0.0, 1.1.5, 1.2.2
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: FailedCase.java, HBASE-16283.patch, HBASE-16283v2.patch
>
>
> If set Append/Increment's ReturnResult attribute to false, and batch the 
> appends/increments to server. The batch operation will always return false.
> The reason is that, since return result is set to false, append/increment 
> will return null instead of Result object. But in 
> ResponseConverter#getResults, there is some check code 
> {code}
> if (requestRegionActionCount != responseRegionActionResultCount) {
>   throw new IllegalStateException("Request mutation count=" + 
> requestRegionActionCount +
>   " does not match response mutation result count=" + 
> responseRegionActionResultCount);
> }
> {code}
> That means if the result count is not meet with request mutation count, it 
> will fail the request.
> The solution is simple, instead of returning a null result, returning a empty 
> result if ReturnResult set to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16229) Cleaning up size and heapSize calculation

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393527#comment-15393527
 ] 

Anoop Sam John commented on HBASE-16229:


very much ok. will get 14921 committed first.  

> Cleaning up size and heapSize calculation
> -
>
> Key: HBASE-16229
> URL: https://issues.apache.org/jira/browse/HBASE-16229
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16229.patch, HBASE-16229_V2.patch, 
> HBASE-16229_V3.patch
>
>
> It is bit ugly now. For eg:
> AbstractMemStore
> {code}
> public final static long FIXED_OVERHEAD = ClassSize.align(
>   ClassSize.OBJECT +
>   (4 * ClassSize.REFERENCE) +
>   (2 * Bytes.SIZEOF_LONG));
>   public final static long DEEP_OVERHEAD = ClassSize.align(FIXED_OVERHEAD +
>   (ClassSize.ATOMIC_LONG + ClassSize.TIMERANGE_TRACKER +
>   ClassSize.CELL_SKIPLIST_SET + ClassSize.CONCURRENT_SKIPLISTMAP));
> {code}
> We include the heap overhead of Segment also here. It will be better the 
> Segment contains its overhead part and the Memstore impl uses the heap size 
> of all of its segments to calculate its size.
> Also this
> {code}
> public long heapSize() {
> return getActive().getSize();
>   }
> {code}
> HeapSize to consider all segment's size not just active's. I am not able to 
> see an override method in CompactingMemstore.
> This jira tries to solve some of these.
> When we create a Segment, we seems pass some initial heap size value to it. 
> Why?  The segment object internally has to know what is its heap size not 
> like some one else dictate it.
> More to add when doing this cleanup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16229) Cleaning up size and heapSize calculation

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393519#comment-15393519
 ] 

Anastasia Braginsky commented on HBASE-16229:
-

So, based on all the comments in HBASE-14921, is it OK to wait with this JIRA 
till 14921 is committed?

> Cleaning up size and heapSize calculation
> -
>
> Key: HBASE-16229
> URL: https://issues.apache.org/jira/browse/HBASE-16229
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16229.patch, HBASE-16229_V2.patch, 
> HBASE-16229_V3.patch
>
>
> It is bit ugly now. For eg:
> AbstractMemStore
> {code}
> public final static long FIXED_OVERHEAD = ClassSize.align(
>   ClassSize.OBJECT +
>   (4 * ClassSize.REFERENCE) +
>   (2 * Bytes.SIZEOF_LONG));
>   public final static long DEEP_OVERHEAD = ClassSize.align(FIXED_OVERHEAD +
>   (ClassSize.ATOMIC_LONG + ClassSize.TIMERANGE_TRACKER +
>   ClassSize.CELL_SKIPLIST_SET + ClassSize.CONCURRENT_SKIPLISTMAP));
> {code}
> We include the heap overhead of Segment also here. It will be better the 
> Segment contains its overhead part and the Memstore impl uses the heap size 
> of all of its segments to calculate its size.
> Also this
> {code}
> public long heapSize() {
> return getActive().getSize();
>   }
> {code}
> HeapSize to consider all segment's size not just active's. I am not able to 
> see an override method in CompactingMemstore.
> This jira tries to solve some of these.
> When we create a Segment, we seems pass some initial heap size value to it. 
> Why?  The segment object internally has to know what is its heap size not 
> like some one else dictate it.
> More to add when doing this cleanup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15721) Optimization in cloning cells into MSLAB

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393491#comment-15393491
 ] 

Hadoop QA commented on HBASE-15721:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color} 
| {color:red} HBASE-15721 does not apply to master. Rebase required? Wrong 
Branch? See https://yetus.apache.org/documentation/0.2.1/precommit-patchnames 
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12801386/HBASE-15721_V2.patch |
| JIRA Issue | HBASE-15721 |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/2769/console |
| Powered by | Apache Yetus 0.2.1   http://yetus.apache.org |


This message was automatically generated.



> Optimization in cloning cells into MSLAB
> 
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch, HBASE-15721_V2.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying 
> it to MSLAB chunk area.  This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output, 
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
>   pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
>   pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
>   }
> {code}
> Copied in 9 steps and we end up parsing all lengths.  When the cell 
> implementation is backed by a single byte[] (Like KeyValue) this can be done 
> in single step copy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15721) Optimization in cloning cells into MSLAB

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393489#comment-15393489
 ] 

Anoop Sam John commented on HBASE-15721:


[~saint@gmail.com]  Any objection in making the above scope change for 
MSLAB?  Than just allocate bytes, it will do allocation + copy and recreation 
of the cell.
This is required in the CompactingMemstore flatten to CellChunkMap. In this 
flattened form, we wont be having Cell objects. We will keep cell's meta data 
that it is in which chunk and at what offset.  Normally from a KeyValue we wont 
get any info like chunkId.  So if we allow this change, then the MSLAB impl can 
create a new Cell impl which keeps the chunkId also.
I need to rework on the patch any way.  Now there is a way not to add write() 
to Streamable

> Optimization in cloning cells into MSLAB
> 
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
>  Issue Type: Sub-task
>  Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch, HBASE-15721_V2.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying 
> it to MSLAB chunk area.  This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output, 
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
>   pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
>   pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
>   }
> {code}
> Copied in 9 steps and we end up parsing all lengths.  When the cell 
> implementation is backed by a single byte[] (Like KeyValue) this can be done 
> in single step copy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9465) Push entries to peer clusters serially

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393481#comment-15393481
 ] 

Hadoop QA commented on HBASE-9465:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 36s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 11m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 46s 
{color} | {color:red} hbase-protocol in master has 93 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s 
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s 
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 42s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s 
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 12m 
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
4s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
55m 30s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 11m 
53s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m 31s 
{color} | {color:red} hbase-client-jdk1.8.0 with JDK v1.8.0 generated 2 new + 
13 unchanged - 0 fixed = 15 total (was 13) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 33s 
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 8m 45s 
{color} | {color:red} hbase-client-jdk1.7.0_80 with JDK v1.7.0_80 generated 2 
new + 13 unchanged - 0 fixed = 15 total (was 13) {color} |
| {color:green}+1{color} | {color:green} 

[jira] [Commented] (HBASE-16281) TestMasterReplication is flaky

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393477#comment-15393477
 ] 

Hudson commented on HBASE-16281:


SUCCESS: Integrated in HBase-1.3-IT #765 (See 
[https://builds.apache.org/job/HBase-1.3-IT/765/])
HBASE-16281 TestMasterReplication is flaky (zhangduo: rev 
7214bad10b19ad1ff36bc80c659fa2ce065cc149)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java


> TestMasterReplication is flaky
> --
>
> Key: HBASE-16281
> URL: https://issues.apache.org/jira/browse/HBASE-16281
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.2, 0.98.20, 1.1.6
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21, 1.2.3, 1.1.7
>
> Attachments: HBASE-16281-v1.patch, HBASE-16281-v1.patch, 
> HBASE-16281-v1.patch
>
>
> In TestMasterReplication we put some mutations and wait until we can read the 
> data from slave cluster. However the waiting time is too short. Replication 
> service in slave cluster may not be initialized and ready to handle 
> replication RPC requests in several seconds. 
> We should wait for more time.
> {quote}
> 2016-07-25 11:47:03,156 WARN  [Time-limited 
> test-EventThread.replicationSource,1.replicationSource.10.235.114.28%2C56313%2C1469418386448,1]
>  regionserver.HBaseInterClusterReplicationEndpoint(310): Can't replicate 
> because of a local or network error: 
> java.io.IOException: java.io.IOException: Replication services are not 
> initialized yet
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2263)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> Caused by: com.google.protobuf.ServiceException: Replication services are not 
> initialized yet
>   at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.replicateWALEntry(RSRpcServices.java:1935)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22751)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2212)
>   ... 3 more
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16280) Use hash based map in SequenceIdAccounting

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-16280:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to master and branch-1.

Thanks [~stack] for reviewing.

> Use hash based map in SequenceIdAccounting
> --
>
> Key: HBASE-16280
> URL: https://issues.apache.org/jira/browse/HBASE-16280
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>  Labels: performance
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16280-branch-1.patch, HBASE-16280-v1.patch, 
> HBASE-16280.patch
>
>
> Its update method is on the write path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16283) Batch Append/Increment will always fail if set ReturnResults to false

[Allan Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393430#comment-15393430
 ] 

Allan Yang commented on HBASE-16283:


you mean the patch? I don't think so, since the patch only modify the server 
side to return a empty result other than a null object.

> Batch Append/Increment will always fail if set ReturnResults to false
> -
>
> Key: HBASE-16283
> URL: https://issues.apache.org/jira/browse/HBASE-16283
> Project: HBase
>  Issue Type: Bug
>  Components: API
>Affects Versions: 2.0.0, 1.1.5, 1.2.2
>Reporter: Allan Yang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: FailedCase.java, HBASE-16283.patch, HBASE-16283v2.patch
>
>
> If set Append/Increment's ReturnResult attribute to false, and batch the 
> appends/increments to server. The batch operation will always return false.
> The reason is that, since return result is set to false, append/increment 
> will return null instead of Result object. But in 
> ResponseConverter#getResults, there is some check code 
> {code}
> if (requestRegionActionCount != responseRegionActionResultCount) {
>   throw new IllegalStateException("Request mutation count=" + 
> requestRegionActionCount +
>   " does not match response mutation result count=" + 
> responseRegionActionResultCount);
> }
> {code}
> That means if the result count is not meet with request mutation count, it 
> will fail the request.
> The solution is simple, instead of returning a null result, returning a empty 
> result if ReturnResult set to false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16280) Use hash based map in SequenceIdAccounting

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-16280:
--
Affects Version/s: 1.4.0
   2.0.0
   Labels: performance  (was: )
Fix Version/s: 1.4.0
   2.0.0
  Component/s: wal

> Use hash based map in SequenceIdAccounting
> --
>
> Key: HBASE-16280
> URL: https://issues.apache.org/jira/browse/HBASE-16280
> Project: HBase
>  Issue Type: Sub-task
>  Components: wal
>Affects Versions: 2.0.0, 1.4.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>  Labels: performance
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16280-branch-1.patch, HBASE-16280-v1.patch, 
> HBASE-16280.patch
>
>
> Its update method is on the write path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)