[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779864#comment-15779864
 ] 

Hadoop QA commented on HBASE-14123:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 5s 
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 23 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 28s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 
10s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s 
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 10s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s 
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s 
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 43s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m 
38s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s 
{color} | {color:red} hbase-server generated 2 new + 0 unchanged - 0 fixed = 2 
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 29s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 44s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s 
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s 

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779861#comment-15779861
 ] 

Chang chen commented on HBASE-17375:


[~ram_krish] i uploaded a patch ,please review

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: HBASE_17375_v1.patch, row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chang chen updated HBASE-17375:
---
Attachment: HBASE_17375_v1.patch

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: HBASE_17375_v1.patch, row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779840#comment-15779840
 ] 

Guanghao Zhang commented on HBASE-17336:


The class ReplicationAdmin has been marked as Deprecated. Thanks.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch, HBASE-17336-v4.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17336) get/update replication peer config requests should be routed through master

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-17336:
---
Attachment: HBASE-17336-v4.patch

Attach a v4 patch addressed review comments.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch, HBASE-17336-v4.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779832#comment-15779832
 ] 

ramkrishna.s.vasudevan commented on HBASE-17375:


So what is your feedback on Prefix tree other than stableness. Are you doing 
some POC now with it? What was your comparison with other encoding algos? You 
are getting better perf with random access? I think your feedback is important 
so that if really users find it interesting, then as a community, it is bound 
to us to make it stable so that users could use it efficiently.
Thanks [~baibaichen].
I could help here but I just forgot what is there inside prefix. So may need 
some time to get back to it.

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779814#comment-15779814
 ] 

ramkrishna.s.vasudevan commented on HBASE-16421:


Attaching a one min profiler to show the garbage created out of Cell object 
from chunk map. The major ~900MB one is in the flush path and the remaining is 
from the scan path.

> Introducing the CellChunkMap as a new additional index variant in the MemStore
> --
>
> Key: HBASE-16421
> URL: https://issues.apache.org/jira/browse/HBASE-16421
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Anastasia Braginsky
> Attachments: CellChunkMapRevived.pdf, ChunkCell_creation.png, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Follow up for HBASE-14921. This is going to be the umbrella JIRA to include 
> all the parts of integration of the CellChunkMap to the MemStore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779815#comment-15779815
 ] 

Liu Junhong commented on HBASE-17374:
-

Thank you for your prompt.
I will try it.

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1279)) - Closed 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red}  2016-12-08 

[jira] [Updated] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-16421:
---
Attachment: ChunkCell_creation.png

> Introducing the CellChunkMap as a new additional index variant in the MemStore
> --
>
> Key: HBASE-16421
> URL: https://issues.apache.org/jira/browse/HBASE-16421
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Anastasia Braginsky
> Attachments: CellChunkMapRevived.pdf, ChunkCell_creation.png, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Follow up for HBASE-14921. This is going to be the umbrella JIRA to include 
> all the parts of integration of the CellChunkMap to the MemStore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779806#comment-15779806
 ] 

ramkrishna.s.vasudevan commented on HBASE-16421:


Another point to be noted - the tests were run with SKIP_WAL.

> Introducing the CellChunkMap as a new additional index variant in the MemStore
> --
>
> Key: HBASE-16421
> URL: https://issues.apache.org/jira/browse/HBASE-16421
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Anastasia Braginsky
> Attachments: CellChunkMapRevived.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Follow up for HBASE-14921. This is going to be the umbrella JIRA to include 
> all the parts of integration of the CellChunkMap to the MemStore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779804#comment-15779804
 ] 

ramkrishna.s.vasudevan commented on HBASE-16421:


I did some experiments. This was my set up:
I had one single node cluster and with YCSB as the test framework.
In order to test scan perf I had to do mixed read and write work load.  So 
created two client instances one which was doing ycsb load with 100 threads 
(recordcount=1000, operationcount=75000).
Another instance of the ycsb client was performing pure scans with 50 threads.  
(this imitates the read/write workload with totally 150 threads)
The load phase used to run for about 16 mins and during this time the GC graphs 
were plotted (We have G1GC configured). 
In YCSB it is not possible to mention the end key so scans are always never 
ending most of the time. Hence I went with this approach. 
>From the 'jmc profiler' I could see that flushes were creating lot of garbage 
>with chunk map as every time we were creating a Cell and it was quite lot of 
>small objects. 
Similarly the scan trace also created cell objects from the chunks. So overall 
with CellChunkmap and read write mode we are generating more garbage (in terms 
of number) but the overall time is not significantly large.
The numbers are as follows

||Type of memstore||Number of pauses||Total time||
|Array map based memstore|566|46.36s|
|Chunk map based memstore |899|51.56s|
|Default memstore (no MSLAB and chunkpool) |446|37.87s|
|Offheap memstore |469|29.8s|

LEt me know if we need to do some more tests or some other methodology to be 
adopted here. But I think with mixed read/write load - yes we do generate more 
garbage (than the pure write). So overall chunk map reduces the GC overhead (I 
can see that the mixed and young GC avg is the lowest among the above) but 
since we have more small objects we have more count. So we can still pursue 
with this CellChunkMap? Thoughts!!!


> Introducing the CellChunkMap as a new additional index variant in the MemStore
> --
>
> Key: HBASE-16421
> URL: https://issues.apache.org/jira/browse/HBASE-16421
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Anastasia Braginsky
> Attachments: CellChunkMapRevived.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Follow up for HBASE-14921. This is going to be the umbrella JIRA to include 
> all the parts of integration of the CellChunkMap to the MemStore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17320:
--
Attachment: HBASE-17320-v4.patch

Fix javadoc issues.

> Add inclusive/exclusive support for startRow and endRow of scan
> ---
>
> Key: HBASE-17320
> URL: https://issues.apache.org/jira/browse/HBASE-17320
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, scan
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17320-v1.patch, HBASE-17320-v2.patch, 
> HBASE-17320-v3.patch, HBASE-17320-v4.patch, HBASE-17320.patch
>
>
> This is especially useful when doing reverse scan. HBASE-17168 maybe a more 
> powerful solution but we need to be careful about the atomicity, and I do not 
> think we will provide the feature to end user. But I think it is OK to 
> provide inclusive/exclusive option to end user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779779#comment-15779779
 ] 

ramkrishna.s.vasudevan commented on HBASE-16421:


One change that could be done here
bq.+  private Map chunkIdMap =
+  new ConcurrentHashMap();
This is creating lot of garbage due to unboxing. We could use the ObjectIntPair 
instead.

> Introducing the CellChunkMap as a new additional index variant in the MemStore
> --
>
> Key: HBASE-16421
> URL: https://issues.apache.org/jira/browse/HBASE-16421
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Anastasia Braginsky
> Attachments: CellChunkMapRevived.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Follow up for HBASE-14921. This is going to be the umbrella JIRA to include 
> all the parts of integration of the CellChunkMap to the MemStore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779712#comment-15779712
 ] 

Ashish Singhi commented on HBASE-17336:
---

I have left some comments on the review board.
It would be good if you can add deprecate tag on the ReplicationAdmin methods 
and suggest devs to use which method of Admin class. May be you can do this in 
another sub-task.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16438) Create a cell type so that chunk id is embedded in it

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-16438:
---
Attachment: MemstoreChunkCell_trunk.patch

This may not apply cleanly now. But this is the patch that allows to create a 
chunk id and also use a new cell variant with chunk id embedded in it.


> Create a cell type so that chunk id is embedded in it
> -
>
> Key: HBASE-16438
> URL: https://issues.apache.org/jira/browse/HBASE-16438
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Attachments: MemstoreChunkCell_trunk.patch
>
>
> For CellChunkMap we may need a cell such that the chunk out of which it was 
> created, the id of the chunk be embedded in it so that when doing flattening 
> we can use the chunk id as a meta data. More details will follow once the 
> initial tasks are completed. 
> Why we need to embed the chunkid in the Cell is described by [~anastas] in 
> this remark over in parent issue 
> https://issues.apache.org/jira/browse/HBASE-14921?focusedCommentId=15244119=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15244119



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779699#comment-15779699
 ] 

Hadoop QA commented on HBASE-17320:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s 
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 37s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 
49s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s 
{color} | {color:red} hbase-server generated 1 new + 1 unchanged - 0 fixed = 2 
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 41s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s 
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s 
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 39s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 
2s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 153m 0s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844719/HBASE-17320-v3.patch |
| 

[jira] [Commented] (HBASE-17174) Refactor the AsyncProcess, BufferedMutatorImpl, and HTable

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779696#comment-15779696
 ] 

Duo Zhang commented on HBASE-17174:
---

Pushed to master. Thanks for fix.

> Refactor the AsyncProcess, BufferedMutatorImpl, and HTable
> --
>
> Key: HBASE-17174
> URL: https://issues.apache.org/jira/browse/HBASE-17174
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17174.addendum.patch, HBASE-17174.v0.patch, 
> HBASE-17174.v1.patch, HBASE-17174.v10.patch, HBASE-17174.v11.patch, 
> HBASE-17174.v11.patch, HBASE-17174.v11.patch, HBASE-17174.v12.patch, 
> HBASE-17174.v13.patch, HBASE-17174.v2.patch, HBASE-17174.v3.patch, 
> HBASE-17174.v4.patch, HBASE-17174.v5.patch, HBASE-17174.v6.patch, 
> HBASE-17174.v7.patch, HBASE-17174.v8.patch, HBASE-17174.v9.patch
>
>
> The following are reasons of refactoring.
> # A update-heavy application, for example, loader, creates many 
> BufferedMutator for batch updates. But these BufferedMutators can’t share a 
> large threadpool because the shutdown() method will be called when closing 
> any BufferedMutator. This patch adds a flag into BufferedMutatorParams for 
> preventing calling the shutdown() method in BufferedMutatorImpl#close
> # The AsyncProcess has the powerful traffic control, but the control is 
> against the single table currently. We should allow alternate traffic control 
> implementation for advanced user who want more control.
> All suggestions are welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779685#comment-15779685
 ] 

Duo Zhang commented on HBASE-17375:
---

Oh sorry, it means IntegrationTestBigLinkedList. It is in the hbase-it module. 
Usually we use this integration test to verify the stability of HBase.

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Unable to read namespace in HBASE.

[sandeep vura]

Hi Team,

I have given "R" permission from hbase shell to namespace "d1" to user
(svura) and group (hadoopdev).

when i logged in as user(svura) to hbase shell . I am unable to find
namespace (d1) since i had given "R" permission.

Below Commands executed as hbase admin:

1. list_namespace from hbase user. (It will list all namespace)
2. grant 'svura','R','@d1'  ( permission to user)
3. grant '@hadoopdev','R','@d1' ( permission to group )
3. Logged in as svura in other terminal listing namespace shows empty.

Note: Even after grant 'RWXC' Also not listing namespace  user(svura). Only
if i give 'RWXCA' to namespace it working.

Please let me know what i am missing here.

Expected solution:

user should only have read permission to namespace(d1) .Should show the
namespace to that user when only grant R permission,

Regards,
San

[jira] [Commented] (HBASE-17291) Remove ImmutableSegment#getKeyValueScanner

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779631#comment-15779631
 ] 

ramkrishna.s.vasudevan commented on HBASE-17291:


I shall commit this patch too then as HBASE-17081 is in now.

> Remove ImmutableSegment#getKeyValueScanner
> --
>
> Key: HBASE-17291
> URL: https://issues.apache.org/jira/browse/HBASE-17291
> Project: HBase
>  Issue Type: Improvement
>  Components: Scanners
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17291.patch, HBASE-17291_1.patch, 
> HBASE-17291_2.patch
>
>
> This is based on a discussion over [~anastas]'s patch. The MemstoreSnapshot 
> uses a KeyValueScanner which actually seems redundant considering we already 
> have a SegmentScanner. The idea is that the snapshot scanner should be a 
> simple iterator type of scanner but it lacks the capability to do the 
> reference counting on that segment that is now used in snapshot. With 
> snapshot having mulitple segments in the latest impl it is better we hold on 
> to the segment by doing ref counting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17291) Remove ImmutableSegment#getKeyValueScanner

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779632#comment-15779632
 ] 

ramkrishna.s.vasudevan commented on HBASE-17291:


Will wait for +1s though - Ping [~anoopsamjohn].

> Remove ImmutableSegment#getKeyValueScanner
> --
>
> Key: HBASE-17291
> URL: https://issues.apache.org/jira/browse/HBASE-17291
> Project: HBase
>  Issue Type: Improvement
>  Components: Scanners
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17291.patch, HBASE-17291_1.patch, 
> HBASE-17291_2.patch
>
>
> This is based on a discussion over [~anastas]'s patch. The MemstoreSnapshot 
> uses a KeyValueScanner which actually seems redundant considering we already 
> have a SegmentScanner. The idea is that the snapshot scanner should be a 
> simple iterator type of scanner but it lacks the capability to do the 
> reference counting on that segment that is now used in snapshot. With 
> snapshot having mulitple segments in the latest impl it is better we hold on 
> to the segment by doing ref counting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-17081:
---
Fix Version/s: 2.0.0

The QA build after the commit looks green. So all fine now I believe.

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Fix For: 2.0.0
>
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779536#comment-15779536
 ] 

Chang chen commented on HBASE-17375:


sorry, what is the meaning of ITBLL?

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779537#comment-15779537
 ] 

Chang chen commented on HBASE-17375:


sorry, what is the meaning of ITBLL?

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chang chen updated HBASE-17375:
---
Comment: was deleted

(was: sorry, what is the meaning of ITBLL?)

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17174) Refactor the AsyncProcess, BufferedMutatorImpl, and HTable

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17174:
--
Attachment: HBASE-17174.addendum.patch

[~Apache9]
There is a spelling error on the public interface. I have attached the 
addendum. Would you please commit it? Thanks.

> Refactor the AsyncProcess, BufferedMutatorImpl, and HTable
> --
>
> Key: HBASE-17174
> URL: https://issues.apache.org/jira/browse/HBASE-17174
> Project: HBase
>  Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17174.addendum.patch, HBASE-17174.v0.patch, 
> HBASE-17174.v1.patch, HBASE-17174.v10.patch, HBASE-17174.v11.patch, 
> HBASE-17174.v11.patch, HBASE-17174.v11.patch, HBASE-17174.v12.patch, 
> HBASE-17174.v13.patch, HBASE-17174.v2.patch, HBASE-17174.v3.patch, 
> HBASE-17174.v4.patch, HBASE-17174.v5.patch, HBASE-17174.v6.patch, 
> HBASE-17174.v7.patch, HBASE-17174.v8.patch, HBASE-17174.v9.patch
>
>
> The following are reasons of refactoring.
> # A update-heavy application, for example, loader, creates many 
> BufferedMutator for batch updates. But these BufferedMutators can’t share a 
> large threadpool because the shutdown() method will be called when closing 
> any BufferedMutator. This patch adds a flag into BufferedMutatorParams for 
> preventing calling the shutdown() method in BufferedMutatorImpl#close
> # The AsyncProcess has the powerful traffic control, but the control is 
> against the single table currently. We should allow alternate traffic control 
> implementation for advanced user who want more control.
> All suggestions are welcome.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17376) ClientAsyncPrefetchScanner may fail due to too many rows

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779527#comment-15779527
 ] 

Hudson commented on HBASE-17376:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2205 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2205/])
HBASE-17376 ClientAsyncPrefetchScanner may fail due to too many rows (tedyu: 
rev e18e9a22daf32306f966641ea02a72fca96dee32)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientAsyncPrefetchScanner.java


> ClientAsyncPrefetchScanner may fail due to too many rows
> 
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE.17376.v0.patch, HBASE.17376.v1.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779515#comment-15779515
 ] 

Duo Zhang commented on HBASE-17375:
---

Unfortunately it is not widely used so no one can confirm that it is stable 
enough...
I've been using it once, and fixed a bunch of bugs with [~ramkrishna] to make 
it pass the ITBLL. But seems ITBLL does not test reverse scan so...

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779498#comment-15779498
 ] 

Chang chen commented on HBASE-17375:


oh my, I didn't see this on [http://hbase.apache.org/book.html#compression]

{quote}
Prefix Tree
Prefix tree encoding was introduced as an experimental feature in HBase 0.96. 
It provides similar memory savings to the Prefix, Diff, and Fast Diff encoder, 
but provides faster random access at a cost of slower encoding speed.
{quote}

We just want to use random access feature of Prefix Tree. Besides stableness 
and slow encoding speed, is there other issues for Prefix tree?

Actually for stableness, I thought it should be stable, since it was introduced 
3 years ago. 

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17320:
--
Attachment: HBASE-17320-v3.patch

Finally I decide to keep the methods compatible with the old behavior and mark 
them as deprecated. Add new methods withStartRow/withStopRow. Add default 
implementation for the new methods introduced in Store interface.

> Add inclusive/exclusive support for startRow and endRow of scan
> ---
>
> Key: HBASE-17320
> URL: https://issues.apache.org/jira/browse/HBASE-17320
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, scan
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17320-v1.patch, HBASE-17320-v2.patch, 
> HBASE-17320-v3.patch, HBASE-17320.patch
>
>
> This is especially useful when doing reverse scan. HBASE-17168 maybe a more 
> powerful solution but we need to be careful about the atomicity, and I do not 
> think we will provide the feature to end user. But I think it is OK to 
> provide inclusive/exclusive option to end user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2

[Vladimir Rodionov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779455#comment-15779455
 ] 

Vladimir Rodionov commented on HBASE-14123:
---

Updated RB:
https://reviews.apache.org/r/52748/

cc: [~saint@gmail.com], [~devaraj], [~te...@apache.org]

> HBase Backup/Restore Phase 2
> 
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 14123-master.v14.txt, 14123-master.v15.txt, 
> 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, 
> 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, 
> 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, 
> 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, 
> 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, 
> 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, 
> 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, 
> 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, 
> 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, 
> 14123.master.v39.patch, 14123.master.v40.patch, 14123.master.v41.patch, 
> 14123.master.v42.patch, 14123.master.v44.patch, 14123.master.v45.patch, 
> HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, 
> HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, 
> HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, 
> HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, 
> HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, 
> HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779452#comment-15779452
 ] 

Guanghao Zhang commented on HBASE-17336:


[~enis] [~ashish singhi] Any concerns?

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14123:
--
Attachment: 14123.master.v45.patch

v45. Changes

# Reverted v44 refactoring into separate module back to v42
# Addressed all the recent comments on review board

> HBase Backup/Restore Phase 2
> 
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Fix For: 2.0.0
>
> Attachments: 14123-master.v14.txt, 14123-master.v15.txt, 
> 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, 
> 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, 
> 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, 
> 14123-master.v27.txt, 14123-master.v28.txt, 14123-master.v29.full.txt, 
> 14123-master.v3.txt, 14123-master.v30.txt, 14123-master.v31.txt, 
> 14123-master.v32.txt, 14123-master.v33.txt, 14123-master.v34.txt, 
> 14123-master.v35.txt, 14123-master.v36.txt, 14123-master.v37.txt, 
> 14123-master.v38.txt, 14123-master.v5.txt, 14123-master.v6.txt, 
> 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, 
> 14123.master.v39.patch, 14123.master.v40.patch, 14123.master.v41.patch, 
> 14123.master.v42.patch, 14123.master.v44.patch, 14123.master.v45.patch, 
> HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, 
> HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, 
> HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, 
> HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, 
> HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, 
> HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17288) Add warn log for huge Cell and huge row

[Guanghao Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779364#comment-15779364
 ] 

Guanghao Zhang commented on HBASE-17288:


ping [~anoop.hbase]

> Add warn log for huge Cell and huge row
> ---
>
> Key: HBASE-17288
> URL: https://issues.apache.org/jira/browse/HBASE-17288
> Project: HBase
>  Issue Type: Improvement
>  Components: scan
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-17288-v1.patch, HBASE-17288-v2.patch, 
> HBASE-17288-v3.patch, HBASE-17288.patch
>
>
> Some log examples from our production cluster.
> {code}
> 2016-12-10,17:08:11,478 WARN 
> org.apache.hadoop.hbase.regionserver.StoreScanner: adding a HUGE KV into 
> result list, kv size:1253360, 
> kv:10567114001-1-c/R:r1/1481360887152/Put/vlen=1253245/ts=923099, from 
> table X
> 2016-12-10,17:08:16,724 WARN 
> org.apache.hadoop.hbase.regionserver.StoreScanner: adding a HUGE KV into 
> result list, kv size:1048680, 
> kv:0220459/I:i_0/1481360889551/Put/vlen=1048576/ts=13642, from table XX
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17376) ClientAsyncPrefetchScanner may fail due to too many rows

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17376:
---
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Thanks for the patch, ChiaPing

> ClientAsyncPrefetchScanner may fail due to too many rows
> 
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE.17376.v0.patch, HBASE.17376.v1.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17376) ClientAsyncPrefetchScanner may fail due to too many rows

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17376:
---
Summary: ClientAsyncPrefetchScanner may fail due to too many rows  (was: 
ClientAsyncPrefetchScanner fails due to too many rows)

> ClientAsyncPrefetchScanner may fail due to too many rows
> 
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE.17376.v0.patch, HBASE.17376.v1.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16010) Put draining function through Admin API

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779161#comment-15779161
 ] 

Jerry He commented on HBASE-16010:
--

I think it should require global admin, similar to the 'balance' operation.

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch, 
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16010) Put draining function through Admin API

[Matt Warhaftig (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778964#comment-15778964
 ] 

Matt Warhaftig commented on HBASE-16010:


{quote}
"Matt Warhaftig, would you like to work on the ACL task?"
{quote}

Would ACL just entail requiring that the user doing the draining functionality 
has admin (table scope) access.

> Put draining function through Admin API
> ---
>
> Key: HBASE-16010
> URL: https://issues.apache.org/jira/browse/HBASE-16010
> Project: HBase
>  Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Matt Warhaftig
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16010-v3.patch, hbase-16010-v1.patch, 
> hbase-16010-v2.patch
>
>
> Currently, there is no Amdin API for draining function. Client has to 
> interact directly with Zookeeper draining node to add and remove draining 
> servers.
> For example, in draining_servers.rb:
> {code}
>   zkw = org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.new(config, 
> "draining_servers", nil)
>   parentZnode = zkw.drainingZNode
>   begin
> for server in servers
>   node = ZKUtil.joinZNode(parentZnode, server)
>   ZKUtil.createAndFailSilent(zkw, node)
> end
>   ensure
> zkw.close()
>   end
> {code}
> This is not good in cases like secure clusters with protected Zookeeper nodes.
> Let's put draining function through Admin API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778880#comment-15778880
 ] 

Hudson commented on HBASE-17081:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2203 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2203/])
HBASE-17081 [Recommit]Flush the entire CompactingMemStore content to 
(ramkrishna: rev 463ffa792a23799d8cf2406321d1c8a3acacded1)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionPipeline.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentFactory.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemstoreSize.java
* (add) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompositeImmutableSegment.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactor.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactingMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWalAndCompactingMemStoreFlush.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMemStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java


> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778824#comment-15778824
 ] 

Hadoop QA commented on HBASE-17336:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s 
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 0s 
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 3s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 17m 
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
54s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s 
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 51s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 17m 
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
54s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 52 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 17s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m 
5s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 52s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s 
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 22s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 9s 
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 110m 53s 
{color} | {color:green} root in the patch passed. {color} 

[jira] [Commented] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778764#comment-15778764
 ] 

Hadoop QA commented on HBASE-17376:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 23s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 9s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
25s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 2s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844693/HBASE.17376.v1.patch |
| JIRA Issue | HBASE-17376 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 843817e56ffd 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 8fa5b0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5050/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5050/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> ClientAsyncPrefetchScanner fails due to too many 

[jira] [Updated] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-17081:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Pushed the V10 patch once again to the trunk. Thanks [~anastas].

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17355) Create a simplifed version of flush scanner

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778607#comment-15778607
 ] 

ramkrishna.s.vasudevan commented on HBASE-17355:


I think this is not going to be easy. So will get back here.

> Create a simplifed version of flush scanner
> ---
>
> Key: HBASE-17355
> URL: https://issues.apache.org/jira/browse/HBASE-17355
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17354.patch, after patch.png, before patch.png
>
>
> Currently we use StoreScanner for performing the flushes which actuallly goes 
> row by row. Probably that is not needed and we could always go ahead with a 
> simple loop in collecting the cells and writing them to the file. Inside 
> write path we have the required sanity check so it is not needed that the 
> store scanner does a sanity check. 
> Also the limit that could be retrieved in one next() call could be equivalent 
> to the block size configured as we do for compaction.
> Are there any filters that we want to do (i mean any version check or 
> deletion) that we need to check in flush? If so then this simplified version 
> will not work. I may be missing something but if so we need to see what are 
> those and add it here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778604#comment-15778604
 ] 

ramkrishna.s.vasudevan commented on HBASE-16421:


I will give a patch for the chunk id in the Cell's bytebuffer.
bq.The scans on top of the CellChunkMap
Started doing it . Will report back tomorrow with some reports.
I can see that flush is generated lot of garbage for sure. But impact on scan 
not pretty much sure how to measure it but I can see how is the Gc impact in a 
mixed read write work load.

> Introducing the CellChunkMap as a new additional index variant in the MemStore
> --
>
> Key: HBASE-16421
> URL: https://issues.apache.org/jira/browse/HBASE-16421
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Anastasia Braginsky
> Attachments: CellChunkMapRevived.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Follow up for HBASE-14921. This is going to be the umbrella JIRA to include 
> all the parts of integration of the CellChunkMap to the MemStore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778590#comment-15778590
 ] 

ramkrishna.s.vasudevan commented on HBASE-17081:


There is one failure. But the tests TestAsyncTableGetMultiThreaded and 
TestHRegionWithInMemoryFlush are passing.

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17376:
--
Status: Patch Available  (was: Open)

> ClientAsyncPrefetchScanner fails due to too many rows
> -
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE.17376.v0.patch, HBASE.17376.v1.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17376:
--
Attachment: HBASE.17376.v1.patch

> ClientAsyncPrefetchScanner fails due to too many rows
> -
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE.17376.v0.patch, HBASE.17376.v1.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778552#comment-15778552
 ] 

Hadoop QA commented on HBASE-17320:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 51s 
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 51s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
26m 8s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 5s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s 
{color} | {color:red} hbase-server generated 1 new + 1 unchanged - 0 fixed = 2 
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 50s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s 
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s 
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 5s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 24s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 
1s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 30s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844686/HBASE-17320-v2.patch |
| JIRA Issue | HBASE-17320 |
| Optional Tests |  asflicense  javac  javadoc  unit  

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778525#comment-15778525
 ] 

Hadoop QA commented on HBASE-17081:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
27m 21s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 5s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 34s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844679/HBASE-17081-V10.patch 
|
| JIRA Issue | HBASE-17081 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 2da1680ec94c 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 8fa5b0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5048/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5048/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5048/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> 

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778484#comment-15778484
 ] 

Ted Yu commented on HBASE-17374:


You can add debug log in the ZKPermissionWatcher method, identifying the 
executor (and stack trace):
{code}
  public void close() {
executor.shutdown();
{code}
It would make correlation easier when RejectedExecutionException happens.

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> 

[jira] [Commented] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[ChiaPing Tsai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778477#comment-15778477
 ] 

ChiaPing Tsai commented on HBASE-17376:
---

I will remove the trivial tests.

> ClientAsyncPrefetchScanner fails due to too many rows
> -
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE.17376.v0.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17376:
--
Status: Open  (was: Patch Available)

> ClientAsyncPrefetchScanner fails due to too many rows
> -
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE.17376.v0.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778468#comment-15778468
 ] 

Ted Yu commented on HBASE-17376:


>From trunk build #2201 :

Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.876 sec - in 
org.apache.hadoop.hbase.client.TestScannersFromClientSide

When I ran the test locally with patch:
{code}
Running org.apache.hadoop.hbase.client.TestScannersFromClientSide
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 70.47 sec - in 
org.apache.hadoop.hbase.client.TestScannersFromClientSide
{code}
Can the new tests be trimmed / shortened ?

> ClientAsyncPrefetchScanner fails due to too many rows
> -
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE.17376.v0.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16421) Introducing the CellChunkMap as a new additional index variant in the MemStore

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778400#comment-15778400
 ] 

Anastasia Braginsky commented on HBASE-16421:
-

Dear [~ram_krish], did you (or may be Anoop) have a chance to reference the to 
issues we talked above?

1. Chunk ID from the Cell and the suggested above idea to keep the Chunk ID as 
a first integer on each chunk’s byte buffer
2. The scans on top of the CellChunkMap

> Introducing the CellChunkMap as a new additional index variant in the MemStore
> --
>
> Key: HBASE-16421
> URL: https://issues.apache.org/jira/browse/HBASE-16421
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Anastasia Braginsky
> Attachments: CellChunkMapRevived.pdf, 
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Follow up for HBASE-14921. This is going to be the umbrella JIRA to include 
> all the parts of integration of the CellChunkMap to the MemStore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17377) MemStoreChunkAllocator

[Anastasia Braginsky (JIRA)]

Anastasia Braginsky created HBASE-17377:
---

 Summary: MemStoreChunkAllocator
 Key: HBASE-17377
 URL: https://issues.apache.org/jira/browse/HBASE-17377
 Project: HBase
  Issue Type: Sub-task
Reporter: Anastasia Braginsky


Refactoring the separation between MemStoreChunkPool and (new) 
MemStoreChunkAllocator. The latter allocates chunks either from heap or from 
pool and assigns them the Chunk IDs. MemStoreChunkAllocator stores the mapping 
between chunks and their IDs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17336) get/update replication peer config requests should be routed through master

[Guanghao Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang updated HBASE-17336:
---
Attachment: HBASE-17336-v3.patch

Fix ut TestReplicationShell.

> get/update replication peer config requests should be routed through master
> ---
>
> Key: HBASE-17336
> URL: https://issues.apache.org/jira/browse/HBASE-17336
> Project: HBase
>  Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17336-v1.patch, HBASE-17336-v2.patch, 
> HBASE-17336-v3.patch
>
>
> As HBASE-11392 description says, we should move replication operations to be 
> routed through master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778358#comment-15778358
 ] 

Hadoop QA commented on HBASE-17376:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
7s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 29s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 8s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
26s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 52s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844676/HBASE.17376.v0.patch |
| JIRA Issue | HBASE-17376 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 7c81e1bdc88f 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 
10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 8fa5b0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5046/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5046/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> ClientAsyncPrefetchScanner fails due to too many 

[jira] [Commented] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Duo Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778337#comment-15778337
 ] 

Duo Zhang commented on HBASE-17320:
---

There is a semantic change for the Scan object. In the old time, we assume the 
scan is a get if we found that the start row and stop row of it are equal. 
After we introduce the inclusive of the stopRow, it is more natural for us to 
require that the includeStopRow to also be true when we decide whether a scan 
is a get.

This lead to the failure of some UTs and I've add a dirty hack in the 
constructor Scan(byte[], byte[]) to fix them. If the startRow and stopRow are 
equal, then I will set the includeStopRow to true. I'm not sure if I should 
keep all the old behaviors as the old behavior seems wrong after we introduce 
the inclusive of stop row.

Suggestions are welcomed. Thanks.

> Add inclusive/exclusive support for startRow and endRow of scan
> ---
>
> Key: HBASE-17320
> URL: https://issues.apache.org/jira/browse/HBASE-17320
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, scan
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17320-v1.patch, HBASE-17320-v2.patch, 
> HBASE-17320.patch
>
>
> This is especially useful when doing reverse scan. HBASE-17168 maybe a more 
> powerful solution but we need to be careful about the atomicity, and I do not 
> think we will provide the feature to end user. But I think it is OK to 
> provide inclusive/exclusive option to end user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17320) Add inclusive/exclusive support for startRow and endRow of scan

[Duo Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang updated HBASE-17320:
--
Attachment: HBASE-17320-v2.patch

Fix the failing tests.

> Add inclusive/exclusive support for startRow and endRow of scan
> ---
>
> Key: HBASE-17320
> URL: https://issues.apache.org/jira/browse/HBASE-17320
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client, scan
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-17320-v1.patch, HBASE-17320-v2.patch, 
> HBASE-17320.patch
>
>
> This is especially useful when doing reverse scan. HBASE-17168 maybe a more 
> powerful solution but we need to be careful about the atomicity, and I do not 
> think we will provide the feature to end user. But I think it is OK to 
> provide inclusive/exclusive option to end user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Anastasia Braginsky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anastasia Braginsky updated HBASE-17081:

Status: Patch Available  (was: Reopened)

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17336) get/update replication peer config requests should be routed through master

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778295#comment-15778295
 ] 

Hadoop QA commented on HBASE-17336:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 51s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 13m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
40s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s 
{color} | {color:red} hbase-protocol-shaded in master has 24 extant Findbugs 
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 44s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m 
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
40s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 52 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 46s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m 
14s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 53s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s 
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 51s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 58s 
{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
56s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 298m 20s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestReplicationShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778284#comment-15778284
 ] 

Anastasia Braginsky commented on HBASE-17081:
-

In order to see Hadoop QA results should I change the status back to to 'Patch 
Available'?

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778231#comment-15778231
 ] 

Hadoop QA commented on HBASE-17374:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 24s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 27s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
14s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 127m 40s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844670/0001-fix-for-HBASE-17374.patch
 |
| JIRA Issue | HBASE-17374 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 9292af6424c8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 8fa5b0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5045/artifact/patchprocess/whitespace-eol.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5045/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5045/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: 

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778219#comment-15778219
 ] 

Anoop Sam John commented on HBASE-17375:


So you guys use Prefix tree..   We were not knowing who all uses this..  It 
is/was never declared a stable one :-)

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Junhong updated HBASE-17374:

Status: Open  (was: Patch Available)

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1279)) - Closed 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red}  2016-12-08 15:00:27,590 ERROR 

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778185#comment-15778185
 ] 

Hadoop QA commented on HBASE-17374:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
23m 26s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 54s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 11s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.access.TestAccessController3 |
|   | hadoop.hbase.security.access.TestAccessController |
|   | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12844665/0001-fix-for-HBASE-17374.patch
 |
| JIRA Issue | HBASE-17374 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 0d83db6c12b8 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 8fa5b0b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5044/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/5044/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/5044/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 

[jira] [Commented] (HBASE-17291) Remove ImmutableSegment#getKeyValueScanner

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778175#comment-15778175
 ] 

Anastasia Braginsky commented on HBASE-17291:
-

Thanks [~ram_krish]!
1. I understand now where the close of the SnapshotScanner happens.
2. I do not need more time with HBASE-17081 unless some unpredictable last 
minute issues will be shown up.

I hope we will be able to commit HBASE-17081 soon and just after this fix.

> Remove ImmutableSegment#getKeyValueScanner
> --
>
> Key: HBASE-17291
> URL: https://issues.apache.org/jira/browse/HBASE-17291
> Project: HBase
>  Issue Type: Improvement
>  Components: Scanners
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17291.patch, HBASE-17291_1.patch, 
> HBASE-17291_2.patch
>
>
> This is based on a discussion over [~anastas]'s patch. The MemstoreSnapshot 
> uses a KeyValueScanner which actually seems redundant considering we already 
> have a SegmentScanner. The idea is that the snapshot scanner should be a 
> simple iterator type of scanner but it lacks the capability to do the 
> reference counting on that segment that is now used in snapshot. With 
> snapshot having mulitple segments in the latest impl it is better we hold on 
> to the segment by doing ref counting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778166#comment-15778166
 ] 

Anastasia Braginsky commented on HBASE-17081:
-

I have just tried to submit the newly re-based patch...

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Anastasia Braginsky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anastasia Braginsky updated HBASE-17081:

Attachment: HBASE-17081-V10.patch

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBASE-17081-V10.patch, 
> HBaseMeetupDecember2016-V02.pptx, Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17376:
--
 Assignee: ChiaPing Tsai
Fix Version/s: 2.0.0
Affects Version/s: 2.0.0
   Status: Patch Available  (was: Open)

> ClientAsyncPrefetchScanner fails due to too many rows
> -
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0
>
> Attachments: HBASE.17376.v0.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17376:
--
Attachment: HBASE.17376.v0.patch

> ClientAsyncPrefetchScanner fails due to too many rows
> -
>
> Key: HBASE-17376
> URL: https://issues.apache.org/jira/browse/HBASE-17376
> Project: HBase
>  Issue Type: Bug
>Reporter: ChiaPing Tsai
> Attachments: HBASE.17376.v0.patch
>
>
> ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it 
> may throw “queue full”. For example, scan with infinite caching or extra data 
> from observer.
> {noformat}
>   @Override
>   protected void initCache() {
> // concurrent cache
> cacheCapacity = calcCacheCapacity();
> cache = new LinkedBlockingQueue(cacheCapacity);
> cacheSizeInBytes = new AtomicLong(0);
> exceptionsQueue = new ConcurrentLinkedQueue();
> prefetchRunnable = new PrefetchRunnable();
> prefetchRunning = new AtomicBoolean(false);
> closingThreadId = new AtomicLong(NO_THREAD);
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17376) ClientAsyncPrefetchScanner fails due to too many rows

[ChiaPing Tsai (JIRA)]

ChiaPing Tsai created HBASE-17376:
-

 Summary: ClientAsyncPrefetchScanner fails due to too many rows
 Key: HBASE-17376
 URL: https://issues.apache.org/jira/browse/HBASE-17376
 Project: HBase
  Issue Type: Bug
Reporter: ChiaPing Tsai


ClientAsyncPrefetchScanner uses a fixed capacity LinkedBlockingQueue, so it may 
throw “queue full”. For example, scan with infinite caching or extra data from 
observer.
{noformat}
  @Override
  protected void initCache() {
// concurrent cache
cacheCapacity = calcCacheCapacity();
cache = new LinkedBlockingQueue(cacheCapacity);
cacheSizeInBytes = new AtomicLong(0);
exceptionsQueue = new ConcurrentLinkedQueue();
prefetchRunnable = new PrefetchRunnable();
prefetchRunning = new AtomicBoolean(false);
closingThreadId = new AtomicLong(NO_THREAD);
  }
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Anastasia Braginsky (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778124#comment-15778124
 ] 

Anastasia Braginsky commented on HBASE-17081:
-

This JIRA is quite an old JIRA with quite a big fix. 
HBASE-17373 should also be not small, as we plan to improve the testing under 
HBASE-17373 as well.
HBASE-17081 and HBASE-17373 are unrelated.
Why to put it all together?

As we have seen many times, it is better to treat separate issues separately.

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBaseMeetupDecember2016-V02.pptx, 
> Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742667#comment-15742667
 ] 

Ted Yu edited comment on HBASE-16179 at 12/26/16 10:49 AM:
---

Can I get some review in order to move this JIRA forward ?


was (Author: yuzhih...@gmail.com):
Can I get some review in order to move this forward ?

> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
>  Issue Type: Bug
>  Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, 
> 16179.v11.txt, 16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 
> 16179.v15.txt, 16179.v16.txt, 16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 
> 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the 
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer 
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778096#comment-15778096
 ] 

Chang chen edited comment on HBASE-17375 at 12/26/16 10:36 AM:
---

I already wrote a test case in a standalone application, I will upload a UT 
patch ASAP


was (Author: baibaichen):
I already write a test case in a standalone application, I will apply a UT 
patch ASAP

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778096#comment-15778096
 ] 

Chang chen commented on HBASE-17375:


I already write a test case in a standalone application, I will apply a UT 
patch ASAP

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778084#comment-15778084
 ] 

ramkrishna.s.vasudevan commented on HBASE-17375:


[~baibaichen]
Thanks for the nice analysis. Do you like to give a patch with a test for the 
above case? 

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778080#comment-15778080
 ] 

Ted Yu commented on HBASE-17374:


The release() method does:
{code}
managerMap.remove(instance.getZKPermissionWatcher().getWatcher());
refCount.remove(instance);
{code}
where watcher is not closed.

Please add debug logging if code analysis alone doesn't explain what you saw.

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> 

[jira] [Updated] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chang chen updated HBASE-17375:
---
Description: 
Recently, we find our hbase compaction thread never end.  Assume we have 
following cells:
{quote}
 1
 1
 1
 1
 1
 1
 1
 1
{quote}

If we encode above datas into prefix tree block, then it looks like:
!row trie example.PNG!

Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
4), then the previous row should be *Aa* (e.g. 2). However 
previousRowInternal return {color:red}A{color}(e.g. 1)

After investigation, I believe it's the bug of 
PrefixTreeArrayReversibleScanner#previousRowInternal.

{code}
  private boolean previousRowInternal() {
//...
while (!beforeFirst) {
  //
  // what if currentRowNode is nub?
  if (currentRowNode.hasOccurrences()) {// escape clause
currentRowNode.resetFanIndex();
return true;// found some values
  }
}
{code}

currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
the case of  currentRowNode.isNub() is true, previousRowInternal should follow 
the previous fan instead of return.

  was:
Recently, we find our hbase compaction thread never end. After investigation, I 
believe it's the bug of PrefixTreeArrayReversibleScanner#previousRowInternal.

{code}
  private boolean previousRowInternal() {
//...
while (!beforeFirst) {
  //
  // what if currentRowNode is nub?
  if (currentRowNode.hasOccurrences()) {// escape clause
currentRowNode.resetFanIndex();
return true;// found some values
  }
}
{code}

Assume we have following cells:
{quote}
 1
 1
 1
 1
 1
 1
 1
 1
{quote}

If the current row is {color:red}Abc{color}, then the preivous row should be 
*Aa*,. However previousRowInternal return {color:red}A{color}


> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end.  Assume we have 
> following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If we encode above datas into prefix tree block, then it looks like:
> !row trie example.PNG!
> Assume the current row is {color:red}Abc{color} (e.g. the current row node is 
> 4), then the previous row should be *Aa* (e.g. 2). However 
> previousRowInternal return {color:red}A{color}(e.g. 1)
> After investigation, I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> currentRowNode.hasOccurrences() only test whether it has cell or not. But in 
> the case of  currentRowNode.isNub() is true, previousRowInternal should 
> follow the previous fan instead of return.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778078#comment-15778078
 ] 

Liu Junhong commented on HBASE-17374:
-

But it is still strange for why 
org.apache.hadoop.hbase.security.access.TableAuthManager.release(TableAuthManager).instance
 is be close, I will look up code later.

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> 

[jira] [Updated] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Junhong updated HBASE-17374:

Description: 
It was occurred many time that  I granted some permission,  but few of some 
regionservers was not token effect and must be restart . When I look up logs,  
I found that :

2016-12-08 15:00:26,238 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
handler.CloseRegionHandler (CloseRegionHandler.java:process(128)) - Processing 
close of data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
{color:red} 2016-12-08 15:00:26,242 DEBUG 
[RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
(HRegion.java:doClose(1163)) - Closing 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
compactions & flushes {color}
2016-12-08 15:00:26,242 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:doClose(1190)) - Updates disabled for region 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
2016-12-08 15:00:26,242 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:internalFlushcache(1753)) - Started memstore 
flush for data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., 
current region memstore size 160
2016-12-08 15:00:26,284 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) - 
Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
2016-12-08 15:00:26,303 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) - 
Committing store file 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
 as 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
2016-12-08 15:00:26,318 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HStore (HStore.java:commitFile(877)) - Added 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
 entries=1, sequenceid=6, filesize=985
2016-12-08 15:00:26,319 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:internalFlushcache(1920)) - Finished 
memstore flush of ~160/160, currentsize=0/0 for region 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
sequenceid=6, compaction requested=false
2016-12-08 15:00:26,323 INFO  
[StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
 regionserver.HStore (HStore.java:close(774)) - Closed cf1
2016-12-08 15:00:26,325 INFO  
[StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
 regionserver.HStore (HStore.java:close(774)) - Closed cf2
2016-12-08 15:00:26,326 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop 
coprocessor org.apache.hadoop.hbase.security.token.TokenProvider
{color:red}  2016-12-08 15:00:26,326 DEBUG 
[RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
(CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
org.apache.hadoop.hbase.security.access.AccessController  {color}
2016-12-08 15:00:26,327 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop 
coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
2016-12-08 15:00:26,327 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop 
coprocessor org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
2016-12-08 15:00:26,328 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:doClose(1279)) - Closed 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
{color:red}  2016-12-08 15:00:27,590 ERROR [regionserver60020-EventThread] 
zookeeper.ClientCnxn (ClientCnxn.java:processEvent(524)) - Error while calling 
watcher
java.util.concurrent.RejectedExecutionException: Task 
java.util.concurrent.FutureTask@1851ab3a rejected from 
java.util.concurrent.ThreadPoolExecutor@19c0794f[Terminated, pool size = 0, 
active threads = 0, queued tasks = 0, completed tasks = 1]
at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at 

[jira] [Updated] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Junhong updated HBASE-17374:

Attachment: (was: 0001-fix-for-HBASE-17374.patch)

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1279)) - Closed 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red}  2016-12-08 15:00:27,590 ERROR 

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778074#comment-15778074
 ] 

Liu Junhong commented on HBASE-17374:
-

Oh, I understand your mean, I had a mistake.

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1279)) - Closed 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red}  2016-12-08 

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778069#comment-15778069
 ] 

Ted Yu commented on HBASE-17374:


Junhong:
Can you come up with a test which shows the defect in the current logic ?

As I stated above, refCount is static - same with getOrCreate() and release().
I don't see the difference your patch makes. Does it fix the problem for you ?

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> 

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778065#comment-15778065
 ] 

ramkrishna.s.vasudevan commented on HBASE-17081:


So is it possible for you to add the fix of HBASE-17373 as part of this JIRA 
only? As this is not committed we could use update the patch here as it was 
having some issues?

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBaseMeetupDecember2016-V02.pptx, 
> Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778066#comment-15778066
 ] 

Liu Junhong commented on HBASE-17374:
-

No seen it .

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1279)) - Closed 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778060#comment-15778060
 ] 

Ted Yu commented on HBASE-17374:


{code}
if (refCount.get(instance) == null || refCount.get(instance) < 1) {
  String msg = "Something wrong with the TableAuthManager reference 
counting: " + instance
  + " whose count is " + refCount.get(instance);
{code}
Do you see the above log in region server log ?



> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> 

[jira] [Updated] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chang chen updated HBASE-17375:
---
Attachment: row trie example.PNG

> PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly
> ---
>
> Key: HBASE-17375
> URL: https://issues.apache.org/jira/browse/HBASE-17375
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 0.98.24
>Reporter: Chang chen
> Attachments: row trie example.PNG
>
>
> Recently, we find our hbase compaction thread never end. After investigation, 
> I believe it's the bug of 
> PrefixTreeArrayReversibleScanner#previousRowInternal.
> {code}
>   private boolean previousRowInternal() {
> //...
> while (!beforeFirst) {
>   //
>   // what if currentRowNode is nub?
>   if (currentRowNode.hasOccurrences()) {// escape clause
> currentRowNode.resetFanIndex();
> return true;// found some values
>   }
> }
> {code}
> Assume we have following cells:
> {quote}
>  1
>  1
>  1
>  1
>  1
>  1
>  1
>  1
> {quote}
> If the current row is {color:red}Abc{color}, then the preivous row should be 
> *Aa*,. However previousRowInternal return {color:red}A{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17375) PrefixTreeArrayReversibleScanner#previousRowInternal doesn't work correctly

[Chang chen (JIRA)]

Chang chen created HBASE-17375:
--

 Summary: PrefixTreeArrayReversibleScanner#previousRowInternal 
doesn't work correctly
 Key: HBASE-17375
 URL: https://issues.apache.org/jira/browse/HBASE-17375
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.24, 0.98.23, 1.1.7
Reporter: Chang chen


Recently, we find our hbase compaction thread never end. After investigation, I 
believe it's the bug of PrefixTreeArrayReversibleScanner#previousRowInternal.

{code}
  private boolean previousRowInternal() {
//...
while (!beforeFirst) {
  //
  // what if currentRowNode is nub?
  if (currentRowNode.hasOccurrences()) {// escape clause
currentRowNode.resetFanIndex();
return true;// found some values
  }
}
{code}

Assume we have following cells:
{quote}
 1
 1
 1
 1
 1
 1
 1
 1
{quote}

If the current row is {color:red}Abc{color}, then the preivous row should be 
*Aa*,. However previousRowInternal return {color:red}A{color}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778054#comment-15778054
 ] 

Liu Junhong commented on HBASE-17374:
-

Reading and Changing the value of refCount is not atomicity.

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1279)) - Closed 
> 

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778050#comment-15778050
 ] 

Liu Junhong commented on HBASE-17374:
-

Sorry for wrong description, i mean when cocurrent call 
org.apache.hadoop.hbase.security.access.TableAuthManager.getOrCreate(ZooKeeperWatcher,
 Configuration) and 
org.apache.hadoop.hbase.security.access.TableAuthManager.release(TableAuthManager)
  , it may make wrong refCount value.
I have submit a patch , please review it.
Thank you.

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> 

[jira] [Issue Comment Deleted] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Junhong updated HBASE-17374:

Comment: was deleted

(was: Sorry for wrong description, i mean when cocurrent call 
org.apache.hadoop.hbase.security.access.TableAuthManager.getOrCreate(ZooKeeperWatcher,
 Configuration) and 
org.apache.hadoop.hbase.security.access.TableAuthManager.release(TableAuthManager)
  , it may make wrong refCount value.
I have submit a patch , please review it.
Thank you.)

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> 

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778049#comment-15778049
 ] 

Ted Yu commented on HBASE-17374:


refCount is static.

What's the difference between proposed fix and current implementation ?

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1279)) - Closed 

[jira] [Commented] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778051#comment-15778051
 ] 

Liu Junhong commented on HBASE-17374:
-

Sorry for wrong description, i mean when cocurrent call 
org.apache.hadoop.hbase.security.access.TableAuthManager.getOrCreate(ZooKeeperWatcher,
 Configuration) and 
org.apache.hadoop.hbase.security.access.TableAuthManager.release(TableAuthManager)
  , it may make wrong refCount value.
I have submit a patch , please review it.
Thank you.

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> 

[jira] [Updated] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Junhong updated HBASE-17374:

Description: 
It was occurred many time that  I granted some permission,  but few of some 
regionservers was not token effect and must be restart . When I look up logs,  
I found that :

2016-12-08 15:00:26,238 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
handler.CloseRegionHandler (CloseRegionHandler.java:process(128)) - Processing 
close of data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
{color:red} 2016-12-08 15:00:26,242 DEBUG 
[RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
(HRegion.java:doClose(1163)) - Closing 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
compactions & flushes {color}
2016-12-08 15:00:26,242 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:doClose(1190)) - Updates disabled for region 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
2016-12-08 15:00:26,242 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:internalFlushcache(1753)) - Started memstore 
flush for data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., 
current region memstore size 160
2016-12-08 15:00:26,284 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) - 
Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
2016-12-08 15:00:26,303 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) - 
Committing store file 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
 as 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
2016-12-08 15:00:26,318 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HStore (HStore.java:commitFile(877)) - Added 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
 entries=1, sequenceid=6, filesize=985
2016-12-08 15:00:26,319 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:internalFlushcache(1920)) - Finished 
memstore flush of ~160/160, currentsize=0/0 for region 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
sequenceid=6, compaction requested=false
2016-12-08 15:00:26,323 INFO  
[StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
 regionserver.HStore (HStore.java:close(774)) - Closed cf1
2016-12-08 15:00:26,325 INFO  
[StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
 regionserver.HStore (HStore.java:close(774)) - Closed cf2
2016-12-08 15:00:26,326 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop 
coprocessor org.apache.hadoop.hbase.security.token.TokenProvider
{color:red}  2016-12-08 15:00:26,326 DEBUG 
[RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
(CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
org.apache.hadoop.hbase.security.access.AccessController  {color}
2016-12-08 15:00:26,327 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop 
coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
2016-12-08 15:00:26,327 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop 
coprocessor org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
2016-12-08 15:00:26,328 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:doClose(1279)) - Closed 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
{color:red}  2016-12-08 15:00:27,590 ERROR [regionserver60020-EventThread] 
zookeeper.ClientCnxn (ClientCnxn.java:processEvent(524)) - Error while calling 
watcher
java.util.concurrent.RejectedExecutionException: Task 
java.util.concurrent.FutureTask@1851ab3a rejected from 
java.util.concurrent.ThreadPoolExecutor@19c0794f[Terminated, pool size = 0, 
active threads = 0, queued tasks = 0, completed tasks = 1]
at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at 

[jira] [Updated] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Junhong updated HBASE-17374:

Status: Patch Available  (was: Open)

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1279)) - Closed 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red}  

[jira] [Updated] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Junhong updated HBASE-17374:

Attachment: 0001-fix-for-HBASE-17374.patch

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
> 2016-12-08 15:00:26,328 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1279)) - Closed 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 

[jira] [Comment Edited] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778017#comment-15778017
 ] 

Ted Yu edited comment on HBASE-17374 at 12/26/16 10:05 AM:
---

getOrCreate() is already synchronized:
{code}
  public synchronized static TableAuthManager getOrCreate(
  ZooKeeperWatcher watcher, Configuration conf) throws IOException {
{code}



was (Author: yuzhih...@gmail.com):
Can the problem be reproduced with 0.98.23 ?

Does the proposed patch fix the problem ?

Thanks

> ZKPermissionWatcher crashed when grant after close region 
> --
>
> Key: HBASE-17374
> URL: https://issues.apache.org/jira/browse/HBASE-17374
> Project: HBase
>  Issue Type: Bug
>  Components: regionserver
>Affects Versions: 0.98.15
>Reporter: Liu Junhong
>Priority: Critical
> Attachments: 0001-fix-for-HBASE-17374.patch
>
>
> It was occurred many time that  I granted some permission,  but few of some 
> regionservers was not token effect and must be restart . When I look up logs, 
>  I found that :
> 2016-12-08 15:00:26,238 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] handler.CloseRegionHandler 
> (CloseRegionHandler.java:process(128)) - Processing close of 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> {color:red} 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1163)) - Closing 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
> compactions & flushes {color}
> 2016-12-08 15:00:26,242 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:doClose(1190)) - Updates disabled for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
> 2016-12-08 15:00:26,242 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1753)) - Started memstore flush for 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., current 
> region memstore size 160
> 2016-12-08 15:00:26,284 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) 
> - Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,303 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
> regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) 
> - Committing store file 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
>  as 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
> 2016-12-08 15:00:26,318 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HStore 
> (HStore.java:commitFile(877)) - Added 
> hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
>  entries=1, sequenceid=6, filesize=985
> 2016-12-08 15:00:26,319 INFO  
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
> (HRegion.java:internalFlushcache(1920)) - Finished memstore flush of 
> ~160/160, currentsize=0/0 for region 
> data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
> sequenceid=6, compaction requested=false
> 2016-12-08 15:00:26,323 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf1
> 2016-12-08 15:00:26,325 INFO  
> [StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
>  regionserver.HStore (HStore.java:close(774)) - Closed cf2
> 2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.token.TokenProvider
> {color:red}  2016-12-08 15:00:26,326 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.AccessController  {color}
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
> (CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> 2016-12-08 15:00:26,327 DEBUG 
> [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778028#comment-15778028
 ] 

Ted Yu commented on HBASE-17081:


Whoever commits this patch, please watch TestHRegion\* in subsequent QA runs.

I still think fix for HBASE-17373 should be committed first which should 
stabilize test result.

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-15787_8.patch, HBASE-17081-V01.patch, 
> HBASE-17081-V02.patch, HBASE-17081-V03.patch, HBASE-17081-V04.patch, 
> HBASE-17081-V05.patch, HBASE-17081-V06.patch, HBASE-17081-V06.patch, 
> HBASE-17081-V07.patch, HBaseMeetupDecember2016-V02.pptx, 
> Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17374) ZKPermissionWatcher crashed when grant after close region

[Liu Junhong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Junhong updated HBASE-17374:

Description: 
It was occurred many time that  I granted some permission,  but few of some 
regionservers was not token effect and must be restart . When I look up logs,  
I found that :

2016-12-08 15:00:26,238 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
handler.CloseRegionHandler (CloseRegionHandler.java:process(128)) - Processing 
close of data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
{color:red} 2016-12-08 15:00:26,242 DEBUG 
[RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] regionserver.HRegion 
(HRegion.java:doClose(1163)) - Closing 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.: disabling 
compactions & flushes {color}
2016-12-08 15:00:26,242 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:doClose(1190)) - Updates disabled for region 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
2016-12-08 15:00:26,242 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:internalFlushcache(1753)) - Started memstore 
flush for data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14., 
current region memstore size 160
2016-12-08 15:00:26,284 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.DefaultStoreFlusher (DefaultStoreFlusher.java:flushSnapshot(95)) - 
Flushed, sequenceid=6, memsize=160, hasBloomFilter=true, into tmp file 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
2016-12-08 15:00:26,303 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegionFileSystem (HRegionFileSystem.java:commitStoreFile(370)) - 
Committing store file 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/.tmp/8d734ce3d93e40628d8f82111e754cb3
 as 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3
2016-12-08 15:00:26,318 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HStore (HStore.java:commitFile(877)) - Added 
hdfs://dx-data-hbase-watcher/hbase/data/default/data-probe-test/5f06cb6447343b602e05996bfd87ce14/cf2/8d734ce3d93e40628d8f82111e754cb3,
 entries=1, sequenceid=6, filesize=985
2016-12-08 15:00:26,319 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:internalFlushcache(1920)) - Finished 
memstore flush of ~160/160, currentsize=0/0 for region 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14. in 77ms, 
sequenceid=6, compaction requested=false
2016-12-08 15:00:26,323 INFO  
[StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
 regionserver.HStore (HStore.java:close(774)) - Closed cf1
2016-12-08 15:00:26,325 INFO  
[StoreCloserThread-data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.-1]
 regionserver.HStore (HStore.java:close(774)) - Closed cf2
2016-12-08 15:00:26,326 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop 
coprocessor org.apache.hadoop.hbase.security.token.TokenProvider
{color:red}  2016-12-08 15:00:26,326 DEBUG 
[RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] coprocessor.CoprocessorHost 
(CoprocessorHost.java:shutdown(292)) - Stop coprocessor 
org.apache.hadoop.hbase.security.access.AccessController  {color}
2016-12-08 15:00:26,327 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop 
coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
2016-12-08 15:00:26,327 DEBUG [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
coprocessor.CoprocessorHost (CoprocessorHost.java:shutdown(292)) - Stop 
coprocessor org.apache.hadoop.hbase.regionserver.ExternalMetricObserver
2016-12-08 15:00:26,328 INFO  [RS_CLOSE_REGION-dx-data-hbase-watcher05:60020-0] 
regionserver.HRegion (HRegion.java:doClose(1279)) - Closed 
data-probe-test,,1481180420784.5f06cb6447343b602e05996bfd87ce14.
{color:red}  2016-12-08 15:00:27,590 ERROR [regionserver60020-EventThread] 
zookeeper.ClientCnxn (ClientCnxn.java:processEvent(524)) - Error while calling 
watcher
java.util.concurrent.RejectedExecutionException: Task 
java.util.concurrent.FutureTask@1851ab3a rejected from 
java.util.concurrent.ThreadPoolExecutor@19c0794f[Terminated, pool size = 0, 
active threads = 0, queued tasks = 0, completed tasks = 1]
at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at