[jira] [Assigned] (HBASE-11386) Replication#table,CF config will be wrong if the table name includes namespace

[Ashish Singhi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-11386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish Singhi reassigned HBASE-11386:
-

Assignee: Ashish Singhi  (was: Qianxi Zhang)

> Replication#table,CF config will be wrong if the table name includes namespace
> --
>
> Key: HBASE-11386
> URL: https://issues.apache.org/jira/browse/HBASE-11386
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Qianxi Zhang
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE_11386_trunk_v1.patch, HBASE_11386_trunk_v2.patch
>
>
> Now we can config the table and CF in Replication, but I think the parse will 
> be wrong if the table name includes namespace
> ReplicationPeer#parseTableCFsFromConfig(line 125)
> {code}
> Map tableCFsMap = null;
> // parse out (table, cf-list) pairs from tableCFsConfig
> // format: "table1:cf1,cf2;table2:cfA,cfB"
> String[] tables = tableCFsConfig.split(";");
> for (String tab : tables) {
>   // 1 ignore empty table config
>   tab = tab.trim();
>   if (tab.length() == 0) {
> continue;
>   }
>   // 2 split to "table" and "cf1,cf2"
>   //   for each table: "table:cf1,cf2" or "table"
>   String[] pair = tab.split(":");
>   String tabName = pair[0].trim();
>   if (pair.length > 2 || tabName.length() == 0) {
> LOG.error("ignore invalid tableCFs setting: " + tab);
> continue;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11386) Replication#table,CF config will be wrong if the table name includes namespace

[Ashish Singhi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-11386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008485#comment-15008485
 ] 

Ashish Singhi commented on HBASE-11386:
---

OK, I will look into this.

> Replication#table,CF config will be wrong if the table name includes namespace
> --
>
> Key: HBASE-11386
> URL: https://issues.apache.org/jira/browse/HBASE-11386
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Qianxi Zhang
>Assignee: Qianxi Zhang
>Priority: Critical
> Attachments: HBASE_11386_trunk_v1.patch, HBASE_11386_trunk_v2.patch
>
>
> Now we can config the table and CF in Replication, but I think the parse will 
> be wrong if the table name includes namespace
> ReplicationPeer#parseTableCFsFromConfig(line 125)
> {code}
> Map tableCFsMap = null;
> // parse out (table, cf-list) pairs from tableCFsConfig
> // format: "table1:cf1,cf2;table2:cfA,cfB"
> String[] tables = tableCFsConfig.split(";");
> for (String tab : tables) {
>   // 1 ignore empty table config
>   tab = tab.trim();
>   if (tab.length() == 0) {
> continue;
>   }
>   // 2 split to "table" and "cf1,cf2"
>   //   for each table: "table:cf1,cf2" or "table"
>   String[] pair = tab.split(":");
>   String tabName = pair[0].trim();
>   if (pair.length > 2 || tabName.length() == 0) {
> LOG.error("ignore invalid tableCFs setting: " + tab);
> continue;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14826) Small improvement in KVHeap seek() API

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14826:
---
  Priority: Minor  (was: Major)
Issue Type: Improvement  (was: Bug)

> Small improvement in KVHeap seek() API
> --
>
> Key: HBASE-14826
> URL: https://issues.apache.org/jira/browse/HBASE-14826
> Project: HBase
>  Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Priority: Minor
>
> Currently in seek/reseek() APIs we tend to do lot of priorityqueue related 
> operations. We initially add the current scanner to the heap, then poll and 
> again add the scanner back if the seekKey is greater than the topkey in that 
> scanner. Since the KVs are always going to be in increasing order and in 
> ideal scan flow every seek/reseek is followed by a next() call it should be 
> ok if we start with checking the current scanner and then do a poll to get 
> the next scanner. Just avoid the initial PQ.add(current) call. This could 
> save some comparisons. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14824) HBaseAdmin.mergeRegions should use full region names instead of encoded region names

[Jean-Marc Spaggiari (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008565#comment-15008565
 ] 

Jean-Marc Spaggiari commented on HBASE-14824:
-

Might be nice to detect the format of the input and merge with the full name or 
the encoded name, if doable. What do you have in mind for the full name? Like 
table:firstkey:etc.? If so we might be able to detect that, right?

> HBaseAdmin.mergeRegions should use full region names instead of encoded 
> region names
> 
>
> Key: HBASE-14824
> URL: https://issues.apache.org/jira/browse/HBASE-14824
> Project: HBase
>  Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14824.patch
>
>
> HBaseAdmin.mergeRegions() calls HBaseAdmin.getRegion() internally. 
> HBaseAdmin.getRegion() requires the full region name. So 
> MetaTableAccessor.getRegion always returns null and this causes one more meta 
> table scan.
> {code}
>   Pair getRegion(final byte[] regionName) throws 
> IOException {
> if (regionName == null) {
>   throw new IllegalArgumentException("Pass a table name or region name");
> }
> Pair pair =
>   MetaTableAccessor.getRegion(connection, regionName);
> if (pair == null) {
> {code}
> I suppose to use full region names instead of encoded region names in 
> HBaseAdmin.mergeRegions().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008465#comment-15008465
 ] 

ramkrishna.s.vasudevan commented on HBASE-14221:


In order to proceed with HBASE-14221, I did some analysis in this JIRa with the 
patches provided and the benchmarks given. What Matt says is true, reseeks with 
LoserTree is faster whereas next() with LoserTree is slower. This is surely 
because of the number of operations and comparisons that we perform. 
In case of next() with KVHeap, we always have do a peek() to the next KVScanner 
and with that we compare the current KV if we really to add the scanner back to 
the KVHeap.  So here there is no change to the heap. and so no comparison. 
Where as in case of LoserTree we need to fetch the next key and always keep 
adjusting the tree. 
In case of reseek() the case with KVHeap is that we do lot of changes to the 
heap, first we simply add the current scanner to the heap, then do a poll and 
again based on the need we again add the current scanner to the heap. But in 
case of LoserTree these many comparisons are avoided and it is similar to the 
next() call with LoserTree.
Am not very sure which one would be better here because both next() and 
reseek() is going to be important when we have filters along with scans. 
Coming to this patch- this patch is more to do with the comparisons in the 
StoreScanner level and Hregion level and it is mainly in comparing the rows. So 
this patch can be seen as a different area than where the KVHeap would come in. 
So it may be worth pursuing? Am still doing some more analysis on the LoserTree 
and its impl and see how we can benefit from them. 

> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
>  Issue Type: Sub-task
>  Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch, 
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch, 
> withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
>   if (ret <= -1) {
> return MatchCode.DONE;
>   } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
>   }
> } else {
>   if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
>   } else if (ret >= 1) {
> return MatchCode.DONE;
>   }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) || 
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
>   this.countPerRow = 0;
>   matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
>   scannerContext.setKeepProgress(true);
>   heap.next(results, scannerContext);
>   scannerContext.setKeepProgress(tmpKeepProgress);
>   nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case.  Was 
> trying to solve this for the MultiCF case but is having lot of cases to 
> solve. But atleast for a single CF case I think these comparison can be 
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a 
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in 
> the heap and so the 3rd comparison can surely be avoided if the 
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that 
> can be avoided if we know that the previous next() call was over due to a new 
> row. Doing all this I found that the compareRows in the profiler which was 
> 19% got reduced to 13%. Initially we can solve for single CF case which can 
> be extended to MultiCF cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14826) Small improvement in KVHeap seek() API

[ramkrishna.s.vasudevan (JIRA)]

ramkrishna.s.vasudevan created HBASE-14826:
--

 Summary: Small improvement in KVHeap seek() API
 Key: HBASE-14826
 URL: https://issues.apache.org/jira/browse/HBASE-14826
 Project: HBase
  Issue Type: Bug
Reporter: ramkrishna.s.vasudevan


Currently in seek/reseek() APIs we tend to do lot of priorityqueue related 
operations. We initially add the current scanner to the heap, then poll and 
again add the scanner back if the seekKey is greater than the topkey in that 
scanner. Since the KVs are always going to be in increasing order and in ideal 
scan flow every seek/reseek is followed by a next() call it should be ok if we 
start with checking the current scanner and then do a poll to get the next 
scanner. Just avoid the initial PQ.add(current) call. This could save some 
comparisons. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14806) Missing sources.jar for several modules when building HBase

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008569#comment-15008569
 ] 

Hudson commented on HBASE-14806:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1133 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1133/])
HBASE-14806 Missing sources.jar for several modules when building HBase 
(zhangduo: rev 5bd078050cb6fc7f7a3b57c594ef3c7dabfbcd1c)
* hbase-common/pom.xml


> Missing sources.jar for several modules when building HBase
> ---
>
> Key: HBASE-14806
> URL: https://issues.apache.org/jira/browse/HBASE-14806
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.17, 1.0.4
>
> Attachments: HBASE-14806.patch
>
>
> Introduced by HBASE-14085. The problem is, for example, in 
> hbase-common/pom.xml, we have
> {code:title=pom.xml}
> 
>   org.apache.maven.plugins
>   maven-source-plugin
>   
> true
> 
>   src/main/java
>   ${project.build.outputDirectory}/META-INF
> 
>   
> 
> {code}
> But in fact, the path inside {{}} tag is relative to source 
> directories, not the project directory. So the maven-source-plugin always end 
> with
> {noformat}
> No sources in project. Archive not created.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14824) HBaseAdmin.mergeRegions should use full region names instead of encoded region names

[Eungsop Yoo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008584#comment-15008584
 ] 

Eungsop Yoo commented on HBASE-14824:
-

Yes, it is.

> HBaseAdmin.mergeRegions should use full region names instead of encoded 
> region names
> 
>
> Key: HBASE-14824
> URL: https://issues.apache.org/jira/browse/HBASE-14824
> Project: HBase
>  Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14824.patch
>
>
> HBaseAdmin.mergeRegions() calls HBaseAdmin.getRegion() internally. 
> HBaseAdmin.getRegion() requires the full region name. So 
> MetaTableAccessor.getRegion always returns null and this causes one more meta 
> table scan.
> {code}
>   Pair getRegion(final byte[] regionName) throws 
> IOException {
> if (regionName == null) {
>   throw new IllegalArgumentException("Pass a table name or region name");
> }
> Pair pair =
>   MetaTableAccessor.getRegion(connection, regionName);
> if (pair == null) {
> {code}
> I suppose to use full region names instead of encoded region names in 
> HBaseAdmin.mergeRegions().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008471#comment-15008471
 ] 

ramkrishna.s.vasudevan commented on HBASE-14221:


One thing I can see is that when reseeks() are more (when we keep filtering 
more and more) then the LoserTree impl would be more ideal. 

> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
>  Issue Type: Sub-task
>  Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch, 
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch, 
> withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
>   if (ret <= -1) {
> return MatchCode.DONE;
>   } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
>   }
> } else {
>   if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
>   } else if (ret >= 1) {
> return MatchCode.DONE;
>   }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) || 
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
>   this.countPerRow = 0;
>   matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
>   scannerContext.setKeepProgress(true);
>   heap.next(results, scannerContext);
>   scannerContext.setKeepProgress(tmpKeepProgress);
>   nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case.  Was 
> trying to solve this for the MultiCF case but is having lot of cases to 
> solve. But atleast for a single CF case I think these comparison can be 
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a 
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in 
> the heap and so the 3rd comparison can surely be avoided if the 
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that 
> can be avoided if we know that the previous next() call was over due to a new 
> row. Doing all this I found that the compareRows in the profiler which was 
> 19% got reduced to 13%. Initially we can solve for single CF case which can 
> be extended to MultiCF cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008473#comment-15008473
 ] 

ramkrishna.s.vasudevan commented on HBASE-9969:
---

Pls see this comment 
https://issues.apache.org/jira/browse/HBASE-14221?focusedCommentId=15008465=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15008465
 over in HBASE-14221.

> Improve KeyValueHeap using loser tree
> -
>
> Key: HBASE-9969
> URL: https://issues.apache.org/jira/browse/HBASE-9969
> Project: HBase
>  Issue Type: Improvement
>  Components: Performance, regionserver
>Reporter: Chao Shi
>Assignee: Chao Shi
> Fix For: 2.0.0
>
> Attachments: 9969-0.94.txt, KeyValueHeapBenchmark_v1.ods, 
> KeyValueHeapBenchmark_v2.ods, hbase-9969-pq-v1.patch, hbase-9969-pq-v2.patch, 
> hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, 
> kvheap-benchmark.png, kvheap-benchmark.txt
>
>
> LoserTree is the better data structure than binary heap. It saves half of the 
> comparisons on each next(), though the time complexity is on O(logN).
> Currently A scan or get will go through two KeyValueHeaps, one is merging KVs 
> read from multiple HFiles in a single store, the other is merging results 
> from multiple stores. This patch should improve the both cases whenever CPU 
> is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811).
> All of the optimization work is done in KeyValueHeap and does not change its 
> public interfaces. The new code looks more cleaner and simpler to understand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14826) Small improvement in KVHeap seek() API

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14826:
---
Attachment: HBASE-14826.patch

Patch for QA.

> Small improvement in KVHeap seek() API
> --
>
> Key: HBASE-14826
> URL: https://issues.apache.org/jira/browse/HBASE-14826
> Project: HBase
>  Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-14826.patch
>
>
> Currently in seek/reseek() APIs we tend to do lot of priorityqueue related 
> operations. We initially add the current scanner to the heap, then poll and 
> again add the scanner back if the seekKey is greater than the topkey in that 
> scanner. Since the KVs are always going to be in increasing order and in 
> ideal scan flow every seek/reseek is followed by a next() call it should be 
> ok if we start with checking the current scanner and then do a poll to get 
> the next scanner. Just avoid the initial PQ.add(current) call. This could 
> save some comparisons. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14803) Add some debug logs to StoreFileScanner

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008919#comment-15008919
 ] 

Heng Chen commented on HBASE-14803:
---

Except that,  patch looks good to me. :)  

> Add some debug logs to StoreFileScanner
> ---
>
> Key: HBASE-14803
> URL: https://issues.apache.org/jira/browse/HBASE-14803
> Project: HBase
>  Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
>  Labels: beginner
> Fix For: 1.2.0
>
> Attachments: HBASE-14803.v0-trunk.patch, HBASE-14803.v1-trunk.patch, 
> HBASE-14803.v2-trunk.patch, HBASE-14803.v3-trunk.patch, 
> HBASE-14803.v4-trunk.patch, HBASE-14803.v4-trunk.patch, 
> HBASE-14803.v5-trunk.patch
>
>
> To validate some behaviors I had to add some logs into StoreFileScanner.
> I think it can be interesting for other people looking for debuging. So 
> sharing the modifications here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14803) Add some debug logs to StoreFileScanner

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008869#comment-15008869
 ] 

Heng Chen commented on HBASE-14803:
---

{quote}
Also, this is only when doing scans for the same file at the same time and both 
skip it at the exact same time. 
{quote}

I see logCounter is static,  is it means all ScanFileScanner will use same 
logCounter ?  :)

> Add some debug logs to StoreFileScanner
> ---
>
> Key: HBASE-14803
> URL: https://issues.apache.org/jira/browse/HBASE-14803
> Project: HBase
>  Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
>  Labels: beginner
> Fix For: 1.2.0
>
> Attachments: HBASE-14803.v0-trunk.patch, HBASE-14803.v1-trunk.patch, 
> HBASE-14803.v2-trunk.patch, HBASE-14803.v3-trunk.patch, 
> HBASE-14803.v4-trunk.patch, HBASE-14803.v4-trunk.patch, 
> HBASE-14803.v5-trunk.patch
>
>
> To validate some behaviors I had to add some logs into StoreFileScanner.
> I think it can be interesting for other people looking for debuging. So 
> sharing the modifications here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14812) Fix ResultBoundedCompletionService deadlock

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008892#comment-15008892
 ] 

stack commented on HBASE-14812:
---

It working? 

Moving the notify of tasks outside check doesn't seem right:

}
75  synchronized (tasks) {
76tasks.notify();
76  }   77  }
77} 78}

Or, is it going to make for excessive notifying (Not looking at full context)  
Don't you want to notify only if completed is set.

Otherwise, LGTM. +1



> Fix ResultBoundedCompletionService deadlock
> ---
>
> Key: HBASE-14812
> URL: https://issues.apache.org/jira/browse/HBASE-14812
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Thrift
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14812-v1.patch, HBASE-14812.patch
>
>
> {code}
> "thrift2-worker-0" #31 daemon prio=5 os_prio=0 tid=0x7f5ad9c45000 
> nid=0x484 in Object.wait() [0x7f5aa3832000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> - locked <0x0006b6ae7670> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1276)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1182)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:194)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:171)
> - locked <0x0006b6ae79c0> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
> at 
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1033)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler.putMultiple(ThriftHBaseServiceHandler.java:268)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler$THBaseServiceMetricsProxy.invoke(ThriftHBaseServiceHandler.java:114)
> at com.sun.proxy.$Proxy10.putMultiple(Unknown Source)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1637)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1621)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at 
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14803) Add some debug logs to StoreFileScanner

[Jean-Marc Spaggiari (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008889#comment-15008889
 ] 

Jean-Marc Spaggiari commented on HBASE-14803:
-

Indeed. Goal is to limit the number of messages logged by this class overall 
when turning on debug mode on heavy scan tests. We do not want to keep one 
counter per region. That will just create to many non-required counters. Here 
we just print that we are skipping some messages and wait a bit before login 
again.

Like in the example copied from above
{quote}
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): d0a0169a565f442d9ed33f730f651503 
skipped because of Bloom Filter.
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): 78a1ffc4559c4691a849e284ce6f36d9 
skipped because of Bloom Filter.
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): d5ea38b1fb9041598eac8db5c5dba954 
skipped because of Bloom Filter.
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): c88d318f87494de5b758d5fc205b9156 
skipped because of Bloom Filter.
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): d6903214874542baad7681888eee57eb 
skipped because of Bloom Filter.
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): 4fa83becd20f45859fcda7c4457b7f6a 
skipped because of Bloom Filter.
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): 4f9d6165a4944aa5ab4afc0b22d564d3 
skipped because of Bloom Filter.
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): 58d8fdf501ae4138ad8530d0d28e2b9b 
skipped because of Bloom Filter.
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): 5dedfb8908494ae5b60e6aa1c8b1b7ad 
skipped because of Bloom Filter.
2015-11-16 09:06:15,347 DEBUG [] r.SFS(479): 7486989f4b574f0ba26534ff2d01b2e8 
skipped because of Bloom Filter.
2015-11-16 09:06:15,348 DEBUG [] r.SFS(441): To many messages logged. Skipping 
new messages.
{quote}
As you can see it's almost always a different store file, etc. We just want to 
reduce the display, what ever ScanFileScanner it is on this RS.

Keep in mind that this is debug information only. We are not saying "This 
specific ScanFileScanner here logged to many lines". Just that ScanFileScanner 
in general logged to many.

So I don't think we should have one counter per instance. But here again, it's 
just 7 bytes to remove on 2 lines if we think we prefer it differently. But 
then on the tests side we might still create many of them and log to much. This 
will have to be tested.

> Add some debug logs to StoreFileScanner
> ---
>
> Key: HBASE-14803
> URL: https://issues.apache.org/jira/browse/HBASE-14803
> Project: HBase
>  Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
>  Labels: beginner
> Fix For: 1.2.0
>
> Attachments: HBASE-14803.v0-trunk.patch, HBASE-14803.v1-trunk.patch, 
> HBASE-14803.v2-trunk.patch, HBASE-14803.v3-trunk.patch, 
> HBASE-14803.v4-trunk.patch, HBASE-14803.v4-trunk.patch, 
> HBASE-14803.v5-trunk.patch
>
>
> To validate some behaviors I had to add some logs into StoreFileScanner.
> I think it can be interesting for other people looking for debuging. So 
> sharing the modifications here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008907#comment-15008907
 ] 

stack commented on HBASE-14793:
---

When G1GC, there is no L2 going on?  If there is an L2, the L1 only has index 
blocksand an index block in L1 needs limiting?  Must be the former case? 
Its good to have a limit regardless.

+1 on patch.

Add release note honey.





> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14803) Add some debug logs to StoreFileScanner

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008906#comment-15008906
 ] 

Heng Chen commented on HBASE-14803:
---

{quote}
 We do not want to keep one counter per region. That will just create to many 
non-required counters.
{quote}

Agree.  My point is if all StoreFileScanner share logCounter,  and logCounter 
type is not Atomic,  is it will be heavy concurrency conflicts?

> Add some debug logs to StoreFileScanner
> ---
>
> Key: HBASE-14803
> URL: https://issues.apache.org/jira/browse/HBASE-14803
> Project: HBase
>  Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
>  Labels: beginner
> Fix For: 1.2.0
>
> Attachments: HBASE-14803.v0-trunk.patch, HBASE-14803.v1-trunk.patch, 
> HBASE-14803.v2-trunk.patch, HBASE-14803.v3-trunk.patch, 
> HBASE-14803.v4-trunk.patch, HBASE-14803.v4-trunk.patch, 
> HBASE-14803.v5-trunk.patch
>
>
> To validate some behaviors I had to add some logs into StoreFileScanner.
> I think it can be interesting for other people looking for debuging. So 
> sharing the modifications here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14824) HBaseAdmin.mergeRegions should use full region names instead of encoded region names

[Eungsop Yoo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eungsop Yoo updated HBASE-14824:

Status: Open  (was: Patch Available)

> HBaseAdmin.mergeRegions should use full region names instead of encoded 
> region names
> 
>
> Key: HBASE-14824
> URL: https://issues.apache.org/jira/browse/HBASE-14824
> Project: HBase
>  Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14824.patch
>
>
> HBaseAdmin.mergeRegions() calls HBaseAdmin.getRegion() internally. 
> HBaseAdmin.getRegion() requires the full region name. So 
> MetaTableAccessor.getRegion always returns null and this causes one more meta 
> table scan.
> {code}
>   Pair getRegion(final byte[] regionName) throws 
> IOException {
> if (regionName == null) {
>   throw new IllegalArgumentException("Pass a table name or region name");
> }
> Pair pair =
>   MetaTableAccessor.getRegion(connection, regionName);
> if (pair == null) {
> {code}
> I suppose to use full region names instead of encoded region names in 
> HBaseAdmin.mergeRegions().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14824) HBaseAdmin.mergeRegions should use full region names instead of encoded region names

[Eungsop Yoo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eungsop Yoo updated HBASE-14824:

Attachment: HBASE-14824-v1.patch

> HBaseAdmin.mergeRegions should use full region names instead of encoded 
> region names
> 
>
> Key: HBASE-14824
> URL: https://issues.apache.org/jira/browse/HBASE-14824
> Project: HBase
>  Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14824-v1.patch, HBASE-14824.patch
>
>
> HBaseAdmin.mergeRegions() calls HBaseAdmin.getRegion() internally. 
> HBaseAdmin.getRegion() requires the full region name. So 
> MetaTableAccessor.getRegion always returns null and this causes one more meta 
> table scan.
> {code}
>   Pair getRegion(final byte[] regionName) throws 
> IOException {
> if (regionName == null) {
>   throw new IllegalArgumentException("Pass a table name or region name");
> }
> Pair pair =
>   MetaTableAccessor.getRegion(connection, regionName);
> if (pair == null) {
> {code}
> I suppose to use full region names instead of encoded region names in 
> HBaseAdmin.mergeRegions().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14824) HBaseAdmin.mergeRegions should use full region names instead of encoded region names

[Eungsop Yoo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eungsop Yoo updated HBASE-14824:

Status: Patch Available  (was: Open)

I implemented Option 1.

> HBaseAdmin.mergeRegions should use full region names instead of encoded 
> region names
> 
>
> Key: HBASE-14824
> URL: https://issues.apache.org/jira/browse/HBASE-14824
> Project: HBase
>  Issue Type: Bug
>Reporter: Eungsop Yoo
>Priority: Minor
> Attachments: HBASE-14824-v1.patch, HBASE-14824.patch
>
>
> HBaseAdmin.mergeRegions() calls HBaseAdmin.getRegion() internally. 
> HBaseAdmin.getRegion() requires the full region name. So 
> MetaTableAccessor.getRegion always returns null and this causes one more meta 
> table scan.
> {code}
>   Pair getRegion(final byte[] regionName) throws 
> IOException {
> if (regionName == null) {
>   throw new IllegalArgumentException("Pass a table name or region name");
> }
> Pair pair =
>   MetaTableAccessor.getRegion(connection, regionName);
> if (pair == null) {
> {code}
> I suppose to use full region names instead of encoded region names in 
> HBaseAdmin.mergeRegions().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache

[Hiroshi Ikeda (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008517#comment-15008517
 ] 

Hiroshi Ikeda commented on HBASE-14708:
---

Certainly it is not trivial to think out a rule to make consistent between 
counter, internal concurrent map and its array cache, with just using CAS. That 
has been done with encapsulating the details in some methods, and it is clear 
how to implement. It is also convenient that you can anytime use the internal 
concurrent map for read operations.

Accessing the counter with multiple threads cause overhead, and it is 
observable in the benchmark result, but it is still 7.5 times faster than 
copy-on-write array implementation. As to read operations, once a read cache is 
created there is no difference except overhead of calling a method (which can 
be optimized by VM, I think) and additional null check.

> Use copy on write Map for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, 
> HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v15.patch, 
> HBASE-14708-v16.patch, HBASE-14708-v17.patch, HBASE-14708-v2.patch, 
> HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, 
> HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, 
> HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, anotherbench2.zip, 
> location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase. 
> > 60% of the time was spent in locating a region. This was while the cluster 
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a 
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache. 
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for 
> reading and writing.
> However most operations will not need to remove or add a region location. 
> There will be potentially several orders of magnitude more reads for cached 
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14826) Small improvement in KVHeap seek() API

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008540#comment-15008540
 ] 

ramkrishna.s.vasudevan commented on HBASE-14826:


Take this case where we have two storefiles (one column family and 2 qualifiers)
Storefile 1 - row0, row1, row2, row3, row5 (all with qual 1)
Storefile 2 : row4 and row6 (with qual 2)

Now after fetching row3 if we try to reseek to row4 - the scanner on store file 
1 would have already fetched row5 and so that next() call would have updated 
the current to the scanner on storefile 2. 
So the reseek to row4 will start on that scanner on storefile 2.
Now if we suppose fetch row2 and then reseek to row4, now the scanner on 
storefile1 will be in row3 so when we do reseek it will see that row3 in that 
scanner is lesser than row4 (reseeked row) and so it will try to get the next 
scanner from the heap (this is as part of reseek). So every time reseek() doing 
an PQ.add(current) can be avoided. 
Let me try the hadoopQA and see if there is something wrong in my point.


> Small improvement in KVHeap seek() API
> --
>
> Key: HBASE-14826
> URL: https://issues.apache.org/jira/browse/HBASE-14826
> Project: HBase
>  Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-14826.patch
>
>
> Currently in seek/reseek() APIs we tend to do lot of priorityqueue related 
> operations. We initially add the current scanner to the heap, then poll and 
> again add the scanner back if the seekKey is greater than the topkey in that 
> scanner. Since the KVs are always going to be in increasing order and in 
> ideal scan flow every seek/reseek is followed by a next() call it should be 
> ok if we start with checking the current scanner and then do a poll to get 
> the next scanner. Just avoid the initial PQ.add(current) call. This could 
> save some comparisons. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14826) Small improvement in KVHeap seek() API

[ramkrishna.s.vasudevan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-14826:
---
Assignee: ramkrishna.s.vasudevan
  Status: Patch Available  (was: Open)

> Small improvement in KVHeap seek() API
> --
>
> Key: HBASE-14826
> URL: https://issues.apache.org/jira/browse/HBASE-14826
> Project: HBase
>  Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-14826.patch
>
>
> Currently in seek/reseek() APIs we tend to do lot of priorityqueue related 
> operations. We initially add the current scanner to the heap, then poll and 
> again add the scanner back if the seekKey is greater than the topkey in that 
> scanner. Since the KVs are always going to be in increasing order and in 
> ideal scan flow every seek/reseek is followed by a next() call it should be 
> ok if we start with checking the current scanner and then do a poll to get 
> the next scanner. Just avoid the initial PQ.add(current) call. This could 
> save some comparisons. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14826) Small improvement in KVHeap seek() API

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008829#comment-15008829
 ] 

Hadoop QA commented on HBASE-14826:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12772726/HBASE-14826.patch
  against master branch at commit db42fc64f1e54ab8b47799eae7920f7298a5a406.
  ATTACHMENT ID: 12772726

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16550//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16550//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16550//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16550//console

This message is automatically generated.

> Small improvement in KVHeap seek() API
> --
>
> Key: HBASE-14826
> URL: https://issues.apache.org/jira/browse/HBASE-14826
> Project: HBase
>  Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-14826.patch
>
>
> Currently in seek/reseek() APIs we tend to do lot of priorityqueue related 
> operations. We initially add the current scanner to the heap, then poll and 
> again add the scanner back if the seekKey is greater than the topkey in that 
> scanner. Since the KVs are always going to be in increasing order and in 
> ideal scan flow every seek/reseek is followed by a next() call it should be 
> ok if we start with checking the current scanner and then do a poll to get 
> the next scanner. Just avoid the initial PQ.add(current) call. This could 
> save some comparisons. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14803) Add some debug logs to StoreFileScanner

[Jean-Marc Spaggiari (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008860#comment-15008860
 ] 

Jean-Marc Spaggiari commented on HBASE-14803:
-

I was hesitant about the AtomicLong. It creates 2 other objects to guarantee 
atomicity. But here, we don't really care about atomicity. If we display 11 
lines of logs instead of 10 because they have been a conflict, it doesn't 
really matters. Also, this is only when doing scans for the same file at the 
same time and both skip it at the exact same time. Chances for this to occurs 
are very small. That's why I went with long instead of AtomicLong.

Now, if we want AtomicLong instead, it's really not a big change ;) I can do 
that. We just need to decide on what we prefer (And at the same time I will 
fix "To" into "Too"...)

> Add some debug logs to StoreFileScanner
> ---
>
> Key: HBASE-14803
> URL: https://issues.apache.org/jira/browse/HBASE-14803
> Project: HBase
>  Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
>  Labels: beginner
> Fix For: 1.2.0
>
> Attachments: HBASE-14803.v0-trunk.patch, HBASE-14803.v1-trunk.patch, 
> HBASE-14803.v2-trunk.patch, HBASE-14803.v3-trunk.patch, 
> HBASE-14803.v4-trunk.patch, HBASE-14803.v4-trunk.patch, 
> HBASE-14803.v5-trunk.patch
>
>
> To validate some behaviors I had to add some logs into StoreFileScanner.
> I think it can be interesting for other people looking for debuging. So 
> sharing the modifications here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14815) TestMobExportSnapshot.testExportFailure timeout occasionally

[Heng Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heng Chen updated HBASE-14815:
--
Description: 
On master,  TestMobExportSnapshot.testExportFailure timeout occasionally.

See
https://builds.apache.org/job/PreCommit-HBASE-Build/16514//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/

https://builds.apache.org/job/PreCommit-HBASE-Build/16511//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/

  was:
On master,  TestMobExportSnapshot.testExportFailure timeout occasionally.

See
https://builds.apache.org/job/PreCommit-HBASE-Build/16514//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/


> TestMobExportSnapshot.testExportFailure timeout occasionally
> 
>
> Key: HBASE-14815
> URL: https://issues.apache.org/jira/browse/HBASE-14815
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14815.patch
>
>
> On master,  TestMobExportSnapshot.testExportFailure timeout occasionally.
> See
> https://builds.apache.org/job/PreCommit-HBASE-Build/16514//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/
> https://builds.apache.org/job/PreCommit-HBASE-Build/16511//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14826) Small improvement in KVHeap seek() API

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008849#comment-15008849
 ] 

Ted Yu commented on HBASE-14826:


lgtm

> Small improvement in KVHeap seek() API
> --
>
> Key: HBASE-14826
> URL: https://issues.apache.org/jira/browse/HBASE-14826
> Project: HBase
>  Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-14826.patch
>
>
> Currently in seek/reseek() APIs we tend to do lot of priorityqueue related 
> operations. We initially add the current scanner to the heap, then poll and 
> again add the scanner back if the seekKey is greater than the topkey in that 
> scanner. Since the KVs are always going to be in increasing order and in 
> ideal scan flow every seek/reseek is followed by a next() call it should be 
> ok if we start with checking the current scanner and then do a poll to get 
> the next scanner. Just avoid the initial PQ.add(current) call. This could 
> save some comparisons. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008673#comment-15008673
 ] 

Sean Busbey commented on HBASE-14708:
-

let's go with the simple change for 1.2 and then open a follow on ticket(s) 
that define a macro benchmark and try out the more complicated version.

> Use copy on write Map for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, 
> HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v15.patch, 
> HBASE-14708-v16.patch, HBASE-14708-v17.patch, HBASE-14708-v2.patch, 
> HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, 
> HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, 
> HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, anotherbench2.zip, 
> location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase. 
> > 60% of the time was spent in locating a region. This was while the cluster 
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a 
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache. 
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for 
> reading and writing.
> However most operations will not need to remove or add a region location. 
> There will be potentially several orders of magnitude more reads for cached 
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14826) Small improvement in KVHeap seek() API

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008678#comment-15008678
 ] 

Heng Chen commented on HBASE-14826:
---

Sound reasonable.  
A little hint.   Last 'if' in patch seems not necessary。
{code}
+  if(scanner == null) {
+current = null;
+return false;
+  }
{code}

If  scanner is null,  loop will exist, and return false. Of course, we should 
set current to be null.

> Small improvement in KVHeap seek() API
> --
>
> Key: HBASE-14826
> URL: https://issues.apache.org/jira/browse/HBASE-14826
> Project: HBase
>  Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-14826.patch
>
>
> Currently in seek/reseek() APIs we tend to do lot of priorityqueue related 
> operations. We initially add the current scanner to the heap, then poll and 
> again add the scanner back if the seekKey is greater than the topkey in that 
> scanner. Since the KVs are always going to be in increasing order and in 
> ideal scan flow every seek/reseek is followed by a next() call it should be 
> ok if we start with checking the current scanner and then do a poll to get 
> the next scanner. Just avoid the initial PQ.add(current) call. This could 
> save some comparisons. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14806) Missing sources.jar for several modules when building HBase

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009020#comment-15009020
 ] 

Hudson commented on HBASE-14806:


FAILURE: Integrated in HBase-0.98-matrix #260 (See 
[https://builds.apache.org/job/HBase-0.98-matrix/260/])
HBASE-14806 Missing sources.jar for several modules when building HBase 
(zhangduo: rev 5bd078050cb6fc7f7a3b57c594ef3c7dabfbcd1c)
* hbase-common/pom.xml


> Missing sources.jar for several modules when building HBase
> ---
>
> Key: HBASE-14806
> URL: https://issues.apache.org/jira/browse/HBASE-14806
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.17, 1.0.4
>
> Attachments: HBASE-14806.patch
>
>
> Introduced by HBASE-14085. The problem is, for example, in 
> hbase-common/pom.xml, we have
> {code:title=pom.xml}
> 
>   org.apache.maven.plugins
>   maven-source-plugin
>   
> true
> 
>   src/main/java
>   ${project.build.outputDirectory}/META-INF
> 
>   
> 
> {code}
> But in fact, the path inside {{}} tag is relative to source 
> directories, not the project directory. So the maven-source-plugin always end 
> with
> {noformat}
> No sources in project. Archive not created.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14806) Missing sources.jar for several modules when building HBase

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009000#comment-15009000
 ] 

Hudson commented on HBASE-14806:


FAILURE: Integrated in HBase-1.0 #1112 (See 
[https://builds.apache.org/job/HBase-1.0/1112/])
HBASE-14806 Missing sources.jar for several modules when building HBase 
(zhangduo: rev 193659eaf772c267dad2e637b3011b36624809e4)
* hbase-common/pom.xml


> Missing sources.jar for several modules when building HBase
> ---
>
> Key: HBASE-14806
> URL: https://issues.apache.org/jira/browse/HBASE-14806
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.17, 1.0.4
>
> Attachments: HBASE-14806.patch
>
>
> Introduced by HBASE-14085. The problem is, for example, in 
> hbase-common/pom.xml, we have
> {code:title=pom.xml}
> 
>   org.apache.maven.plugins
>   maven-source-plugin
>   
> true
> 
>   src/main/java
>   ${project.build.outputDirectory}/META-INF
> 
>   
> 
> {code}
> But in fact, the path inside {{}} tag is relative to source 
> directories, not the project directory. So the maven-source-plugin always end 
> with
> {noformat}
> No sources in project. Archive not created.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14793) Allow limiting size of block into L1 block cache.

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14793:
--
Attachment: HBASE-14793-v7.patch

Fixed the boolean counting issue.

> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793-v7.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14793) Allow limiting size of block into L1 block cache.

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14793:
--
Release Note: Very large blocks can fragment the heap and cause bad issues 
for the garbage collector, especially the G1GC. Now there is a maximum size 
that a block can be and still stick in the LruBlockCache. That size defaults to 
16mb but can be controlled by changing "hbase.lru.max.block.size"

> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793-v7.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009015#comment-15009015
 ] 

Elliott Clark commented on HBASE-14793:
---

bq.When G1GC, there is no L2 going on?
That's one of the things we tried. A larger on heap cache with the g1.

bq.Add release note honey.
Will do honey.

> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14826) Small improvement in KVHeap seek() API

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008553#comment-15008553
 ] 

ramkrishna.s.vasudevan commented on HBASE-14826:


Using the benchmark class attached in HBASE-9969 and testing reseek with and 
without the patch


With patch

next/reseek,numColsPerRow,prefixLength,impl,numScanners,opsPerSec(K),topScannerNull(K),next
 comparisons(K),heap comparisons(K),GS comparisons(K)
reseek,16,0,KVHeap,1,2987,0,0,-0,0
reseek,16,0,KVHeap,2,2724,0,0,-0,0
reseek,16,0,KVHeap,4,3329,0,0,-0,0
reseek,16,0,KVHeap,8,4333,0,0,-0,0
reseek,16,0,KVHeap,16,3495,0,0,-0,0
reseek,16,0,KVHeap,32,3361,0,0,-0,0

Without patch
==
next/reseek,numColsPerRow,prefixLength,impl,numScanners,opsPerSec(K),topScannerNull(K),next
 comparisons(K),heap comparisons(K),GS comparisons(K)
reseek,16,0,KVHeap,1,2539,0,0,-0,0
reseek,16,0,KVHeap,2,2118,0,0,-0,0
reseek,16,0,KVHeap,4,2675,0,0,-0,0
reseek,16,0,KVHeap,8,3628,0,0,-0,0
reseek,16,0,KVHeap,16,2583,0,0,-0,0
reseek,16,0,KVHeap,32,2284,0,0,-0,0

Where the 6th col indicates the opsPerSec.  We can see a gain of 7% to 10%

> Small improvement in KVHeap seek() API
> --
>
> Key: HBASE-14826
> URL: https://issues.apache.org/jira/browse/HBASE-14826
> Project: HBase
>  Issue Type: Improvement
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Attachments: HBASE-14826.patch
>
>
> Currently in seek/reseek() APIs we tend to do lot of priorityqueue related 
> operations. We initially add the current scanner to the heap, then poll and 
> again add the scanner back if the seekKey is greater than the topkey in that 
> scanner. Since the KVs are always going to be in increasing order and in 
> ideal scan flow every seek/reseek is followed by a next() call it should be 
> ok if we start with checking the current scanner and then do a poll to get 
> the next scanner. Just avoid the initial PQ.add(current) call. This could 
> save some comparisons. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008636#comment-15008636
 ] 

Ted Yu commented on HBASE-13153:


+1 from me as well

> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, 
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, 
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, 
> HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, 
> HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, 
> HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk 
> Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk 
> Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster 
> tolerance scenario.But we encounter an issue that we will use bulkload very 
> frequently,because bulkload bypass write path, and will not generate WAL, so 
> the data will not be replicated to backup cluster. It's inappropriate to 
> bukload twice both on active cluster and backup cluster. So i advise do some 
> modification to bulkload feature to enable bukload to both active cluster and 
> backup cluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14799) Commons-collections object deserialization remote command execution vulnerability

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009177#comment-15009177
 ] 

Andrew Purtell commented on HBASE-14799:


Upgrading commons-collection will close the immediate vulnerability but not the 
door on other possible vulnerable serializable classes on our classpath. 

We could split the difference as suggested: Upgrade commons-collection 
everywhere, then patch 0.98 to disable object serialization in that legacy 
HbaseObjectWritable by default. 

> Commons-collections object deserialization remote command execution 
> vulnerability 
> --
>
> Key: HBASE-14799
> URL: https://issues.apache.org/jira/browse/HBASE-14799
> Project: HBase
>  Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.94.28, 0.98.17
>
> Attachments: HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, 
> HBASE-14799-0.94.patch, HBASE-14799-0.98.patch
>
>
> Read: 
> http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/
> TL;DR: If you have commons-collections on your classpath and accept and 
> process Java object serialization data, then you probably have an exploitable 
> remote command execution vulnerability. 
> 0.94 and earlier HBase releases are vulnerable because we might read in and 
> rehydrate serialized Java objects out of RPC packet data in 
> HbaseObjectWritable using ObjectInputStream#readObject (see 
> https://hbase.apache.org/0.94/xref/org/apache/hadoop/hbase/io/HbaseObjectWritable.html#714)
>  and we have commons-collections on the classpath on the server.
> 0.98 also carries some limited exposure to this problem through inclusion of 
> backwards compatible deserialization code in 
> HbaseObjectWritableFor96Migration. This is used by the 0.94-to-0.98 migration 
> utility, and by the AccessController when reading permissions from the ACL 
> table serialized in legacy format by 0.94. Unprivileged users cannot run the 
> tool nor access the ACL table.
> Unprivileged users can however attack a 0.94 installation. An attacker might 
> be able to use the method discussed on that blog post to capture valid HBase 
> RPC payloads for 0.94 and prior versions, rewrite them to embed an exploit, 
> and replay them to trigger a remote command execution with the privileges of 
> the account under which the HBase RegionServer daemon is running.
> We need to make a patch release of 0.94 that changes HbaseObjectWritable to 
> disallow processing of random Java object serializations. This will be a 
> compatibility break that might affect old style coprocessors, which quite 
> possibly may rely on this catch-all in HbaseObjectWritable for custom object 
> (de)serialization. We can introduce a new configuration setting, 
> "hbase.allow.legacy.object.serialization", defaulting to false.
> To be thorough, we can also use the new configuration setting  
> "hbase.allow.legacy.object.serialization" (defaulting to false) in 0.98 to 
> prevent the AccessController from falling back to the vulnerable legacy code. 
> This turns out to not affect the ability to migrate permissions because 
> TablePermission implements Writable, which is safe, not Serializable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14799) Commons-collections object deserialization remote command execution vulnerability

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009124#comment-15009124
 ] 

Jonathan Hsieh commented on HBASE-14799:


+1 for the 0.98 patch. 


For the 0.94, Though not ideal could we make it completely modal?
1) defaulting to allowing legacy true and using the old pair serialization to 
if true
2) have the legacy setting to false and use the new pair serialization 
mechanism.  (would we be able to roll forward like this or would this require 
full shutdown?) 

We'd also want documentation about the limitation in the last 0.94 release 
notes. 

One other suggestion -- 3.2.2 came out upstream[1].  Would upgrading to it 
alone be sufficient? 


[1] https://commons.apache.org/proper/commons-collections/release_3_2_2.html

> Commons-collections object deserialization remote command execution 
> vulnerability 
> --
>
> Key: HBASE-14799
> URL: https://issues.apache.org/jira/browse/HBASE-14799
> Project: HBase
>  Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.94.28, 0.98.17
>
> Attachments: HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, 
> HBASE-14799-0.94.patch, HBASE-14799-0.98.patch
>
>
> Read: 
> http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/
> TL;DR: If you have commons-collections on your classpath and accept and 
> process Java object serialization data, then you probably have an exploitable 
> remote command execution vulnerability. 
> 0.94 and earlier HBase releases are vulnerable because we might read in and 
> rehydrate serialized Java objects out of RPC packet data in 
> HbaseObjectWritable using ObjectInputStream#readObject (see 
> https://hbase.apache.org/0.94/xref/org/apache/hadoop/hbase/io/HbaseObjectWritable.html#714)
>  and we have commons-collections on the classpath on the server.
> 0.98 also carries some limited exposure to this problem through inclusion of 
> backwards compatible deserialization code in 
> HbaseObjectWritableFor96Migration. This is used by the 0.94-to-0.98 migration 
> utility, and by the AccessController when reading permissions from the ACL 
> table serialized in legacy format by 0.94. Unprivileged users cannot run the 
> tool nor access the ACL table.
> Unprivileged users can however attack a 0.94 installation. An attacker might 
> be able to use the method discussed on that blog post to capture valid HBase 
> RPC payloads for 0.94 and prior versions, rewrite them to embed an exploit, 
> and replay them to trigger a remote command execution with the privileges of 
> the account under which the HBase RegionServer daemon is running.
> We need to make a patch release of 0.94 that changes HbaseObjectWritable to 
> disallow processing of random Java object serializations. This will be a 
> compatibility break that might affect old style coprocessors, which quite 
> possibly may rely on this catch-all in HbaseObjectWritable for custom object 
> (de)serialization. We can introduce a new configuration setting, 
> "hbase.allow.legacy.object.serialization", defaulting to false.
> To be thorough, we can also use the new configuration setting  
> "hbase.allow.legacy.object.serialization" (defaulting to false) in 0.98 to 
> prevent the AccessController from falling back to the vulnerable legacy code. 
> This turns out to not affect the ability to migrate permissions because 
> TablePermission implements Writable, which is safe, not Serializable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14812) Fix ResultBoundedCompletionService deadlock

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14812:
--
Attachment: HBASE-14812-v2.patch

Attached patch with more explanation on why we're notifying more.

So far this looks to have fixed it.

> Fix ResultBoundedCompletionService deadlock
> ---
>
> Key: HBASE-14812
> URL: https://issues.apache.org/jira/browse/HBASE-14812
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Thrift
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14812-v1.patch, HBASE-14812-v2.patch, 
> HBASE-14812.patch
>
>
> {code}
> "thrift2-worker-0" #31 daemon prio=5 os_prio=0 tid=0x7f5ad9c45000 
> nid=0x484 in Object.wait() [0x7f5aa3832000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> - locked <0x0006b6ae7670> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1276)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1182)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:194)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:171)
> - locked <0x0006b6ae79c0> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
> at 
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1033)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler.putMultiple(ThriftHBaseServiceHandler.java:268)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler$THBaseServiceMetricsProxy.invoke(ThriftHBaseServiceHandler.java:114)
> at com.sun.proxy.$Proxy10.putMultiple(Unknown Source)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1637)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1621)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at 
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14820) Region becomes unavailable after a region split is rolled back

[Clara Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009219#comment-15009219
 ] 

Clara Xiong commented on HBASE-14820:
-

2015-11-16 07:26:43,434 DEBUG [MemStoreFlusher.1] 
regionserver.CompactSplitThread: Split requested for 
[TableName],\x85\xDF`\x7F\xFF\xFE\xCE 
,\xB6{\xD0\xA5\x15'\xD7\xD6g\x8E_k\xB5\x13\x16g%\x9D\x06\xC5q
\xCB!\xBFI\x01,1447477545041.4c467284476f9627dd8e7a4f34dfd31a..  
compaction_queue=(0:0), split_queue=0, merge_queue=0
2015-11-16 07:26:43,788 DEBUG [regionserver60020-splits-1447687603434] 
lock.ZKInterProcessLockBase: Acquired a lock for 
/hbase/table-lock/[TableName]/read-regionserver:60020009873
2015-11-16 07:26:43,789 INFO  [regionserver60020-splits-1447687603434] 
regionserver.SplitTransaction: Starting split of region 
[TableName],\x85\xDF`\x7F\xFF\xFE\xCE ,\xB6{\xD0\xA5\x15'\xD7\xD6g\x8E_k\xB
5\x13\x16g%\x9D\x06\xC5q\xCB!\xBFI\x01,1447477545041.4c467284476f9627dd8e7a4f34dfd31a.
2015-11-16 07:26:43,789 DEBUG [regionserver60020-splits-1447687603434] 
regionserver.SplitTransaction: regionserver:60020-0x1503f523d9f032a, 
quorum=hm4.[Host]:2181,hm3.[Host]:
2181,hm2.[Host]:2181,hm1.[Host]:2181,hm5.[Host]:2181, baseZNode=/hbase Creating 
ephemeral node for 4c467284476f9627dd8e7a4f34dfd31a in PENDING_SPLIT state
2015-11-16 07:26:43,790 DEBUG [regionserver60020-splits-1447687603434] 
regionserver.SplitTransaction: Still waiting for master to process the 
pending_split for 4c467284476f9627dd8e7a4f34dfd31a
2015-11-16 07:26:43,790 DEBUG [regionserver60020-splits-1447687603434] 
zookeeper.ZKAssign: regionserver:60020-0x1503f523d9f032a, 
quorum=hm4.[Host]:2181,hm3.[Host]:2181,hm2.[Host][Host]:2181,hm1.[Host]:2181,hm5.[Host]:2181,
 baseZNode=/hbase Transitioning 4c467284476f9627dd8e7a4f34dfd31a from 
RS_ZK_REQUEST_REGION_SPLIT to RS_ZK_REQUEST_REGI
ON_SPLIT
2015-11-16 07:26:43,791 DEBUG [regionserver60020-splits-1447687603434] 
zookeeper.ZKAssign: regionserver:60020-0x1503f523d9f032a, 
quorum=hm4.[Host]:2181,hm3.[Host]:2181,hm2.[Host]:2181,hm1.[Host]:2181,hm5.[Host]:2181,
 baseZNode=/hbase Transitioned node 4c467284476f9627dd8e7a4f34dfd31a from 
RS_ZK_REQUEST_REGION_SPLIT to RS_ZK_REQUEST_
REGION_SPLIT
2015-11-16 07:26:43,986 DEBUG [regionserver60020-splits-1447687603434] 
regionserver.HRegion: Closing [TableName],\x85\xDF`\x7F\xFF\xFE\xCE 
,\xB6{\xD0\xA5\x15'\xD7\xD6g\x8E_k\xB5\x13\x16g%\x9D\x06\xC5q\x
CB!\xBFI\x01,1447477545041.4c467284476f9627dd8e7a4f34dfd31a.: disabling 
compactions & flushes
2015-11-16 07:26:43,986 DEBUG [regionserver60020-splits-1447687603434] 
regionserver.HRegion: Updates disabled for region 
[TableName],\x85\xDF`\x7F\xFF\xFE\xCE 
,\xB6{\xD0\xA5\x15'\xD7\xD6g\x8E_k\xB5\x13\
x16g%\x9D\x06\xC5q\xCB!\xBFI\x01,1447477545041.4c467284476f9627dd8e7a4f34dfd31a.
2015-11-16 07:26:43,986 INFO  [regionserver60020-splits-1447687603434] 
regionserver.HRegion: Started memstore flush for 
[TableName],\x85\xDF`\x7F\xFF\xFE\xCE 
,\xB6{\xD0\xA5\x15'\xD7\xD6g\x8E_k\xB5\x13\x
16g%\x9D\x06\xC5q\xCB!\xBFI\x01,1447477545041.4c467284476f9627dd8e7a4f34dfd31a.,
 current region memstore size 363.0 K
2015-11-16 07:26:44,527 INFO  [regionserver60020-splits-1447687603434] 
regionserver.DefaultStoreFlusher: Flushed, sequenceid=12742653, memsize=363.0 
K, hasBloomFilter=false, into tmp file 
hdfs://hm1.[Host]]:9000/hbase/data/default/[TableName]/4c467284476f9627dd8e7a4f34dfd31a/.tmp/fe0ca3b83b794cef8c1cb357ccc08db6
2015-11-16 07:26:44,532 DEBUG [regionserver60020-splits-1447687603434] 
regionserver.HRegionFileSystem: Committing store file 
hdfs://hm1.[Host]:9000/hbase/data/default/[TableName]/4c4672844
76f9627dd8e7a4f34dfd31a/.tmp/fe0ca3b83b794cef8c1cb357ccc08db6 as 
hdfs://hm1.[Host]:9000/hbase/data/default/[TableName]/4c467284476f9627dd8e7a4f34dfd31a/e/fe0ca3b83b794cef8c1cb357ccc08db6
2015-11-16 07:26:44,536 INFO  [regionserver60020-splits-1447687603434] 
regionserver.HStore: Added 
hdfs://hm1.[Host]:9000/hbase/data/default/[TableName]/4c467284476f9627dd8e7a4f34dfd31a/e/f
e0ca3b83b794cef8c1cb357ccc08db6, entries=368, sequenceid=12742653, 
filesize=82.7 K
2015-11-16 07:26:44,536 INFO  [regionserver60020-splits-1447687603434] 
regionserver.HRegion: Finished memstore flush of ~363.0 K/371744, 
currentsize=0/0 for region [TableName],\x85\xDF`\x7F\xFF\xFE\xCE 
,\xB6{\xD0\xA5\x15'\xD7\xD6g\x8E_k\xB5\x13\x16g%\x9D\x06\xC5q\xCB!\xBFI\x01,1447477545041.4c467284476f9627dd8e7a4f34dfd31a.
 in 550ms, sequenceid=12742653, compaction requested=false
2015-11-16 07:26:44,672 INFO  
[StoreCloserThread-[TableName],\x85\xDF`\x7F\xFF\xFE\xCE 
,\xB6{\xD0\xA5\x15'\xD7\xD6g\x8E_k\xB5\x13\x16g%\x9D\x06\xC5q\xCB!\xBFI\x01,1447477545041.4c467284476f9627dd8e7a4f3
4dfd31a.-1] regionserver.HStore: Closed e
2015-11-16 07:26:44,673 INFO  [regionserver60020-splits-1447687603434] 
regionserver.HRegion: Closed [TableName],\x85\xDF`\x7F\xFF\xFE\xCE 
,\xB6{\xD0\xA5\x15'\xD7\xD6g\x8E_k\xB5\x13\x16g%\x9D\x06\xC5q\xC

[jira] [Updated] (HBASE-14793) Allow limiting size of block into L1 block cache.

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14793:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793-v7.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14803) Add some debug logs to StoreFileScanner

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009230#comment-15009230
 ] 

Elliott Clark commented on HBASE-14803:
---

These look much more like trace logs than debug.

> Add some debug logs to StoreFileScanner
> ---
>
> Key: HBASE-14803
> URL: https://issues.apache.org/jira/browse/HBASE-14803
> Project: HBase
>  Issue Type: Bug
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
>  Labels: beginner
> Fix For: 1.2.0
>
> Attachments: HBASE-14803.v0-trunk.patch, HBASE-14803.v1-trunk.patch, 
> HBASE-14803.v2-trunk.patch, HBASE-14803.v3-trunk.patch, 
> HBASE-14803.v4-trunk.patch, HBASE-14803.v4-trunk.patch, 
> HBASE-14803.v5-trunk.patch
>
>
> To validate some behaviors I had to add some logs into StoreFileScanner.
> I think it can be interesting for other people looking for debuging. So 
> sharing the modifications here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-13153) Bulk Loaded HFile Replication

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-13153:
---
Release Note: 
This enhances the HBase replication to support replication of bulk loaded data. 
This is configurable, by default it is set to false which means it will not 
replicate the bulk loaded data to its peer(s). To enable it set 
"hbase.replication.bulkload.enabled" to true.

Following are the additional configurations added for this enhancement,
 a. hbase.replication.cluster.id - This is manadatory to configure in cluster 
where replication for bulk loaded data is enabled. A source cluster is uniquely 
identified by sink cluster using this id. This should be configured in the 
source cluster configuration file for all the RS.
 b. hbase.replication.conf.dir - This represents the directory where all the 
active cluster's file system client configurations are defined in subfolders 
corresponding to their respective replication cluster id in peer cluster. This 
should be configured in the peer cluster configuration file for all the RS. 
Default is HBASE_CONF_DIR.
 c. hbase.replication.source.fs.conf.provider - This represents the class which 
provides the source cluster file system client configuration to peer cluster. 
This should be configured in the peer cluster configuration file for all the 
RS. Default is 
org.apache.hadoop.hbase.replication.regionserver.DefaultSourceFSConfigurationProvider

 For example: If source cluster FS client configurations are copied in peer 
cluster under directory /home/user/dc1/ then  hbase.replication.cluster.id 
should be configured as dc1 and hbase.replication.conf.dir as /home/user

Note: 
 a. Any modification to source cluster FS client configuration files in peer 
cluster side replication configuration directory then it needs to restart all 
its peer(s) cluster RS with default hbase.replication.source.fs.conf.provider.
 b. Only 'xml' type files will be loaded by the default 
hbase.replication.source.fs.conf.provider.

As part of this we have made following changes to LoadIncrementalHFiles class 
which is marked as Public and Stable class,
 a. Raised the visibility scope of LoadQueueItem class from package private to 
public.
 b. Added a new method loadHFileQueue, which loads the queue of LoadQueueItem 
into the table as per the region keys provided.

  was:
This enhances the HBase replication to support replication of bulk loaded data. 
This is configurable, by default it is set to false which means it will not 
replicate the bulk loaded data to its peer(s). To enable it set 
"hbase.replication.bulkload.enabled" to true.

Following are the additional configurations are added for this enhacement,
 a. hbase.replication.cluster.id - This is manadatory to configure in cluster 
where replication for bulk loaded data is enabled. A source cluster is uniquely 
identified by sink cluster using this id. This should be configured in the 
source cluster configuration file for all the RS.
 b. hbase.replication.conf.dir - This represents the directory where all the 
active cluster's file system client configurations are defined in their 
respective replication cluster id subfolders in peer cluster. This should be 
configured in the peer cluster configuration file for all the RS. Default 
HBASE_CONF_DIR.
 c. hbase.replication.source.fs.conf.provider - This represents the class which 
provides the source cluster file system client configuration to peer cluster. 
This should be configured in the peer cluster configuration file for all the 
RS. Default 
org.apache.hadoop.hbase.replication.regionserver.DefaultSourceFSConfigurationProvider

 For example: If source cluster FS client configurations are copied in peer 
cluster under directory /home/user/dc1/ then  hbase.replication.cluster.id 
should be configured as dc1 and hbase.replication.conf.dir as /home/user

Note: 
 a. Any modification in source cluster FS client configuration files in peer 
cluster side replication configuration directory then it needs to restart all 
its peer(s) cluster RS with default hbase.replication.source.fs.conf.provider.
 b. Only 'xml' type files will be loaded by the default 
hbase.replication.source.fs.conf.provider.

As part of this we made have made following changes to LoadIncrementalHFiles 
class which is marked as Public and Stable class,
 a. Raised the visbility scope of LoadQueueItem class from package private to 
public.
 b. Added a new method loadHFileQueue, which loads the queue of LoadQueueItem 
into the table as per the region keys provided.


> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
>  Issue Type: New Feature
>  Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix 

[jira] [Commented] (HBASE-14782) FuzzyRowFilter skips valid rows

[Vladimir Rodionov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009250#comment-15009250
 ] 

Vladimir Rodionov commented on HBASE-14782:
---

{quote}
Currently, it is impossible that fuzzyKeyMeta.length >= result.length
{quote}

That is not correct.
{code}
byte[] result =
Arrays.copyOf(fuzzyKeyBytes, length > fuzzyKeyBytes.length ? length : 
fuzzyKeyBytes.length);
{code}

result size is maximum of fuzzyKeyBytes length and last row length.

{quote}
But if it does, for example, result is '\x01\x01\x01' and fuzzyKeyMeta is
{1, 0, 0, 1}
trimTrailingZeroes will return '\x01' ?
{quote}

Result size is always equals or greater than fuzzyKeyMeta, therefore your case 
is impossible.




> FuzzyRowFilter skips valid rows
> ---
>
> Key: HBASE-14782
> URL: https://issues.apache.org/jira/browse/HBASE-14782
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14782-v3.patch, HBASE-14782.patch, 
> HBASE-14782.patch
>
>
> The issue may affect not only master branch, but previous releases as well.
> This is from one of our customers:
> {quote}
> We are experiencing a problem with the FuzzyRowFilter for HBase scan. We 
> think that it is a bug. 
> Fuzzy filter should pick a row if it matches filter criteria irrespective of 
> other rows present in table but filter is dropping a row depending on some 
> other row present in table. 
> Details/Step to reproduce/Sample outputs below: 
> Missing row key: \x9C\x00\x044\x00\x00\x00\x00 
> Causing row key: \x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX 
> Prerequisites 
> 1. Create a test table. HBase shell command -- create 'fuzzytest','d' 
> 2. Insert some test data. HBase shell commands: 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x01\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x01\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x01",'d:a','junk' 
> • put 'fuzzytest',"\x9B\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> • put 'fuzzytest',"\x9D\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> Now when you run the code, you will find \x9C\x00\x044\x00\x00\x00\x00 in 
> output because it matches filter criteria. (Refer how to run code below) 
> Insert the row key causing bug: 
> HBase shell command: put 
> 'fuzzytest',"\x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX",'d:a','junk' 
> Now when you run the code, you will not find \x9C\x00\x044\x00\x00\x00\x00 in 
> output even though it still matches filter criteria. 
> {quote}
> Verified the issue on master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14799) Commons-collections object deserialization remote command execution vulnerability

[Jonathan Hsieh (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009272#comment-15009272
 ] 

Jonathan Hsieh commented on HBASE-14799:


sgtm.

> Commons-collections object deserialization remote command execution 
> vulnerability 
> --
>
> Key: HBASE-14799
> URL: https://issues.apache.org/jira/browse/HBASE-14799
> Project: HBase
>  Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.94.28, 0.98.17
>
> Attachments: HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, 
> HBASE-14799-0.94.patch, HBASE-14799-0.98.patch
>
>
> Read: 
> http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/
> TL;DR: If you have commons-collections on your classpath and accept and 
> process Java object serialization data, then you probably have an exploitable 
> remote command execution vulnerability. 
> 0.94 and earlier HBase releases are vulnerable because we might read in and 
> rehydrate serialized Java objects out of RPC packet data in 
> HbaseObjectWritable using ObjectInputStream#readObject (see 
> https://hbase.apache.org/0.94/xref/org/apache/hadoop/hbase/io/HbaseObjectWritable.html#714)
>  and we have commons-collections on the classpath on the server.
> 0.98 also carries some limited exposure to this problem through inclusion of 
> backwards compatible deserialization code in 
> HbaseObjectWritableFor96Migration. This is used by the 0.94-to-0.98 migration 
> utility, and by the AccessController when reading permissions from the ACL 
> table serialized in legacy format by 0.94. Unprivileged users cannot run the 
> tool nor access the ACL table.
> Unprivileged users can however attack a 0.94 installation. An attacker might 
> be able to use the method discussed on that blog post to capture valid HBase 
> RPC payloads for 0.94 and prior versions, rewrite them to embed an exploit, 
> and replay them to trigger a remote command execution with the privileges of 
> the account under which the HBase RegionServer daemon is running.
> We need to make a patch release of 0.94 that changes HbaseObjectWritable to 
> disallow processing of random Java object serializations. This will be a 
> compatibility break that might affect old style coprocessors, which quite 
> possibly may rely on this catch-all in HbaseObjectWritable for custom object 
> (de)serialization. We can introduce a new configuration setting, 
> "hbase.allow.legacy.object.serialization", defaulting to false.
> To be thorough, we can also use the new configuration setting  
> "hbase.allow.legacy.object.serialization" (defaulting to false) in 0.98 to 
> prevent the AccessController from falling back to the vulnerable legacy code. 
> This turns out to not affect the ability to migrate permissions because 
> TablePermission implements Writable, which is safe, not Serializable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14791) Batch Deletes in MapReduce jobs (0.98)

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009826#comment-15009826
 ] 

Lars Hofhansl commented on HBASE-14791:
---

+1, thanks [~alexaraujo]

Going to commit now.

> Batch Deletes in MapReduce jobs (0.98)
> --
>
> Key: HBASE-14791
> URL: https://issues.apache.org/jira/browse/HBASE-14791
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.16
>Reporter: Lars Hofhansl
>Assignee: Alex Araujo
>  Labels: mapreduce
> Fix For: 0.98.17
>
> Attachments: HBASE-14791-0.98-v1.patch, HBASE-14791-0.98-v2.patch, 
> HBASE-14791-0.98.patch
>
>
> We found that some of our copy table job run for many hours, even when there 
> isn't that much data to copy.
> [~vik.karma] did his magic and found that the issue is with copying delete 
> markers (we use raw mode to also move deletes across).
> Looking at the code in 0.98 it's immediately obvious that deletes (unlike 
> puts) are not batched and hence sent to the other side one by one, causing a 
> network RTT for each delete marker.
> Looks like in trunk it's doing the right thing (using BufferedMutators for 
> all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, 
> 1.2?) issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-14172) Upgrade existing thrift binding using thrift 0.9.2 compiler.

[Josh Elser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser reassigned HBASE-14172:
--

Assignee: Josh Elser

> Upgrade existing thrift binding using thrift 0.9.2 compiler.
> 
>
> Key: HBASE-14172
> URL: https://issues.apache.org/jira/browse/HBASE-14172
> Project: HBase
>  Issue Type: Improvement
>Reporter: Srikanth Srungarapu
>Assignee: Josh Elser
>Priority: Minor
> Attachments: HBASE-14172-branch-1.patch, HBASE-14172.001.patch, 
> HBASE-14172.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14822) Renewing leases of scanners doesn't work

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009839#comment-15009839
 ] 

Lars Hofhansl commented on HBASE-14822:
---

Shame on me for not adding a test to HBASE-1.
Looking.

> Renewing leases of scanners doesn't work
> 
>
> Key: HBASE-14822
> URL: https://issues.apache.org/jira/browse/HBASE-14822
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14
>Reporter: Samarth Jain
>Assignee: Lars Hofhansl
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14812) Fix ResultBoundedCompletionService deadlock

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009745#comment-15009745
 ] 

stack commented on HBASE-14812:
---

This is my test and it is flakey, working on it elsewhere. Unrelated to this 
patch.

> Fix ResultBoundedCompletionService deadlock
> ---
>
> Key: HBASE-14812
> URL: https://issues.apache.org/jira/browse/HBASE-14812
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Thrift
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14812-v1.patch, HBASE-14812-v2.patch, 
> HBASE-14812.patch
>
>
> {code}
> "thrift2-worker-0" #31 daemon prio=5 os_prio=0 tid=0x7f5ad9c45000 
> nid=0x484 in Object.wait() [0x7f5aa3832000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> - locked <0x0006b6ae7670> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1276)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1182)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:194)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:171)
> - locked <0x0006b6ae79c0> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
> at 
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1033)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler.putMultiple(ThriftHBaseServiceHandler.java:268)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler$THBaseServiceMetricsProxy.invoke(ThriftHBaseServiceHandler.java:114)
> at com.sun.proxy.$Proxy10.putMultiple(Unknown Source)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1637)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1621)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at 
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14820) Region becomes unavailable after a region split is rolled back

[Clara Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Clara Xiong updated HBASE-14820:

Attachment: HBSE-14820-hmaster.log

> Region becomes unavailable after a region split is rolled back
> --
>
> Key: HBASE-14820
> URL: https://issues.apache.org/jira/browse/HBASE-14820
> Project: HBase
>  Issue Type: Bug
>  Components: master, regionserver
>Affects Versions: 0.98.15
>Reporter: Clara Xiong
> Attachments: HBASE-14820-RegionServer.log, HBSE-14820-hmaster.log
>
>
> After the region server rolls back a timed out attempt of  region split, the 
> region becomes unavailable. 
> Symptoms:
> The RS displays the region open in the web UI.
> The meta table still points to the RS
> Requests for the regions receive a NotServingRegionException. 
> hbck reports 0 inconsistencies. 
> Moving the region fails. 
> Restarting the region server fixes the problem.
> We have see multiple occurrences which require operation intervention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009774#comment-15009774
 ] 

Hudson commented on HBASE-14708:


SUCCESS: Integrated in HBase-1.3-IT #317 (See 
[https://builds.apache.org/job/HBase-1.3-IT/317/])
HBASE-14708 Use copy on write Map for region location cache (eclark: rev 
4d5ac316b626baa43880a1b65916e008e3d73b69)
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/types/CopyOnWriteArrayMap.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaCache.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
* 
hbase-common/src/test/java/org/apache/hadoop/hbase/types/TestCopyOnWriteMaps.java


> Use copy on write Map for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, 
> HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v15.patch, 
> HBASE-14708-v16.patch, HBASE-14708-v17.patch, HBASE-14708-v2.patch, 
> HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, 
> HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, 
> HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, anotherbench2.zip, 
> location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase. 
> > 60% of the time was spent in locating a region. This was while the cluster 
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a 
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache. 
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for 
> reading and writing.
> However most operations will not need to remove or add a region location. 
> There will be potentially several orders of magnitude more reads for cached 
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14820) Region becomes unavailable after a region split is rolled back

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009859#comment-15009859
 ] 

Heng Chen commented on HBASE-14820:
---

Oh,  as region server log shows,  when split rollback,   split state is at 
OFFLINED_PARENT

And the logic in rollback when state at OFFLINED_PARENT is just add parent 
region back to online.  But parent region has been closed when state as 
CLOSED_PARENT_REGION.  
Relates code below

{code}
  case CLOSED_PARENT_REGION:
try {
  // So, this returns a seqid but if we just closed and then reopened, 
we
  // should be ok. On close, we flushed using sequenceid obtained from
  // hosting regionserver so no need to propagate the sequenceid 
returned
  // out of initialize below up into regionserver as we normally do.
  // TODO: Verify.
  this.parent.initialize();
} catch (IOException e) {
  LOG.error("Failed rollbacking CLOSED_PARENT_REGION of region " +
parent.getRegionInfo().getRegionNameAsString(), e);
  throw new RuntimeException(e);
}
break;
 ...
  case OFFLINED_PARENT:
if (services != null) services.addToOnlineRegions(this.parent);
break;
{code}


IMO before we add parent region back to online,  we should do 
{{this.parent.initialize()}}.   Thoughts?

> Region becomes unavailable after a region split is rolled back
> --
>
> Key: HBASE-14820
> URL: https://issues.apache.org/jira/browse/HBASE-14820
> Project: HBase
>  Issue Type: Bug
>  Components: master, regionserver
>Affects Versions: 0.98.15
>Reporter: Clara Xiong
> Attachments: HBASE-14820-RegionServer.log, HBSE-14820-hmaster.log
>
>
> After the region server rolls back a timed out attempt of  region split, the 
> region becomes unavailable. 
> Symptoms:
> The RS displays the region open in the web UI.
> The meta table still points to the RS
> Requests for the regions receive a NotServingRegionException. 
> hbck reports 0 inconsistencies. 
> Moving the region fails. 
> Restarting the region server fixes the problem.
> We have see multiple occurrences which require operation intervention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14782) FuzzyRowFilter skips valid rows

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009766#comment-15009766
 ] 

Heng Chen commented on HBASE-14782:
---

{quote}
Result size is always equals or greater than fuzzyKeyMeta, therefore your case 
is impossible.
{quote}

My point is if result.size always  equals or greater than fuzzyKeyMeta,  why 
not we replace 

{code}
+int off = fuzzyKeyMeta.length >= result.length? result.length -1:
+   fuzzyKeyMeta.length -1; 
{code}

to be  

{code}
  if (fuzzyKeyMeta.length >= result.length) {
  return result;
  }
{code}


Otherwise,  as a standalone method,  the logic in trimTrailingZeroes is not 
correct.

wdyt?  [~tedyu] [~vrodionov]

> FuzzyRowFilter skips valid rows
> ---
>
> Key: HBASE-14782
> URL: https://issues.apache.org/jira/browse/HBASE-14782
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.17
>
> Attachments: HBASE-14782-0.98-v4.patch, HBASE-14782-v3.patch, 
> HBASE-14782-v4.patch, HBASE-14782.patch, HBASE-14782.patch
>
>
> The issue may affect not only master branch, but previous releases as well.
> This is from one of our customers:
> {quote}
> We are experiencing a problem with the FuzzyRowFilter for HBase scan. We 
> think that it is a bug. 
> Fuzzy filter should pick a row if it matches filter criteria irrespective of 
> other rows present in table but filter is dropping a row depending on some 
> other row present in table. 
> Details/Step to reproduce/Sample outputs below: 
> Missing row key: \x9C\x00\x044\x00\x00\x00\x00 
> Causing row key: \x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX 
> Prerequisites 
> 1. Create a test table. HBase shell command -- create 'fuzzytest','d' 
> 2. Insert some test data. HBase shell commands: 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x01\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x01\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x01",'d:a','junk' 
> • put 'fuzzytest',"\x9B\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> • put 'fuzzytest',"\x9D\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> Now when you run the code, you will find \x9C\x00\x044\x00\x00\x00\x00 in 
> output because it matches filter criteria. (Refer how to run code below) 
> Insert the row key causing bug: 
> HBase shell command: put 
> 'fuzzytest',"\x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX",'d:a','junk' 
> Now when you run the code, you will not find \x9C\x00\x044\x00\x00\x00\x00 in 
> output even though it still matches filter criteria. 
> {quote}
> Verified the issue on master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14812) Fix ResultBoundedCompletionService deadlock

[Elliott Clark (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elliott Clark updated HBASE-14812:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

> Fix ResultBoundedCompletionService deadlock
> ---
>
> Key: HBASE-14812
> URL: https://issues.apache.org/jira/browse/HBASE-14812
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Thrift
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14812-v1.patch, HBASE-14812-v2.patch, 
> HBASE-14812.patch
>
>
> {code}
> "thrift2-worker-0" #31 daemon prio=5 os_prio=0 tid=0x7f5ad9c45000 
> nid=0x484 in Object.wait() [0x7f5aa3832000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> - locked <0x0006b6ae7670> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1276)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1182)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:194)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:171)
> - locked <0x0006b6ae79c0> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
> at 
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1033)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler.putMultiple(ThriftHBaseServiceHandler.java:268)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler$THBaseServiceMetricsProxy.invoke(ThriftHBaseServiceHandler.java:114)
> at com.sun.proxy.$Proxy10.putMultiple(Unknown Source)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1637)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1621)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at 
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009772#comment-15009772
 ] 

Hudson commented on HBASE-14793:


SUCCESS: Integrated in HBase-1.3-IT #317 (See 
[https://builds.apache.org/job/HBase-1.3-IT/317/])
HBASE-14793 Allow limiting size of block into L1 block cache. (eclark: rev 
604c9b2cca3db8b40eccd37652d0bb9d348480f3)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMetricsRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java


> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793-v7.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14791) Batch Deletes in MapReduce jobs (0.98)

[Alex Araujo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009765#comment-15009765
 ] 

Alex Araujo commented on HBASE-14791:
-

Based on [~vik.karma]'s perf analysis, this should fix the reported issue. 
We'll deploy the patch and verify.

> Batch Deletes in MapReduce jobs (0.98)
> --
>
> Key: HBASE-14791
> URL: https://issues.apache.org/jira/browse/HBASE-14791
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.16
>Reporter: Lars Hofhansl
>Assignee: Alex Araujo
>  Labels: mapreduce
> Fix For: 0.98.17
>
> Attachments: HBASE-14791-0.98-v1.patch, HBASE-14791-0.98-v2.patch, 
> HBASE-14791-0.98.patch
>
>
> We found that some of our copy table job run for many hours, even when there 
> isn't that much data to copy.
> [~vik.karma] did his magic and found that the issue is with copying delete 
> markers (we use raw mode to also move deletes across).
> Looking at the code in 0.98 it's immediately obvious that deletes (unlike 
> puts) are not batched and hence sent to the other side one by one, causing a 
> network RTT for each delete marker.
> Looks like in trunk it's doing the right thing (using BufferedMutators for 
> all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, 
> 1.2?) issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14584) TestNamespacesInstanceModel fails on jdk8

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009773#comment-15009773
 ] 

Hudson commented on HBASE-14584:


SUCCESS: Integrated in HBase-1.3-IT #317 (See 
[https://builds.apache.org/job/HBase-1.3-IT/317/])
HBASE-14584 TestNamespacesInstanceModel fails on jdk8 (apurtell: rev 
5b3a49d8350799de394b603f253d0a1d3ddb7a8e)
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/model/NamespacesInstanceModel.java
* hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestModelBase.java
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestNamespacesInstanceModel.java


> TestNamespacesInstanceModel fails on jdk8
> -
>
> Key: HBASE-14584
> URL: https://issues.apache.org/jira/browse/HBASE-14584
> Project: HBase
>  Issue Type: Test
>  Components: REST, test
>Reporter: Nick Dimiduk
>Assignee: Matt Warhaftig
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: 14584.00.patch, disable.txt, hbase-14584-v1.patch
>
>
> Noticed this in the [build 
> output|https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/235/consoleFull]
>  of HBASE-12911. Seems this test has been failing for a long time, got all 
> the way back to {{6534583}}, {{master~44}} and it's still failing there. I 
> guess tests usually fail in {{hbase-server}}, so we don't often get to 
> {{hbase-rest}} module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11386) Replication#table,CF config will be wrong if the table name includes namespace

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-11386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009769#comment-15009769
 ] 

Enis Soztutar commented on HBASE-11386:
---

bq. To fix this may be we can keep existing practice of specifying table CFs as 
it is
This is wrong. We should not rely on string parsing and delimiters, etc at all. 

> Replication#table,CF config will be wrong if the table name includes namespace
> --
>
> Key: HBASE-11386
> URL: https://issues.apache.org/jira/browse/HBASE-11386
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Qianxi Zhang
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE_11386_trunk_v1.patch, HBASE_11386_trunk_v2.patch
>
>
> Now we can config the table and CF in Replication, but I think the parse will 
> be wrong if the table name includes namespace
> ReplicationPeer#parseTableCFsFromConfig(line 125)
> {code}
> Map tableCFsMap = null;
> // parse out (table, cf-list) pairs from tableCFsConfig
> // format: "table1:cf1,cf2;table2:cfA,cfB"
> String[] tables = tableCFsConfig.split(";");
> for (String tab : tables) {
>   // 1 ignore empty table config
>   tab = tab.trim();
>   if (tab.length() == 0) {
> continue;
>   }
>   // 2 split to "table" and "cf1,cf2"
>   //   for each table: "table:cf1,cf2" or "table"
>   String[] pair = tab.split(":");
>   String tabName = pair[0].trim();
>   if (pair.length > 2 || tabName.length() == 0) {
> LOG.error("ignore invalid tableCFs setting: " + tab);
> continue;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14340) Add second bulk load option to Spark Bulk Load to send puts as the value

[Ted Malaska (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009797#comment-15009797
 ] 

Ted Malaska commented on HBASE-14340:
-

Thank u Andrew.  Let me know if there r any other jiras u would like me to
look at.

Thank again

On Tuesday, November 17, 2015, Andrew Purtell (JIRA) 



-- 
Sent from Gmail Mobile


> Add second bulk load option to Spark Bulk Load to send puts as the value
> 
>
> Key: HBASE-14340
> URL: https://issues.apache.org/jira/browse/HBASE-14340
> Project: HBase
>  Issue Type: New Feature
>  Components: spark
>Reporter: Ted Malaska
>Assignee: Ted Malaska
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14340.1.patch, HBASE-14340.2.patch
>
>
> The initial bulk load option for Spark bulk load sends values over one by one 
> through the shuffle.  This is the similar to how the original MR bulk load 
> worked.
> How ever the MR bulk loader have more then one bulk load option.  There is a 
> second option that allows for all the Column Families, Qualifiers, and Values 
> or a row to be combined in the map side.
> This only works if the row is not super wide.
> But if the row is not super wide this method of sending values through the 
> shuffle will reduce the data and work the shuffle has to deal with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14782) FuzzyRowFilter skips valid rows

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009798#comment-15009798
 ] 

Heng Chen commented on HBASE-14782:
---

{quote}
Can you prove that?
{quote}

My point is as a standalone method,  we can't depend param check outside the 
method.   Currently,  because of what you mentioned above,  the param passed in 
we can ensure {{result.length>=fuzzyKeyMeta.length}},  but if other guys want 
to call this method in some other places?  They don't know we should ensure 
{{result.length>=fuzzyKeyMeta.length}}

{quote}
I do not do that to make sure that we handle pathological case when 
fuzzyKeyMeta ends with 0's.
{quote}

Make sense,  so we can  do similar check after  
{code}
+for( ; off >= 0; off--){
+  if(fuzzyKeyMeta[off] != 0) break;
+}
{code}



> FuzzyRowFilter skips valid rows
> ---
>
> Key: HBASE-14782
> URL: https://issues.apache.org/jira/browse/HBASE-14782
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.17
>
> Attachments: HBASE-14782-0.98-v4.patch, HBASE-14782-v3.patch, 
> HBASE-14782-v4.patch, HBASE-14782.patch, HBASE-14782.patch
>
>
> The issue may affect not only master branch, but previous releases as well.
> This is from one of our customers:
> {quote}
> We are experiencing a problem with the FuzzyRowFilter for HBase scan. We 
> think that it is a bug. 
> Fuzzy filter should pick a row if it matches filter criteria irrespective of 
> other rows present in table but filter is dropping a row depending on some 
> other row present in table. 
> Details/Step to reproduce/Sample outputs below: 
> Missing row key: \x9C\x00\x044\x00\x00\x00\x00 
> Causing row key: \x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX 
> Prerequisites 
> 1. Create a test table. HBase shell command -- create 'fuzzytest','d' 
> 2. Insert some test data. HBase shell commands: 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x01\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x01\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x01",'d:a','junk' 
> • put 'fuzzytest',"\x9B\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> • put 'fuzzytest',"\x9D\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> Now when you run the code, you will find \x9C\x00\x044\x00\x00\x00\x00 in 
> output because it matches filter criteria. (Refer how to run code below) 
> Insert the row key causing bug: 
> HBase shell command: put 
> 'fuzzytest',"\x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX",'d:a','junk' 
> Now when you run the code, you will not find \x9C\x00\x044\x00\x00\x00\x00 in 
> output even though it still matches filter criteria. 
> {quote}
> Verified the issue on master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14812) Fix ResultBoundedCompletionService deadlock

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009746#comment-15009746
 ] 

Elliott Clark commented on HBASE-14812:
---

Sounds good. I'll push it then.

Thanks.

> Fix ResultBoundedCompletionService deadlock
> ---
>
> Key: HBASE-14812
> URL: https://issues.apache.org/jira/browse/HBASE-14812
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Thrift
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14812-v1.patch, HBASE-14812-v2.patch, 
> HBASE-14812.patch
>
>
> {code}
> "thrift2-worker-0" #31 daemon prio=5 os_prio=0 tid=0x7f5ad9c45000 
> nid=0x484 in Object.wait() [0x7f5aa3832000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> - locked <0x0006b6ae7670> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1276)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1182)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:194)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:171)
> - locked <0x0006b6ae79c0> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
> at 
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1033)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler.putMultiple(ThriftHBaseServiceHandler.java:268)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler$THBaseServiceMetricsProxy.invoke(ThriftHBaseServiceHandler.java:114)
> at com.sun.proxy.$Proxy10.putMultiple(Unknown Source)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1637)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1621)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at 
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14782) FuzzyRowFilter skips valid rows

[Vladimir Rodionov (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009780#comment-15009780
 ] 

Vladimir Rodionov commented on HBASE-14782:
---

{quote}
Otherwise, as a standalone method, the logic in trimTrailingZeroes is not 
correct.
{quote}

Can you prove that? 

{quote}
if (fuzzyKeyMeta.length >= result.length) {
  return result;
  }
{quote}

I do not do that to make sure that we handle pathological case when 
fuzzyKeyMeta ends with 0's.


> FuzzyRowFilter skips valid rows
> ---
>
> Key: HBASE-14782
> URL: https://issues.apache.org/jira/browse/HBASE-14782
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.17
>
> Attachments: HBASE-14782-0.98-v4.patch, HBASE-14782-v3.patch, 
> HBASE-14782-v4.patch, HBASE-14782.patch, HBASE-14782.patch
>
>
> The issue may affect not only master branch, but previous releases as well.
> This is from one of our customers:
> {quote}
> We are experiencing a problem with the FuzzyRowFilter for HBase scan. We 
> think that it is a bug. 
> Fuzzy filter should pick a row if it matches filter criteria irrespective of 
> other rows present in table but filter is dropping a row depending on some 
> other row present in table. 
> Details/Step to reproduce/Sample outputs below: 
> Missing row key: \x9C\x00\x044\x00\x00\x00\x00 
> Causing row key: \x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX 
> Prerequisites 
> 1. Create a test table. HBase shell command -- create 'fuzzytest','d' 
> 2. Insert some test data. HBase shell commands: 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x01\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x01\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x01",'d:a','junk' 
> • put 'fuzzytest',"\x9B\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> • put 'fuzzytest',"\x9D\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> Now when you run the code, you will find \x9C\x00\x044\x00\x00\x00\x00 in 
> output because it matches filter criteria. (Refer how to run code below) 
> Insert the row key causing bug: 
> HBase shell command: put 
> 'fuzzytest',"\x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX",'d:a','junk' 
> Now when you run the code, you will not find \x9C\x00\x044\x00\x00\x00\x00 in 
> output even though it still matches filter criteria. 
> {quote}
> Verified the issue on master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009812#comment-15009812
 ] 

Hudson commented on HBASE-14793:


FAILURE: Integrated in HBase-Trunk_matrix #475 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/475/])
HBASE-14793 Allow limiting size of block into L1 block cache. (eclark: rev 
a48d30984a9afad0b759aba6933e1b9807fdcd26)
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMetricsRegionServer.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java


> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793-v7.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009813#comment-15009813
 ] 

Hudson commented on HBASE-14708:


FAILURE: Integrated in HBase-Trunk_matrix #475 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/475/])
HBASE-14708 Use copy on write Map for region location cache (eclark: rev 
a591df1073c81a0fd1029f4a0db62d7ef8a95207)
* 
hbase-common/src/test/java/org/apache/hadoop/hbase/types/TestCopyOnWriteMaps.java
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/types/CopyOnWriteArrayMap.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaCache.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java


> Use copy on write Map for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, 
> HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v15.patch, 
> HBASE-14708-v16.patch, HBASE-14708-v17.patch, HBASE-14708-v2.patch, 
> HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, 
> HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, 
> HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, anotherbench2.zip, 
> location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase. 
> > 60% of the time was spent in locating a region. This was while the cluster 
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a 
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache. 
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for 
> reading and writing.
> However most operations will not need to remove or add a region location. 
> There will be potentially several orders of magnitude more reads for cached 
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14827) Potential memory leak on REST server if state-full scanner is not deleted

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009852#comment-15009852
 ] 

Enis Soztutar commented on HBASE-14827:
---

We need the leases mechanism in the REST scan resources as well. 

One open question is given stateless scanners, should statefull scanners be 
deprecated and removed or not? 


> Potential memory leak on REST server if state-full scanner is not deleted
> -
>
> Key: HBASE-14827
> URL: https://issues.apache.org/jira/browse/HBASE-14827
> Project: HBase
>  Issue Type: Bug
>Reporter: Duo Xu
>
> Currently when people are using REST APIs to create state-full scanner, do 
> scan with scanner id and delete scanner. However, if the clients never call 
> delete scanner or if the clients are on cloud VMs and VMs get reboot before 
> delete requests are sent, the resources associated with the scanner will 
> never be released on the REST server, "hbase.client.scanner.timeout.period" 
> will release the resources on region server side though.
> So this will cause memory leak on REST server in some cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-14827) Potential memory leak on REST server if state-full scanner is not deleted

[Duo Xu (JIRA)]

Duo Xu created HBASE-14827:
--

 Summary: Potential memory leak on REST server if state-full 
scanner is not deleted
 Key: HBASE-14827
 URL: https://issues.apache.org/jira/browse/HBASE-14827
 Project: HBase
  Issue Type: Bug
Reporter: Duo Xu


Currently when people are using REST APIs to create state-full scanner, do scan 
with scanner id and delete scanner. However, if the clients never call delete 
scanner or if the clients are on cloud VMs and VMs get reboot before delete 
requests are sent, the resources associated with the scanner will never be 
released on the REST server, "hbase.client.scanner.timeout.period" will release 
the resources on region server side though.

So this will cause memory leak on REST server in some cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11386) Replication#table,CF config will be wrong if the table name includes namespace

[Enis Soztutar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-11386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009764#comment-15009764
 ] 

Enis Soztutar commented on HBASE-11386:
---

[~ashish singhi] agreed that this is critical. However, the way to solve this 
should be through HBASE-11393. [~chenheng] has a nice patch that I could not 
review yet. Maybe you can take a look at that. 

> Replication#table,CF config will be wrong if the table name includes namespace
> --
>
> Key: HBASE-11386
> URL: https://issues.apache.org/jira/browse/HBASE-11386
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Qianxi Zhang
>Assignee: Ashish Singhi
>Priority: Critical
> Attachments: HBASE_11386_trunk_v1.patch, HBASE_11386_trunk_v2.patch
>
>
> Now we can config the table and CF in Replication, but I think the parse will 
> be wrong if the table name includes namespace
> ReplicationPeer#parseTableCFsFromConfig(line 125)
> {code}
> Map tableCFsMap = null;
> // parse out (table, cf-list) pairs from tableCFsConfig
> // format: "table1:cf1,cf2;table2:cfA,cfB"
> String[] tables = tableCFsConfig.split(";");
> for (String tab : tables) {
>   // 1 ignore empty table config
>   tab = tab.trim();
>   if (tab.length() == 0) {
> continue;
>   }
>   // 2 split to "table" and "cf1,cf2"
>   //   for each table: "table:cf1,cf2" or "table"
>   String[] pair = tab.split(":");
>   String tabName = pair[0].trim();
>   if (pair.length > 2 || tabName.length() == 0) {
> LOG.error("ignore invalid tableCFs setting: " + tab);
> continue;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14172) Upgrade existing thrift binding using thrift 0.9.2 compiler.

[Josh Elser (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser updated HBASE-14172:
---
Attachment: HBASE-14172.001.patch

001. In hopes of moving things forward: here's a patch that bumps up to 0.9.3.

As far as the actual changes (non-generated code), here's the nuts and bolts:

{noformat}
diff --git a/pom.xml b/pom.xml
index cb01cea..fa7400e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1195,7 +1195,7 @@
 1.10.8
 2.5.0
 thrift
-0.9.2
+0.9.3
 3.4.6
 1.7.7
 4.0.3
{noformat}

{noformat}
diff --git a/hbase-thrift/pom.xml b/hbase-thrift/pom.xml
index f7ab7f1..ff999dc 100644
--- a/hbase-thrift/pom.xml
+++ b/hbase-thrift/pom.xml
@@ -415,12 +415,12 @@
 
   thrift.version
   "The Thrift version must be 
specified."
-  0\.9\.2
+  0\.9\.3
   
 -
 -
 [FATAL] 
==
-[FATAL] HBase Thrift requires the thrift generator version 0.9.2.
+[FATAL] HBase Thrift requires the thrift generator version 0.9.3.
 [FATAL] Setting it to something else needs to be reviewed for wire and 
behavior compatibility.
 [FATAL] 
==
 -
{noformat}

{noformat}
diff --git 
a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
 
b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
index c5e44d5..4d46a31 100644
--- 
a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
+++ 
b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java
@@ -539,7 +539,7 @@ public class ThriftServerRunner implements Runnable {
 CallQueue callQueue =
 new CallQueue(new LinkedBlockingQueue(), metrics);
 ExecutorService executorService = createExecutor(
-callQueue, serverArgs.getWorkerThreads());
+callQueue, serverArgs.getMinWorkerThreads(), 
serverArgs.getMaxWorkerThreads());
 serverArgs.executorService(executorService)
   .processor(processor)
   .transportFactory(transportFactory)
@@ -551,7 +551,7 @@ public class ThriftServerRunner implements Runnable {
 CallQueue callQueue =
 new CallQueue(new LinkedBlockingQueue(), metrics);
 ExecutorService executorService = createExecutor(
-callQueue, serverArgs.getWorkerThreads());
+callQueue, serverArgs.getWorkerThreads(), 
serverArgs.getWorkerThreads());
 serverArgs.executorService(executorService)
   .processor(processor)
   .transportFactory(transportFactory)
@@ -597,11 +597,11 @@ public class ThriftServerRunner implements Runnable {
   }

   ExecutorService createExecutor(BlockingQueue callQueue,
- int workerThreads) {
+ int minWorkers, int maxWorkers) {
 ThreadFactoryBuilder tfb = new ThreadFactoryBuilder();
 tfb.setDaemon(true);
 tfb.setNameFormat("thrift-worker-%d");
-return new ThreadPoolExecutor(workerThreads, workerThreads,
+return new ThreadPoolExecutor(minWorkers, maxWorkers,
 Long.MAX_VALUE, TimeUnit.SECONDS, callQueue, tfb.build());
   }
{noformat}

{noformat}
diff --git 
a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java 
b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
index b5e17ef..a5c7275 100644
--- 
a/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
+++ 
b/hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java
@@ -245,10 +245,11 @@ public class ThriftServer extends Configured implements 
Tool {
 log.info("starting HBase HsHA Thrift server on " + 
inetSocketAddress.toString());
 THsHaServer.Args serverArgs = new THsHaServer.Args(serverTransport);
 if (workerThreads > 0) {
-  serverArgs.workerThreads(workerThreads);
+  // Could support the min & max threads, avoiding to preserve existing 
functionality.
+  
serverArgs.minWorkerThreads(workerThreads).maxWorkerThreads(workerThreads);
 }
 ExecutorService executorService = createExecutor(
-serverArgs.getWorkerThreads(), metrics);
+workerThreads, metrics);
 serverArgs.executorService(executorService);
 serverArgs.processor(processor);
 serverArgs.transportFactory(transportFactory);
{noformat}

> Upgrade existing thrift binding using thrift 0.9.2 compiler.
> 
>
> Key: HBASE-14172
> URL: https://issues.apache.org/jira/browse/HBASE-14172
> Project: HBase
>  Issue Type: Improvement
>

[jira] [Updated] (HBASE-14791) Batch Deletes in MapReduce jobs (0.98)

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-14791:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed to 0.98 only.
Thanks [~alexaraujo]

> Batch Deletes in MapReduce jobs (0.98)
> --
>
> Key: HBASE-14791
> URL: https://issues.apache.org/jira/browse/HBASE-14791
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.16
>Reporter: Lars Hofhansl
>Assignee: Alex Araujo
>  Labels: mapreduce
> Fix For: 0.98.17
>
> Attachments: HBASE-14791-0.98-v1.patch, HBASE-14791-0.98-v2.patch, 
> HBASE-14791-0.98.patch
>
>
> We found that some of our copy table job run for many hours, even when there 
> isn't that much data to copy.
> [~vik.karma] did his magic and found that the issue is with copying delete 
> markers (we use raw mode to also move deletes across).
> Looking at the code in 0.98 it's immediately obvious that deletes (unlike 
> puts) are not batched and hence sent to the other side one by one, causing a 
> network RTT for each delete marker.
> Looks like in trunk it's doing the right thing (using BufferedMutators for 
> all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, 
> 1.2?) issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14468) Compaction improvements: FIFO compaction policy

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14468:
--
Attachment: HBASE-14468-v10.patch

v10 patch.

> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-14468-v1.patch, HBASE-14468-v10.patch, 
> HBASE-14468-v2.patch, HBASE-14468-v3.patch, HBASE-14468-v4.patch, 
> HBASE-14468-v5.patch, HBASE-14468-v6.patch, HBASE-14468-v7.patch, 
> HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The 
> column family MUST have non-default TTL. 
> Essentially, FIFO compactor does only one job: collects expired store files. 
> I see many applications for this policy:
> # use it for very high volume raw data which has low TTL and which is the 
> source of another data (after additional processing). Example: Raw 
> time-series vs. time-based rollup aggregates and compacted time-series. We 
> collect raw time-series and store them into CF with FIFO compaction policy, 
> periodically we run  task which creates rollup aggregates and compacts 
> time-series, the original raw data can be discarded after that.
> # use it for data which can be kept entirely in a a block cache (RAM/SSD). 
> Say we have local SSD (1TB) which we can use as a block cache. No need for 
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and 
> network), we do not evict hot data from a block cache. The result: improved 
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
> 
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY, 
>   FIFOCompactionPolicy.class.getName());
> {code} 
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
> 
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY, 
>   FIFOCompactionPolicy.class.getName());
> {code}
> Make sure, that table has disabled region splits (either by setting 
> explicitly DisabledRegionSplitPolicy or by setting 
> ConstantSizeRegionSplitPolicy and very large max region size). You will have 
> to increase to a very large number store's blocking file number : 
> *hbase.hstore.blockingStoreFiles* as well.
>  
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14812) Fix ResultBoundedCompletionService deadlock

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009861#comment-15009861
 ] 

Hudson commented on HBASE-14812:


SUCCESS: Integrated in HBase-1.2-IT #287 (See 
[https://builds.apache.org/job/HBase-1.2-IT/287/])
HBASE-14812 Fix ResultBoundedCompletionService deadlock (eclark: rev 
d9127491bc6c02cdf3a7cdb55b770638bda42134)
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallableWithReplicas.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultBoundedCompletionService.java


> Fix ResultBoundedCompletionService deadlock
> ---
>
> Key: HBASE-14812
> URL: https://issues.apache.org/jira/browse/HBASE-14812
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Thrift
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14812-v1.patch, HBASE-14812-v2.patch, 
> HBASE-14812.patch
>
>
> {code}
> "thrift2-worker-0" #31 daemon prio=5 os_prio=0 tid=0x7f5ad9c45000 
> nid=0x484 in Object.wait() [0x7f5aa3832000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> - locked <0x0006b6ae7670> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1276)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1182)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:194)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:171)
> - locked <0x0006b6ae79c0> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
> at 
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1033)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler.putMultiple(ThriftHBaseServiceHandler.java:268)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler$THBaseServiceMetricsProxy.invoke(ThriftHBaseServiceHandler.java:114)
> at com.sun.proxy.$Proxy10.putMultiple(Unknown Source)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1637)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1621)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at 
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14468) Compaction improvements: FIFO compaction policy

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14468:
--
Status: Patch Available  (was: Open)

> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-14468-v1.patch, HBASE-14468-v10.patch, 
> HBASE-14468-v2.patch, HBASE-14468-v3.patch, HBASE-14468-v4.patch, 
> HBASE-14468-v5.patch, HBASE-14468-v6.patch, HBASE-14468-v7.patch, 
> HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The 
> column family MUST have non-default TTL. 
> Essentially, FIFO compactor does only one job: collects expired store files. 
> I see many applications for this policy:
> # use it for very high volume raw data which has low TTL and which is the 
> source of another data (after additional processing). Example: Raw 
> time-series vs. time-based rollup aggregates and compacted time-series. We 
> collect raw time-series and store them into CF with FIFO compaction policy, 
> periodically we run  task which creates rollup aggregates and compacts 
> time-series, the original raw data can be discarded after that.
> # use it for data which can be kept entirely in a a block cache (RAM/SSD). 
> Say we have local SSD (1TB) which we can use as a block cache. No need for 
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and 
> network), we do not evict hot data from a block cache. The result: improved 
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
> 
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY, 
>   FIFOCompactionPolicy.class.getName());
> {code} 
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
> 
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY, 
>   FIFOCompactionPolicy.class.getName());
> {code}
> Make sure, that table has disabled region splits (either by setting 
> explicitly DisabledRegionSplitPolicy or by setting 
> ConstantSizeRegionSplitPolicy and very large max region size). You will have 
> to increase to a very large number store's blocking file number : 
> *hbase.hstore.blockingStoreFiles* as well.
>  
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14468) Compaction improvements: FIFO compaction policy

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14468:
--
Status: Open  (was: Patch Available)

> Compaction improvements: FIFO compaction policy
> ---
>
> Key: HBASE-14468
> URL: https://issues.apache.org/jira/browse/HBASE-14468
> Project: HBase
>  Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0
>
> Attachments: HBASE-14468-v1.patch, HBASE-14468-v10.patch, 
> HBASE-14468-v2.patch, HBASE-14468-v3.patch, HBASE-14468-v4.patch, 
> HBASE-14468-v5.patch, HBASE-14468-v6.patch, HBASE-14468-v7.patch, 
> HBASE-14468-v8.patch, HBASE-14468-v9.patch
>
>
> h2. FIFO Compaction
> h3. Introduction
> FIFO compaction policy selects only files which have all cells expired. The 
> column family MUST have non-default TTL. 
> Essentially, FIFO compactor does only one job: collects expired store files. 
> I see many applications for this policy:
> # use it for very high volume raw data which has low TTL and which is the 
> source of another data (after additional processing). Example: Raw 
> time-series vs. time-based rollup aggregates and compacted time-series. We 
> collect raw time-series and store them into CF with FIFO compaction policy, 
> periodically we run  task which creates rollup aggregates and compacts 
> time-series, the original raw data can be discarded after that.
> # use it for data which can be kept entirely in a a block cache (RAM/SSD). 
> Say we have local SSD (1TB) which we can use as a block cache. No need for 
> compaction of a raw data at all.
> Because we do not do any real compaction, we do not use CPU and IO (disk and 
> network), we do not evict hot data from a block cache. The result: improved 
> throughput and latency both write and read.
> See: https://github.com/facebook/rocksdb/wiki/FIFO-compaction-style
> h3. To enable FIFO compaction policy
> For table:
> {code}
> HTableDescriptor desc = new HTableDescriptor(tableName);
> 
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY, 
>   FIFOCompactionPolicy.class.getName());
> {code} 
> For CF:
> {code}
> HColumnDescriptor desc = new HColumnDescriptor(family);
> 
> desc.setConfiguration(DefaultStoreEngine.DEFAULT_COMPACTION_POLICY_CLASS_KEY, 
>   FIFOCompactionPolicy.class.getName());
> {code}
> Make sure, that table has disabled region splits (either by setting 
> explicitly DisabledRegionSplitPolicy or by setting 
> ConstantSizeRegionSplitPolicy and very large max region size). You will have 
> to increase to a very large number store's blocking file number : 
> *hbase.hstore.blockingStoreFiles* as well.
>  
> h3. Limitations
> Do not use FIFO compaction if :
> * Table/CF has MIN_VERSION > 0
> * Table/CF has TTL = FOREVER (HColumnDescriptor.DEFAULT_TTL)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14584) TestNamespacesInstanceModel fails on jdk8

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009860#comment-15009860
 ] 

Hudson commented on HBASE-14584:


SUCCESS: Integrated in HBase-1.2-IT #287 (See 
[https://builds.apache.org/job/HBase-1.2-IT/287/])
HBASE-14584 TestNamespacesInstanceModel fails on jdk8 (apurtell: rev 
be873bbda647280c4cc406ac77e640515f753c9a)
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/model/NamespacesInstanceModel.java
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestNamespacesInstanceModel.java
* hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestModelBase.java


> TestNamespacesInstanceModel fails on jdk8
> -
>
> Key: HBASE-14584
> URL: https://issues.apache.org/jira/browse/HBASE-14584
> Project: HBase
>  Issue Type: Test
>  Components: REST, test
>Reporter: Nick Dimiduk
>Assignee: Matt Warhaftig
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: 14584.00.patch, disable.txt, hbase-14584-v1.patch
>
>
> Noticed this in the [build 
> output|https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/235/consoleFull]
>  of HBASE-12911. Seems this test has been failing for a long time, got all 
> the way back to {{6534583}}, {{master~44}} and it's still failing there. I 
> guess tests usually fail in {{hbase-server}}, so we don't often get to 
> {{hbase-rest}} module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14815) TestMobExportSnapshot.testExportFailure timeout occasionally

[Heng Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009902#comment-15009902
 ] 

Heng Chen commented on HBASE-14815:
---

Could you help me review this patch? [~stack]  :)

> TestMobExportSnapshot.testExportFailure timeout occasionally
> 
>
> Key: HBASE-14815
> URL: https://issues.apache.org/jira/browse/HBASE-14815
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14815.patch
>
>
> On master,  TestMobExportSnapshot.testExportFailure timeout occasionally.
> See
> https://builds.apache.org/job/PreCommit-HBASE-Build/16514//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/
> https://builds.apache.org/job/PreCommit-HBASE-Build/16511//testReport/org.apache.hadoop.hbase.snapshot/TestMobExportSnapshot/testExportFailure/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14782) FuzzyRowFilter skips valid rows

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14782:
--
Attachment: HBASE-14782-v4.patch

v4 patch.

> FuzzyRowFilter skips valid rows
> ---
>
> Key: HBASE-14782
> URL: https://issues.apache.org/jira/browse/HBASE-14782
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14782-v3.patch, HBASE-14782-v4.patch, 
> HBASE-14782.patch, HBASE-14782.patch
>
>
> The issue may affect not only master branch, but previous releases as well.
> This is from one of our customers:
> {quote}
> We are experiencing a problem with the FuzzyRowFilter for HBase scan. We 
> think that it is a bug. 
> Fuzzy filter should pick a row if it matches filter criteria irrespective of 
> other rows present in table but filter is dropping a row depending on some 
> other row present in table. 
> Details/Step to reproduce/Sample outputs below: 
> Missing row key: \x9C\x00\x044\x00\x00\x00\x00 
> Causing row key: \x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX 
> Prerequisites 
> 1. Create a test table. HBase shell command -- create 'fuzzytest','d' 
> 2. Insert some test data. HBase shell commands: 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x01\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x01\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x01",'d:a','junk' 
> • put 'fuzzytest',"\x9B\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> • put 'fuzzytest',"\x9D\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> Now when you run the code, you will find \x9C\x00\x044\x00\x00\x00\x00 in 
> output because it matches filter criteria. (Refer how to run code below) 
> Insert the row key causing bug: 
> HBase shell command: put 
> 'fuzzytest',"\x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX",'d:a','junk' 
> Now when you run the code, you will not find \x9C\x00\x044\x00\x00\x00\x00 in 
> output even though it still matches filter criteria. 
> {quote}
> Verified the issue on master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-14799) Commons-collections object deserialization remote command execution vulnerability

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009457#comment-15009457
 ] 

Andrew Purtell edited comment on HBASE-14799 at 11/17/15 8:44 PM:
--

Updated patches.

File "HBASE-14799.patch" applies from master down to branch-1.0. Updates the 
commons-collections dependency to 3.2.2 and removes unused code in our Base64 
utility class that can do object serialization and deserialization, which 
removes any use of that from our code base except for the BucketCache and 
internally in Thrift generated classes.

File "HBASE-14799-0.98.patch" disables object serialization support in the 
legacy HbaseObjectWritable by default, adds a configuration parameter for 
enabling it, adds unit tests, updates the commons-collections dependency to 
3.2.2 , and removes unused code in our Base64 utility class.

File "HBASE-14799-0.94.patch" updates the commons-collections dependency to 
3.2.2 and removes unused code in our Base64 utility class. The latter part 
isn't strictly needed but why not remove unused and potentially problematic 
code.


was (Author: apurtell):
Updated patches.

File "HBASE-14799.patch" applies from master down to branch-1.0. Updates the 
commons-collections dependency to 3.2.2 and removes unused code in our Base64 
utility class that can do object serialization and deserialization, which 
removes any use of that from our code base except for the BucketCache and 
internally in Thrift generated classes.

File "HBASE-14799-0.98.patch" disables object serialization support in the 
legacy HbaseObjectWritable, adds unit tests, updates the commons-collections 
dependency to 3.2.2 , and removes unused code in our Base64 utility class.

File "HBASE-14799-0.94.patch" updates the commons-collections dependency to 
3.2.2 and removes unused code in our Base64 utility class. The latter part 
isn't strictly needed but why not remove unused and potentially problematic 
code.

> Commons-collections object deserialization remote command execution 
> vulnerability 
> --
>
> Key: HBASE-14799
> URL: https://issues.apache.org/jira/browse/HBASE-14799
> Project: HBase
>  Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.94.28, 0.98.17
>
> Attachments: HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, 
> HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, HBASE-14799-0.98.patch, 
> HBASE-14799-0.98.patch, HBASE-14799.patch
>
>
> Read: 
> http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/
> TL;DR: If you have commons-collections on your classpath and accept and 
> process Java object serialization data, then you probably have an exploitable 
> remote command execution vulnerability. 
> 0.94 and earlier HBase releases are vulnerable because we might read in and 
> rehydrate serialized Java objects out of RPC packet data in 
> HbaseObjectWritable using ObjectInputStream#readObject (see 
> https://hbase.apache.org/0.94/xref/org/apache/hadoop/hbase/io/HbaseObjectWritable.html#714)
>  and we have commons-collections on the classpath on the server.
> 0.98 also carries some limited exposure to this problem through inclusion of 
> backwards compatible deserialization code in 
> HbaseObjectWritableFor96Migration. This is used by the 0.94-to-0.98 migration 
> utility, and by the AccessController when reading permissions from the ACL 
> table serialized in legacy format by 0.94. Unprivileged users cannot run the 
> tool nor access the ACL table.
> Unprivileged users can however attack a 0.94 installation. An attacker might 
> be able to use the method discussed on that blog post to capture valid HBase 
> RPC payloads for 0.94 and prior versions, rewrite them to embed an exploit, 
> and replay them to trigger a remote command execution with the privileges of 
> the account under which the HBase RegionServer daemon is running.
> We need to make a patch release of 0.94 that changes HbaseObjectWritable to 
> disallow processing of random Java object serializations. This will be a 
> compatibility break that might affect old style coprocessors, which quite 
> possibly may rely on this catch-all in HbaseObjectWritable for custom object 
> (de)serialization. We can introduce a new configuration setting, 
> "hbase.allow.legacy.object.serialization", defaulting to false.
> To be thorough, we can also use the new configuration setting  
> "hbase.allow.legacy.object.serialization" (defaulting to false) in 0.98 to 
> prevent the AccessController from falling back to the vulnerable legacy code. 
> This turns out to not affect the ability to migrate permissions because 
> TablePermission 

[jira] [Updated] (HBASE-14782) FuzzyRowFilter skips valid rows

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-14782:
---
Fix Version/s: 0.98.17
   1.1.4
   1.3.0
   1.2.0
   2.0.0

> FuzzyRowFilter skips valid rows
> ---
>
> Key: HBASE-14782
> URL: https://issues.apache.org/jira/browse/HBASE-14782
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.17
>
> Attachments: HBASE-14782-v3.patch, HBASE-14782-v4.patch, 
> HBASE-14782.patch, HBASE-14782.patch
>
>
> The issue may affect not only master branch, but previous releases as well.
> This is from one of our customers:
> {quote}
> We are experiencing a problem with the FuzzyRowFilter for HBase scan. We 
> think that it is a bug. 
> Fuzzy filter should pick a row if it matches filter criteria irrespective of 
> other rows present in table but filter is dropping a row depending on some 
> other row present in table. 
> Details/Step to reproduce/Sample outputs below: 
> Missing row key: \x9C\x00\x044\x00\x00\x00\x00 
> Causing row key: \x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX 
> Prerequisites 
> 1. Create a test table. HBase shell command -- create 'fuzzytest','d' 
> 2. Insert some test data. HBase shell commands: 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x01\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x01\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x01",'d:a','junk' 
> • put 'fuzzytest',"\x9B\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> • put 'fuzzytest',"\x9D\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> Now when you run the code, you will find \x9C\x00\x044\x00\x00\x00\x00 in 
> output because it matches filter criteria. (Refer how to run code below) 
> Insert the row key causing bug: 
> HBase shell command: put 
> 'fuzzytest',"\x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX",'d:a','junk' 
> Now when you run the code, you will not find \x9C\x00\x044\x00\x00\x00\x00 in 
> output even though it still matches filter criteria. 
> {quote}
> Verified the issue on master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14782) FuzzyRowFilter skips valid rows

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14782:
--
Status: Patch Available  (was: Open)

> FuzzyRowFilter skips valid rows
> ---
>
> Key: HBASE-14782
> URL: https://issues.apache.org/jira/browse/HBASE-14782
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14782-v3.patch, HBASE-14782-v4.patch, 
> HBASE-14782.patch, HBASE-14782.patch
>
>
> The issue may affect not only master branch, but previous releases as well.
> This is from one of our customers:
> {quote}
> We are experiencing a problem with the FuzzyRowFilter for HBase scan. We 
> think that it is a bug. 
> Fuzzy filter should pick a row if it matches filter criteria irrespective of 
> other rows present in table but filter is dropping a row depending on some 
> other row present in table. 
> Details/Step to reproduce/Sample outputs below: 
> Missing row key: \x9C\x00\x044\x00\x00\x00\x00 
> Causing row key: \x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX 
> Prerequisites 
> 1. Create a test table. HBase shell command -- create 'fuzzytest','d' 
> 2. Insert some test data. HBase shell commands: 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x01\x00\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x00\x01\x00",'d:a','junk' 
> • put 'fuzzytest',"\x9C\x00\x044\x00\x01\x00\x01",'d:a','junk' 
> • put 'fuzzytest',"\x9B\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> • put 'fuzzytest',"\x9D\x00\x044e\xBB\xB2\xBB",'d:a','junk' 
> Now when you run the code, you will find \x9C\x00\x044\x00\x00\x00\x00 in 
> output because it matches filter criteria. (Refer how to run code below) 
> Insert the row key causing bug: 
> HBase shell command: put 
> 'fuzzytest',"\x9C\x00\x03\xE9e\xBB{X\x1Fwts\x1F\x15vRX",'d:a','junk' 
> Now when you run the code, you will not find \x9C\x00\x044\x00\x00\x00\x00 in 
> output even though it still matches filter criteria. 
> {quote}
> Verified the issue on master.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009357#comment-15009357
 ] 

Hudson commented on HBASE-14708:


SUCCESS: Integrated in HBase-1.2-IT #286 (See 
[https://builds.apache.org/job/HBase-1.2-IT/286/])
HBASE-14708 Use copy on write Map for region location cache (eclark: rev 
c178595e7f84ee7e38f95b0388796d1518b9f86a)
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/types/CopyOnWriteArrayMap.java
* 
hbase-common/src/test/java/org/apache/hadoop/hbase/types/TestCopyOnWriteMaps.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/MetaCache.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java


> Use copy on write Map for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, 
> HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v15.patch, 
> HBASE-14708-v16.patch, HBASE-14708-v17.patch, HBASE-14708-v2.patch, 
> HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, 
> HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, 
> HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, anotherbench2.zip, 
> location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase. 
> > 60% of the time was spent in locating a region. This was while the cluster 
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a 
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache. 
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for 
> reading and writing.
> However most operations will not need to remove or add a region location. 
> There will be potentially several orders of magnitude more reads for cached 
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009356#comment-15009356
 ] 

Hudson commented on HBASE-14793:


SUCCESS: Integrated in HBase-1.2-IT #286 (See 
[https://builds.apache.org/job/HBase-1.2-IT/286/])
HBASE-14793 Allow limiting size of block into L1 block cache. (eclark: rev 
a5a12223059ef18c9cbb2d76c89277a927b97b99)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java
* 
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java
* 
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMetricsRegionServer.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java


> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793-v7.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009379#comment-15009379
 ] 

Hadoop QA commented on HBASE-14793:
---

{color:green}+1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12772782/HBASE-14793-v7.patch
  against master branch at commit db42fc64f1e54ab8b47799eae7920f7298a5a406.
  ATTACHMENT ID: 12772782

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 11 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16552//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16552//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16552//artifact/patchprocess/checkstyle-aggregate.html

  Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16552//console

This message is automatically generated.

> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793-v7.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14820) Region becomes unavailable after a region split is rolled back

[Clara Xiong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Clara Xiong updated HBASE-14820:

Attachment: HBASE-14820-RegionServer.log

> Region becomes unavailable after a region split is rolled back
> --
>
> Key: HBASE-14820
> URL: https://issues.apache.org/jira/browse/HBASE-14820
> Project: HBase
>  Issue Type: Bug
>  Components: master, regionserver
>Affects Versions: 0.98.15
>Reporter: Clara Xiong
> Attachments: HBASE-14820-RegionServer.log
>
>
> After the region server rolls back a timed out attempt of  region split, the 
> region becomes unavailable. 
> Symptoms:
> The RS displays the region open in the web UI.
> The meta table still points to the RS
> Requests for the regions receive a NotServingRegionException. 
> hbck reports 0 inconsistencies. 
> Moving the region fails. 
> Restarting the region server fixes the problem.
> We have see multiple occurrences which require operation intervention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14818) user_permission does not list namespace permissions

[Steven Hancz (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009333#comment-15009333
 ] 

Steven Hancz commented on HBASE-14818:
--

Hi Ted,

Using the command user_permission '@ns3' works.
But this assume that one knows the name of the namespace. I hopping that the 
command with '.*' option to list all permissions.
Alternatively I can think of a command called group_permission that will list 
all group permissions.

Regards,
Steven

> user_permission does not list namespace permissions
> ---
>
> Key: HBASE-14818
> URL: https://issues.apache.org/jira/browse/HBASE-14818
> Project: HBase
>  Issue Type: Bug
>  Components: hbase
>Affects Versions: 1.0.2
>Reporter: Steven Hancz
>Priority: Minor
>
> The user_permission command does not list namespace permissions:
> For example: if I create a new namespace or use an existing namespace and 
> grant a user privileges to that namespace the command user_permission does 
> not list the. The permission is visible in the acl table.
> Example:
> hbase(main):005:0>  create_namespace 'ns3'
> 0 row(s) in 0.1640 seconds
> hbase(main):007:0> grant 'test_user','RWXAC','@ns3'
> 0 row(s) in 0.5680 seconds
> hbase(main):008:0> user_permission '.*'
> User   
> Namespace,Table,Family,Qualifier:Permission   
>  
>  sh82993   finance,finance:emp,,: [Permission: 
> actions=READ,WRITE,EXEC,CREATE,ADMIN]  
>  @hbaseglobaldba   hbase,hbase:acl,,: [Permission: 
> actions=EXEC,CREATE,ADMIN] 
>  @hbaseglobaloper  hbase,hbase:acl,,: [Permission: 
> actions=EXEC,ADMIN]
>  hdfs  hbase,hbase:acl,,: [Permission: 
> actions=READ,WRITE,CREATE,ADMIN,EXEC]  
>  sh82993   ns1,ns1:tbl1,,: [Permission: 
> actions=READ,WRITE,EXEC,CREATE,ADMIN] 
>  ns1admin  ns1,ns1:tbl2,,: [Permission: 
> actions=EXEC,CREATE,ADMIN]
>  @hbaseappltest_ns1funct   ns1,ns1:tbl2,,: [Permission: 
> actions=READ,WRITE,EXEC]  
>  ns1funct  ns1,ns1:tbl2,,: [Permission: 
> actions=READ,WRITE,EXEC,CREATE,ADMIN] 
>  hbase ns2,ns2:tbl1,,: [Permission: 
> actions=READ,WRITE,EXEC,CREATE,ADMIN] 
> 9 row(s) in 1.8090 seconds
> As you can see user test_user does not appear in the output, but on can see 
> the permission in the ACL table. 
> hbase(main):001:0>  scan 'hbase:acl'
> ROWCOLUMN+CELL
> 
>  @finance  column=l:sh82993, timestamp=105519510, 
> value=RWXCA 
>  @gcbcppdn column=l:hdfs, timestamp=1446141119602, 
> value=RWCXA
>  @hbasecolumn=l:hdfs, timestamp=1446141485136, 
> value=RWCAX
>  @ns1  column=l:@hbaseappltest_ns1admin, 
> timestamp=1447437007467, value=RWXCA 
>  @ns1  column=l:@hbaseappltest_ns1funct, 
> timestamp=1447427366835, value=RWX   
>  @ns2  column=l:@hbaseappltest_ns2admin, 
> timestamp=1446674470456, value=XCA   
>  @ns2  column=l:test_user, 
> timestamp=1447692840030, value=RWAC   
>  
>  @ns3  column=l:test_user, 
> timestamp=1447692860434, value=RWXAC  
>  
>  finance:emp   column=l:sh82993, timestamp=107723316, 
> value=RWXCA 
>  hbase:acl column=l:@hbaseglobaldba, 
> timestamp=1446590375370, value=XCA   
>  hbase:acl column=l:@hbaseglobaloper, 
> timestamp=1446590387965, value=XA   
>  hbase:acl column=l:hdfs, timestamp=1446141737213, 
> value=RWCAX
>  ns1:tbl1  column=l:sh82993, timestamp=1446674153058, 
> value=RWXCA 
>  ns1:tbl2  

[jira] [Updated] (HBASE-14799) Commons-collections object deserialization remote command execution vulnerability

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-14799:
---
Attachment: HBASE-14799-0.98.patch
HBASE-14799-0.94.patch
HBASE-14799.patch

Updated patches.

File "HBASE-14799.patch" applies from master down to branch-1.0. Updates the 
commons-collections dependency to 3.2.2 and removes unused code in our Base64 
utility class that can do object serialization and deserialization, which 
removes any use of that from our code base except for the BucketCache and 
internally in Thrift generated classes.

File "HBASE-14799-0.98.patch" disables object serialization support in the 
legacy HbaseObjectWritable, adds unit tests, updates the commons-collections 
dependency to 3.2.2 , and removes unused code in our Base64 utility class.

File "HBASE-14799-0.94.patch" updates the commons-collections dependency to 
3.2.2 and removes unused code in our Base64 utility class. The latter part 
isn't strictly needed but why not remove unused and potentially problematic 
code.

> Commons-collections object deserialization remote command execution 
> vulnerability 
> --
>
> Key: HBASE-14799
> URL: https://issues.apache.org/jira/browse/HBASE-14799
> Project: HBase
>  Issue Type: Bug
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.94.28, 0.98.17
>
> Attachments: HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, 
> HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, HBASE-14799-0.98.patch, 
> HBASE-14799-0.98.patch, HBASE-14799.patch
>
>
> Read: 
> http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/
> TL;DR: If you have commons-collections on your classpath and accept and 
> process Java object serialization data, then you probably have an exploitable 
> remote command execution vulnerability. 
> 0.94 and earlier HBase releases are vulnerable because we might read in and 
> rehydrate serialized Java objects out of RPC packet data in 
> HbaseObjectWritable using ObjectInputStream#readObject (see 
> https://hbase.apache.org/0.94/xref/org/apache/hadoop/hbase/io/HbaseObjectWritable.html#714)
>  and we have commons-collections on the classpath on the server.
> 0.98 also carries some limited exposure to this problem through inclusion of 
> backwards compatible deserialization code in 
> HbaseObjectWritableFor96Migration. This is used by the 0.94-to-0.98 migration 
> utility, and by the AccessController when reading permissions from the ACL 
> table serialized in legacy format by 0.94. Unprivileged users cannot run the 
> tool nor access the ACL table.
> Unprivileged users can however attack a 0.94 installation. An attacker might 
> be able to use the method discussed on that blog post to capture valid HBase 
> RPC payloads for 0.94 and prior versions, rewrite them to embed an exploit, 
> and replay them to trigger a remote command execution with the privileges of 
> the account under which the HBase RegionServer daemon is running.
> We need to make a patch release of 0.94 that changes HbaseObjectWritable to 
> disallow processing of random Java object serializations. This will be a 
> compatibility break that might affect old style coprocessors, which quite 
> possibly may rely on this catch-all in HbaseObjectWritable for custom object 
> (de)serialization. We can introduce a new configuration setting, 
> "hbase.allow.legacy.object.serialization", defaulting to false.
> To be thorough, we can also use the new configuration setting  
> "hbase.allow.legacy.object.serialization" (defaulting to false) in 0.98 to 
> prevent the AccessController from falling back to the vulnerable legacy code. 
> This turns out to not affect the ability to migrate permissions because 
> TablePermission implements Writable, which is safe, not Serializable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14791) [0.98] CopyTable is extremely slow when moving delete markers

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009355#comment-15009355
 ] 

Andrew Purtell commented on HBASE-14791:


Comments on the v2 patch, basically LGTM:

- BufferedHTable should be annotated as Private

- I take back my earlier suggestion that buffering deletes should be disabled 
by default. FWIW


> [0.98] CopyTable is extremely slow when moving delete markers
> -
>
> Key: HBASE-14791
> URL: https://issues.apache.org/jira/browse/HBASE-14791
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.16
>Reporter: Lars Hofhansl
>Assignee: Alex Araujo
> Attachments: HBASE-14791-0.98-v1.patch, HBASE-14791-0.98-v2.patch
>
>
> We found that some of our copy table job run for many hours, even when there 
> isn't that much data to copy.
> [~vik.karma] did his magic and found that the issue is with copying delete 
> markers (we use raw mode to also move deletes across).
> Looking at the code in 0.98 it's immediately obvious that deletes (unlike 
> puts) are not batched and hence sent to the other side one by one, causing a 
> network RTT for each delete marker.
> Looks like in trunk it's doing the right thing (using BufferedMutators for 
> all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, 
> 1.2?) issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14793) Allow limiting size of block into L1 block cache.

[Appy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009422#comment-15009422
 ] 

Appy commented on HBASE-14793:
--

+1 And...

Yummyy!
!http://fairtradeusa.org/sites/default/files/imagecache/feature_image/products/photo_carousel/honey.jpg|width=200!

> Allow limiting size of block into L1 block cache.
> -
>
> Key: HBASE-14793
> URL: https://issues.apache.org/jira/browse/HBASE-14793
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14793-v1.patch, HBASE-14793-v3.patch, 
> HBASE-14793-v4.patch, HBASE-14793-v5.patch, HBASE-14793-v6.patch, 
> HBASE-14793-v7.patch, HBASE-14793.patch
>
>
> G1GC does really badly with long lived large objects. Lets allow limiting the 
> size of a block that can be kept in the block cache.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14812) Fix ResultBoundedCompletionService deadlock

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009487#comment-15009487
 ] 

stack commented on HBASE-14812:
---

+1

> Fix ResultBoundedCompletionService deadlock
> ---
>
> Key: HBASE-14812
> URL: https://issues.apache.org/jira/browse/HBASE-14812
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Thrift
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14812-v1.patch, HBASE-14812-v2.patch, 
> HBASE-14812.patch
>
>
> {code}
> "thrift2-worker-0" #31 daemon prio=5 os_prio=0 tid=0x7f5ad9c45000 
> nid=0x484 in Object.wait() [0x7f5aa3832000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> - locked <0x0006b6ae7670> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1276)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1182)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:194)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:171)
> - locked <0x0006b6ae79c0> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
> at 
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1033)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler.putMultiple(ThriftHBaseServiceHandler.java:268)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler$THBaseServiceMetricsProxy.invoke(ThriftHBaseServiceHandler.java:114)
> at com.sun.proxy.$Proxy10.putMultiple(Unknown Source)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1637)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1621)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at 
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14822) Renewing leases of scanners doesn't work

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-14822:
--
Status: Patch Available  (was: Open)

> Renewing leases of scanners doesn't work
> 
>
> Key: HBASE-14822
> URL: https://issues.apache.org/jira/browse/HBASE-14822
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14
>Reporter: Samarth Jain
>Assignee: Lars Hofhansl
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: 14822-0.98.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-14777:
--
Attachment: HBASE-14777-1.patch

V1: 
I am not sure how [~Bhupendra] actually encountered the issue, because there is 
another bug that actually masks the issue mentioned here.

Doing Future#get() returns an Integer object and using this Integer 
object in ArrayList#remove calls the overloaded ArrayList#remove(Object) method 
(no auto-unboxing will happen since this method exists), which basically means 
that it does not remove anything.

However, if you replace Integer with an int , you will encounter the 
ArrayIndexOutofBoundsException.

Anyways, I fixed both the issues and added a simple unit test to test inter 
cluster replication on mini clusters.

> Replication fails with IndexOutOfBoundsException
> 
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14777-1.patch, HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException 
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
>  threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 
> 1
>   at java.util.ArrayList.rangeCheck(Unknown Source)
>   at java.util.ArrayList.remove(Unknown Source)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication 
> entries list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14791) Batch Deletes in MapReduce jobs (0.98)

[Lars Hofhansl (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009987#comment-15009987
 ] 

Lars Hofhansl commented on HBASE-14791:
---

I do not see any java doc warnings pertaining to the files changed in this 
patch. Is that a dud?

> Batch Deletes in MapReduce jobs (0.98)
> --
>
> Key: HBASE-14791
> URL: https://issues.apache.org/jira/browse/HBASE-14791
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.16
>Reporter: Lars Hofhansl
>Assignee: Alex Araujo
>  Labels: mapreduce
> Fix For: 0.98.17
>
> Attachments: HBASE-14791-0.98-v1.patch, HBASE-14791-0.98-v2.patch, 
> HBASE-14791-0.98.patch
>
>
> We found that some of our copy table job run for many hours, even when there 
> isn't that much data to copy.
> [~vik.karma] did his magic and found that the issue is with copying delete 
> markers (we use raw mode to also move deletes across).
> Looking at the code in 0.98 it's immediately obvious that deletes (unlike 
> puts) are not batched and hence sent to the other side one by one, causing a 
> network RTT for each delete marker.
> Looks like in trunk it's doing the right thing (using BufferedMutators for 
> all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, 
> 1.2?) issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14030) HBase Backup/Restore Phase 1

[Vladimir Rodionov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov updated HBASE-14030:
--
Attachment: HBASE-14030-v16.patch

Final (hopefully) v16. Contains some bug fixes discovered during testing 
period. 

> HBase Backup/Restore Phase 1
> 
>
> Key: HBASE-14030
> URL: https://issues.apache.org/jira/browse/HBASE-14030
> Project: HBase
>  Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, 
> HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, 
> HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, 
> HBASE-14030-v16.patch, HBASE-14030-v2.patch, HBASE-14030-v3.patch, 
> HBASE-14030-v4.patch, HBASE-14030-v5.patch, HBASE-14030-v6.patch, 
> HBASE-14030-v7.patch, HBASE-14030-v8.patch
>
>
> This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design 
> doc for the phase description.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14822) Renewing leases of scanners doesn't work

[Andrew Purtell (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010011#comment-15010011
 ] 

Andrew Purtell commented on HBASE-14822:


bq. The only concern I'd have with the test is that it uses wall time, so on 
really slow VMs it could fail because of this (the least could expire before we 
get to renew it.)

Would it work to use EnvironmentEdgeManager#injectEdge to inject 
ManualEnvironmentEdge for explicitly fiddling with the (virtual) time? 

> Renewing leases of scanners doesn't work
> 
>
> Key: HBASE-14822
> URL: https://issues.apache.org/jira/browse/HBASE-14822
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14
>Reporter: Samarth Jain
>Assignee: Lars Hofhansl
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4
>
> Attachments: 14822-0.98.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14584) TestNamespacesInstanceModel fails on jdk8

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010077#comment-15010077
 ] 

Hudson commented on HBASE-14584:


SUCCESS: Integrated in HBase-1.2 #379 (See 
[https://builds.apache.org/job/HBase-1.2/379/])
HBASE-14584 TestNamespacesInstanceModel fails on jdk8 (apurtell: rev 
be873bbda647280c4cc406ac77e640515f753c9a)
* 
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestNamespacesInstanceModel.java
* 
hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/model/NamespacesInstanceModel.java
* hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/model/TestModelBase.java


> TestNamespacesInstanceModel fails on jdk8
> -
>
> Key: HBASE-14584
> URL: https://issues.apache.org/jira/browse/HBASE-14584
> Project: HBase
>  Issue Type: Test
>  Components: REST, test
>Reporter: Nick Dimiduk
>Assignee: Matt Warhaftig
> Fix For: 2.0.0, 1.2.0, 1.3.0, 0.98.17
>
> Attachments: 14584.00.patch, disable.txt, hbase-14584-v1.patch
>
>
> Noticed this in the [build 
> output|https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/235/consoleFull]
>  of HBASE-12911. Seems this test has been failing for a long time, got all 
> the way back to {{6534583}}, {{master~44}} and it's still failing there. I 
> guess tests usually fail in {{hbase-server}}, so we don't often get to 
> {{hbase-rest}} module.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14812) Fix ResultBoundedCompletionService deadlock

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010078#comment-15010078
 ] 

Hudson commented on HBASE-14812:


SUCCESS: Integrated in HBase-1.2 #379 (See 
[https://builds.apache.org/job/HBase-1.2/379/])
HBASE-14812 Fix ResultBoundedCompletionService deadlock (eclark: rev 
d9127491bc6c02cdf3a7cdb55b770638bda42134)
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ResultBoundedCompletionService.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallableWithReplicas.java


> Fix ResultBoundedCompletionService deadlock
> ---
>
> Key: HBASE-14812
> URL: https://issues.apache.org/jira/browse/HBASE-14812
> Project: HBase
>  Issue Type: Bug
>  Components: Client, Thrift
>Affects Versions: 1.2.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14812-v1.patch, HBASE-14812-v2.patch, 
> HBASE-14812.patch
>
>
> {code}
> "thrift2-worker-0" #31 daemon prio=5 os_prio=0 tid=0x7f5ad9c45000 
> nid=0x484 in Object.wait() [0x7f5aa3832000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> - locked <0x0006b6ae7670> (a 
> [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.loadCache(ClientSmallReversedScanner.java:212)
> at 
> org.apache.hadoop.hbase.client.ClientSmallReversedScanner.next(ClientSmallReversedScanner.java:186)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1276)
> at 
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1182)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> at 
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:194)
> at 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:171)
> - locked <0x0006b6ae79c0> (a 
> org.apache.hadoop.hbase.client.BufferedMutatorImpl)
> at 
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1033)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler.putMultiple(ThriftHBaseServiceHandler.java:268)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> org.apache.hadoop.hbase.thrift2.ThriftHBaseServiceHandler$THBaseServiceMetricsProxy.invoke(ThriftHBaseServiceHandler.java:114)
> at com.sun.proxy.$Proxy10.putMultiple(Unknown Source)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1637)
> at 
> org.apache.hadoop.hbase.thrift2.generated.THBaseService$Processor$putMultiple.getResult(THBaseService.java:1621)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at 
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
> at org.apache.thrift.server.Invocation.run(Invocation.java:18)
> at 
> org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14822) Renewing leases of scanners doesn't work

[Lars Hofhansl (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-14822:
--
Attachment: 14822-0.98.txt

Here's a patch for 0.98 with test that actually tests that the lease does not 
time out. Use -1 to indicate a close request, 0 can be used to simply renew the 
scanner.

The only concern I'd have with the test is that it uses wall time, so on 
*really* slow VMs it could fail because of this (the least could expire before 
we get to renew it.)


> Renewing leases of scanners doesn't work
> 
>
> Key: HBASE-14822
> URL: https://issues.apache.org/jira/browse/HBASE-14822
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.14
>Reporter: Samarth Jain
>Assignee: Lars Hofhansl
> Attachments: 14822-0.98.txt
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009960#comment-15009960
 ] 

Elliott Clark commented on HBASE-14708:
---

If we need to we can always go with a sharded set of arrays. That will mean 
that each write will only do 1/N of the copies and array holder gets a little 
more complex. However I don't expect it to be too pressing as even people with 
very large tables aren't talking to 100k regions from one client.

> Use copy on write Map for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, 
> HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v15.patch, 
> HBASE-14708-v16.patch, HBASE-14708-v17.patch, HBASE-14708-v2.patch, 
> HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, 
> HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, 
> HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, anotherbench2.zip, 
> location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase. 
> > 60% of the time was spent in locating a region. This was while the cluster 
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a 
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache. 
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for 
> reading and writing.
> However most operations will not need to remove or add a region location. 
> There will be potentially several orders of magnitude more reads for cached 
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14708) Use copy on write Map for region location cache

[Elliott Clark (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009961#comment-15009961
 ] 

Elliott Clark commented on HBASE-14708:
---

If we need to we can always go with a sharded set of arrays. That will mean 
that each write will only do 1/N of the copies and array holder gets a little 
more complex. However I don't expect it to be too pressing as even people with 
very large tables aren't talking to 100k regions from one client.

> Use copy on write Map for region location cache
> ---
>
> Key: HBASE-14708
> URL: https://issues.apache.org/jira/browse/HBASE-14708
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Affects Versions: 1.1.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14708-v10.patch, HBASE-14708-v11.patch, 
> HBASE-14708-v12.patch, HBASE-14708-v13.patch, HBASE-14708-v15.patch, 
> HBASE-14708-v16.patch, HBASE-14708-v17.patch, HBASE-14708-v2.patch, 
> HBASE-14708-v3.patch, HBASE-14708-v4.patch, HBASE-14708-v5.patch, 
> HBASE-14708-v6.patch, HBASE-14708-v7.patch, HBASE-14708-v8.patch, 
> HBASE-14708-v9.patch, HBASE-14708.patch, anotherbench.zip, anotherbench2.zip, 
> location_cache_times.pdf, result.csv
>
>
> Internally a co-worker profiled their application that was talking to HBase. 
> > 60% of the time was spent in locating a region. This was while the cluster 
> was stable and no regions were moving.
> To figure out if there was a faster way to cache region location I wrote up a 
> benchmark here: https://github.com/elliottneilclark/benchmark-hbase-cache
> This tries to simulate a heavy load on the location cache. 
> * 24 different threads.
> * 2 Deleting location data
> * 2 Adding location data
> * Using floor to get the result.
> To repeat my work just run ./run.sh and it should produce a result.csv
> Results:
> ConcurrentSkiplistMap is a good middle ground. It's got equal speed for 
> reading and writing.
> However most operations will not need to remove or add a region location. 
> There will be potentially several orders of magnitude more reads for cached 
> locations than there will be on clearing the cache.
> So I propose a copy on write tree map.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14791) Batch Deletes in MapReduce jobs (0.98)

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009962#comment-15009962
 ] 

Hadoop QA commented on HBASE-14791:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12772837/HBASE-14791-0.98.patch
  against 0.98 branch at commit dadfe7da0484be81ae09ad61f976967b9893c38d.
  ATTACHMENT ID: 12772837

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 4 new 
or modified tests.

{color:green}+1 hadoop versions{color}. The patch compiles with all 
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 
2.7.1)

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 protoc{color}.  The applied patch does not increase the 
total number of protoc compiler warnings.

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 
28 warning messages.

{color:green}+1 checkstyle{color}.  The applied patch does not increase the 
total number of checkstyle errors

{color:green}+1 findbugs{color}.  The patch does not introduce any  new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn post-site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16556//testReport/
Release Findbugs (version 2.0.3)warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16556//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16556//artifact/patchprocess/checkstyle-aggregate.html

  Javadoc warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16556//artifact/patchprocess/patchJavadocWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/16556//console

This message is automatically generated.

> Batch Deletes in MapReduce jobs (0.98)
> --
>
> Key: HBASE-14791
> URL: https://issues.apache.org/jira/browse/HBASE-14791
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 0.98.16
>Reporter: Lars Hofhansl
>Assignee: Alex Araujo
>  Labels: mapreduce
> Fix For: 0.98.17
>
> Attachments: HBASE-14791-0.98-v1.patch, HBASE-14791-0.98-v2.patch, 
> HBASE-14791-0.98.patch
>
>
> We found that some of our copy table job run for many hours, even when there 
> isn't that much data to copy.
> [~vik.karma] did his magic and found that the issue is with copying delete 
> markers (we use raw mode to also move deletes across).
> Looking at the code in 0.98 it's immediately obvious that deletes (unlike 
> puts) are not batched and hence sent to the other side one by one, causing a 
> network RTT for each delete marker.
> Looks like in trunk it's doing the right thing (using BufferedMutators for 
> all mutations in TableOutputFormat). So likely only a 0.98 (and 1.0, 1.1, 
> 1.2?) issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-14777) Replication fails with IndexOutOfBoundsException

[Ashu Pachauri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashu Pachauri updated HBASE-14777:
--
Attachment: HBASE-14777-2.patch

V2: 
slight performance improvement in the test.

> Replication fails with IndexOutOfBoundsException
> 
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
>  Issue Type: Bug
>  Components: Replication
>Reporter: Bhupendra Kumar Jain
>Assignee: Bhupendra Kumar Jain
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14777-1.patch, HBASE-14777-2.patch, 
> HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException 
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939): 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
>  threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size: 
> 1
>   at java.util.ArrayList.rangeCheck(Unknown Source)
>   at java.util.ArrayList.remove(Unknown Source)
>   at 
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication 
> entries list. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)