[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13096353#comment-13096353
]
Hudson commented on HBASE-4027:
---
Integrated in HBase-TRUNK #2172 (See
[https://builds.apache.org/job/HBase-TRUNK/2172/])
HBASE-4027 Off Heap Cache never creates Slabs
stack :
Files :
* /hbase/trunk/CHANGES.txt
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Fix For: 0.92.0
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff,
hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff,
hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13091638#comment-13091638
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-08-26 06:39:36, Michael Stack wrote:
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line
1
bq. https://reviews.apache.org/r/1214/diff/25/?file=34984#file34984line1
bq.
bq. When would we have an instance of Cacheable where we did not need to
serialize AND deserialize? (Wondering if worth having this extra
CacheableDeserializer)
If you wanted something to only reside in the on heap cache, you'd return 0 for
getSerializedLength, and return null for getDeserializer.
bq. On 2011-08-26 06:39:36, Michael Stack wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 44
bq. https://reviews.apache.org/r/1214/diff/25/?file=34983#file34983line44
bq.
bq. Is it ok that already serialize and can't be serialized return
same response out of this method?
Theres no longer an already serialized state. I now remove the object
entirely from the heap. I'll edit the comments.
bq. On 2011-08-26 06:39:36, Michael Stack wrote:
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line
49
bq. https://reviews.apache.org/r/1214/diff/25/?file=34990#file34990line49
bq.
bq. Whats this mean ('an exception will be thrown..') Will we ever try
to put something into a block that is larger than the block? Its a programming
error if we do? If a serialized object is larger than a 'block', what do we do?
Its a programming error if we do, we shouldn't ever do it, the sizer should
make sure of that. I'll add that to the comments.
bq. On 2011-08-26 06:39:36, Michael Stack wrote:
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line
52
bq. https://reviews.apache.org/r/1214/diff/25/?file=34990#file34990line52
bq.
bq. Where is this ConcurrentLinkedHashMap? Its not in this class? I
see a ConcurrentMap. Its made in the depths of guava?
Its made by Guava's MapMaker. I'll add it to the comments.
- Li
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1653
---
On 2011-08-24 04:08:40, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-24 04:08:40)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
ae75299
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
7ce4f14
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
3efba7f
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
7b7bf73
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
f0a9832
bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13091951#comment-13091951
]
stack commented on HBASE-4027:
--
Before commit I ran tests about 2/3rds completed. TestAdmin failed but
figured that not because of this patch.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Fix For: 0.92.0
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff,
hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff,
hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13091965#comment-13091965
]
Li Pi commented on HBASE-4027:
--
I'm running a test against the previous version without 4027. I'll let you know
what fails.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Fix For: 0.92.0
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff,
hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff,
hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092040#comment-13092040
]
Hudson commented on HBASE-4027:
---
Integrated in HBase-TRUNK #2147 (See
[https://builds.apache.org/job/HBase-TRUNK/2147/])
HBASE-4027 Enable direct byte buffers LruBlockCache
stack :
Files :
* /hbase/trunk/CHANGES.txt
* /hbase/trunk/conf/hbase-env.sh
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
*
/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
* /hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
*
/hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Fix For: 0.92.0
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff,
hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff,
hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092070#comment-13092070
]
Li Pi commented on HBASE-4027:
--
Huh:
Running org.apache.hadoop.hbase.client.TestAdmin
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD FAILURE
without the patch. Tests don't run?
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Fix For: 0.92.0
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff,
hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff,
hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092072#comment-13092072
]
stack commented on HBASE-4027:
--
I get this in TRUNK at mo:
{code}
Running org.apache.hadoop.hbase.client.TestAdmin
Tests run: 28, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 227.229 sec
FAILURE!
{code}
I'm taking a looksee.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Fix For: 0.92.0
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v15.3.diff, hbase-4027v6.diff, hbase4027v11.5.diff,
hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff,
hbase4027v12.1.diff, hbase4027v12.diff, hbase4027v15.2.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13091254#comment-13091254
]
Ted Yu commented on HBASE-4027:
---
+1 on latest patch.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff,
hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff,
hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089849#comment-13089849 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java, line 45 bq. https://reviews.apache.org/r/1214/diff/19/?file=34146#file34146line45 bq. bq. is it required still that the block contents be wrapped in a ByteBuffer? does Cacheable enforce this? if not, should this javadoc be updated? Fixed. Also updated Javadocs in BlockCache. bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 27 bq. https://reviews.apache.org/r/1214/diff/19/?file=34147#file34147line27 bq. bq. extra newline (maybe move it to below the public class). bq. bq. also missing a space after CacheStats fixed. bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 55 bq. https://reviews.apache.org/r/1214/diff/19/?file=34151#file34151line55 bq. bq. looks like this javadoc is accidentally applied on the CacheStats private var instead of constructor fixed. bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 39 bq. https://reviews.apache.org/r/1214/diff/19/?file=34156#file34156line39 bq. bq. i think this is the only place you use the term chunks. should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same :) I'm probably gonna change this to the MemCached model, where the slabs are pages, and blocks are chunks. Or we can just go our own way. bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 89 bq. https://reviews.apache.org/r/1214/diff/19/?file=34157#file34157line89 bq. bq. javadoc Will do. bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 28 bq. https://reviews.apache.org/r/1214/diff/19/?file=34158#file34158line28 bq. bq. add @param @return javadoc Will do. bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 93 bq. https://reviews.apache.org/r/1214/diff/19/?file=34152#file34152line93 bq. bq. should this be moved somewhere else? seems strange to have all this code up in the top of the class in the middle of class variable members Where should I declare statics? Moving it to the bottom would work. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1594 --- On 2011-08-23 23:02:05, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-23 23:02:05) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089850#comment-13089850
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-23 23:02:05.583625)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
fixed race condition. rebased on top of another patch. addressed some of
jgray's comments.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff,
hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff,
hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089947#comment-13089947
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-24 02:09:44.785814)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
added documentation according to jgrays changes.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff,
hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff,
hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089948#comment-13089948 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 39 bq. https://reviews.apache.org/r/1214/diff/19/?file=34156#file34156line39 bq. bq. i think this is the only place you use the term chunks. should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same :) bq. bq. Li Pi wrote: bq. I'm probably gonna change this to the MemCached model, where the slabs are pages, and blocks are chunks. Or we can just go our own way. We're going to call it Slabs and Blocks. Either way, easy enough change. Will make it consistent. bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 89 bq. https://reviews.apache.org/r/1214/diff/19/?file=34157#file34157line89 bq. bq. javadoc bq. bq. Li Pi wrote: bq. Will do. fixed. bq. On 2011-08-22 23:32:16, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java, line 28 bq. https://reviews.apache.org/r/1214/diff/19/?file=34158#file34158line28 bq. bq. add @param @return javadoc bq. bq. Li Pi wrote: bq. Will do. Fixed. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1594 --- On 2011-08-24 02:09:44, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-24 02:09:44) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089964#comment-13089964
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-24 02:46:25.061572)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
updated test case.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff,
hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff,
hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089977#comment-13089977
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-08-19 22:49:55, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 55
bq. https://reviews.apache.org/r/1214/diff/17/?file=33818#file33818line55
bq.
bq. does this always return itself? I still think that once something
has been serialized, we should throw away the original object, and when we
re-instantiate it, it should create a new object.
Fixed. Changed the interface.
bq. On 2011-08-19 22:49:55, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1467
bq. https://reviews.apache.org/r/1214/diff/17/?file=33821#file33821line1467
bq.
bq. this object confuses me. once we've serialized something, all we
should need is the ByteBuffer.
Fixed it.
bq. On 2011-08-19 22:49:55, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1518
bq. https://reviews.apache.org/r/1214/diff/17/?file=33821#file33821line1518
bq.
bq. this isn't right - you need to take into account arrayOffset also. I
think this .equals method should move to the tests, or a static method like
blocksAndContentsEqual()
Fixed. Comparisons are done with if
(this.buf.compareTo(castedComparison.buf) != 0) { instead.
- Li
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1575
---
On 2011-08-24 02:46:25, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-24 02:46:25)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
ae75299
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
7ce4f14
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
3efba7f
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
7b7bf73
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
f0a9832
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089985#comment-13089985 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1612 --- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java https://reviews.apache.org/r/1214/#comment3644 User may get confused by this javadoc since CacheableDeserializer is interface. How about: Returns CacheableDeserializer instance which reconstructs original object from ByteBuffer. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java https://reviews.apache.org/r/1214/#comment3643 This javadoc should be above interface declaration. - Ted On 2011-08-24 02:46:25, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-24 02:46:25) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089989#comment-13089989
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-24 03:57:16.993844)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
addressed ted yu's reviews above.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff,
hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff,
hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089990#comment-13089990 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-08-24 03:46:12, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 52 bq. https://reviews.apache.org/r/1214/diff/23/?file=34898#file34898line52 bq. bq. User may get confused by this javadoc since CacheableDeserializer is interface. bq. How about: bq. Returns CacheableDeserializer instance which reconstructs original object from ByteBuffer. fixed. bq. On 2011-08-24 03:46:12, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 8 bq. https://reviews.apache.org/r/1214/diff/23/?file=34899#file34899line8 bq. bq. This javadoc should be above interface declaration. fixed. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1612 --- On 2011-08-24 03:57:16, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-24 03:57:16) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff,
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089992#comment-13089992
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1614
---
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
https://reviews.apache.org/r/1214/#comment3647
@return should stay with deserialize() method.
- Ted
On 2011-08-24 03:57:16, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-24 03:57:16)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
ae75299
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
7ce4f14
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
3efba7f
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
7b7bf73
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
f0a9832
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff,
hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff,
hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089994#comment-13089994 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-08-24 04:00:58, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java, line 10 bq. https://reviews.apache.org/r/1214/diff/24/?file=34934#file34934line10 bq. bq. @return should stay with deserialize() method. fixed comment. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1614 --- On 2011-08-24 04:08:40, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-24 04:08:40) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff Java offers the creation of direct byte buffers which are allocated outside of the heap. They need to be manually free'd, which can be
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089993#comment-13089993
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-24 04:08:40.398370)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
fixed comment.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 3efba7f
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff,
hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff,
hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089059#comment-13089059
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-22 21:55:23.810893)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
CacheableSerializer interface added. HFileBlock now fully serialized.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff,
hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089134#comment-13089134
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-22 23:09:44.370020)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
fixed concurrency issue. now with readwritelocks.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
CHANGES.txt 0bb7c65
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java
34513f1
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 8f5600d
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java a817c37
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java d6fa01b
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheColumnFamilySummary.java
cc4abc6
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFileBlockCacheSummary.java
f8a854c
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v13.1.diff,
hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff,
hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff,
hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089148#comment-13089148 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1594 --- Looking good! Still some lines 80 chars and whitespace to fix, but otherwise not much to comment on. As we discussed, I find the Cacheable interface name a bit confusing. Maybe rename it, or just fill the classes and methods with awesome javadoc that explains how it all works :) Also, should probably add a section to the book about this? Some of these parameters are pretty important and the defaults could be wasteful for some of the default use cases (for example, always giving 20% of heap to blocks 1.1X block size) src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java https://reviews.apache.org/r/1214/#comment3591 is it required still that the block contents be wrapped in a ByteBuffer? does Cacheable enforce this? if not, should this javadoc be updated? src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java https://reviews.apache.org/r/1214/#comment3592 extra newline (maybe move it to below the public class). also missing a space after CacheStats src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment3594 looks like this javadoc is accidentally applied on the CacheStats private var instead of constructor src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3595 should this be moved somewhere else? seems strange to have all this code up in the top of the class in the middle of class variable members src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java https://reviews.apache.org/r/1214/#comment3596 is this necessary since we throw a runtime exception? could this leave the RS in a weird state? should we actually halt or abort? or will that happen when this gets thrown? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java https://reviews.apache.org/r/1214/#comment3599 i think this is the only place you use the term chunks. should we just call the slab chunks blocks? or should we call them chunks everywhere to avoid confusion with hfile blocks? although they are kinda the same :) src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java https://reviews.apache.org/r/1214/#comment3597 javadoc src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java https://reviews.apache.org/r/1214/#comment3600 add @param @return javadoc - Jonathan On 2011-08-22 23:09:44, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-22 23:09:44) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.CHANGES.txt 0bb7c65 bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java bfd863e bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheColumnFamilySummary.java 34513f1 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ae75299 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 7ce4f14 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088211#comment-13088211 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1585 --- In SingleSizeCache.cacheBlock(): CacheablePair newEntry = new CacheablePair( toBeCached.serialize(storedBlock), storedBlock); The above operation splits toBeCached into two parts: the first is for on-heap and is slim, storedBlock is for off-heap. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java https://reviews.apache.org/r/1214/#comment3574 I think the word 'itself' in the javadoc above introduced confusion. It should be removed. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3573 As Pi explained in Cacheable interface, serialize() offloads majority of data to off-heap ByteBuffer. What gets returned is the skeleton that lives on-heap. - Ted On 2011-08-19 20:21:35, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-19 20:21:35) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff,
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087743#comment-13087743 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1549 --- src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java https://reviews.apache.org/r/1214/#comment3524 The following 10 lines of code is very similar to those starting line 65. Should be able to write another method so that both of them can call. - Ted On 2011-08-19 04:16:44, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-19 04:16:44) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.CHANGES.txt 0478003 bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff Java offers the creation of direct byte buffers which are allocated outside of the
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087941#comment-13087941
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-19 20:21:35.572967)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Forgot to override .equals when redoing HFileBlock. Fixed, tests now work.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff,
hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088015#comment-13088015 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-08-19 13:30:39, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 67 bq. https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line67 bq. bq. Still some white spaces. fixed. bq. On 2011-08-19 13:30:39, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 244 bq. https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line244 bq. bq. This is an important class related to Cacheable. Should have added javadoc for class and methods. it's just a pair, added a comment though. bq. On 2011-08-19 13:30:39, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 255 bq. https://reviews.apache.org/r/1214/diff/16/?file=33705#file33705line255 bq. bq. REFERENCE refers to serializedData, right ? bq. I think it would be clearer if the order of second and third parameters gets exchanged. fixed. added comment. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1546 --- On 2011-08-19 20:21:35, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-19 20:21:35) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff,
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088038#comment-13088038
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1573
---
Ship it!
All tests passed based on hbase-4027v12.7.diff
- Ted
On 2011-08-19 20:21:35, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-19 20:21:35)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
1338453
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
886c31d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
e2c6c93
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
7b7bf73
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
f0a9832
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff,
hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088046#comment-13088046 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1575 --- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java https://reviews.apache.org/r/1214/#comment3551 does this always return itself? I still think that once something has been serialized, we should throw away the original object, and when we re-instantiate it, it should create a new object. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3553 this object confuses me. once we've serialized something, all we should need is the ByteBuffer. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3555 this isn't right - you need to take into account arrayOffset also. I think this .equals method should move to the tests, or a static method like blocksAndContentsEqual() - Todd On 2011-08-19 20:21:35, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-19 20:21:35) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff,
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088084#comment-13088084 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-08-19 14:36:26, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 78 bq. https://reviews.apache.org/r/1214/diff/16/?file=33706#file33706line78 bq. bq. The following 10 lines of code is very similar to those starting line 65. bq. Should be able to write another method so that both of them can call. fixed. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1549 --- On 2011-08-19 20:21:35, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-19 20:21:35) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff Java offers the creation of direct byte buffers which are allocated outside of the heap. They need to be manually free'd, which can be accomplished using an documented
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088089#comment-13088089
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-08-19 22:49:55, Todd Lipcon wrote:
bq.
I haven't added in the serializer interface yet. That will be there very very
soon - and should allow us to reduce the heapsize by another 60 bytes or so per
entry.
- Li
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1575
---
On 2011-08-19 20:21:35, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-19 20:21:35)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
1338453
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
886c31d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
e2c6c93
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
7b7bf73
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java
f0a9832
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff,
hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086886#comment-13086886
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-18 08:31:59.585101)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
heapsize added. also addressed ted's review above. fixed formatting in
hfileblock. (was inconsistent).
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff,
hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087233#comment-13087233 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-08-17 04:44:54, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 12 bq. https://reviews.apache.org/r/1214/diff/13/?file=32972#file32972line12 bq. bq. Please add javadoc for these methods. Done. bq. On 2011-08-17 04:44:54, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 92 bq. https://reviews.apache.org/r/1214/diff/13/?file=32974#file32974line92 bq. bq. This shows that Cacheable can reside in on-heap cache. bq. The description for Cacheable should be refined. Done. Cacheable can reside in either. bq. On 2011-08-17 04:44:54, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 79 bq. https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line79 bq. bq. HeapSize is covered by Cacheable so is not needed here. Done. bq. On 2011-08-17 04:44:54, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1528 bq. https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1528 bq. bq. Indentation is incorrect here. Done. bq. On 2011-08-17 04:44:54, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1535 bq. https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1535 bq. bq. Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means. bq. Please add javadoc. Done. bq. On 2011-08-17 04:44:54, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1540 bq. https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1540 bq. bq. Please add javadoc for what this method does. Done. bq. On 2011-08-17 04:44:54, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1541 bq. https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1541 bq. bq. Whitespace. Done. bq. On 2011-08-17 04:44:54, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java, line 1554 bq. https://reviews.apache.org/r/1214/diff/13/?file=32975#file32975line1554 bq. bq. Indentation. Done. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1489 --- On 2011-08-18 08:31:59, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-18 08:31:59) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087260#comment-13087260 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1525 --- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java https://reviews.apache.org/r/1214/#comment3476 Whitespace. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment3477 Whitespace. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3478 Auto-formatter weirdness. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3480 this and all other formatting fixes should probably be in a seperate jira. filed. - Li On 2011-08-18 08:31:59, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-18 08:31:59) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087479#comment-13087479
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-19 03:05:29.951253)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Cacheable interface now far less confusing. HFileBlock has reformatting
reverted.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
CHANGES.txt 763ddbc
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313
src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java
8a69a39
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff,
hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087482#comment-13087482 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1543 --- Todd: I'll get the fully serialized model out by tonight. This still leaves around 60 bytes of stuff on the heap. - Li On 2011-08-19 03:05:29, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-19 03:05:29) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.CHANGES.txt 763ddbc bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313 bq. src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39 bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff Java offers the creation of direct byte buffers which are allocated outside of the heap. They need to be manually free'd, which can be accomplished using
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087511#comment-13087511 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1544 --- Cacheable interface is much more intuitive now. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java https://reviews.apache.org/r/1214/#comment3516 Change an to the. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java https://reviews.apache.org/r/1214/#comment3515 Do I see an incomplete sentence here ? src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java https://reviews.apache.org/r/1214/#comment3514 If self is always returned, why do we need the return value here ? - Ted On 2011-08-19 03:05:29, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-19 03:05:29) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.CHANGES.txt 763ddbc bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 9a71fdf bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 431f313 bq. src/test/java/org/apache/hadoop/hbase/TestFSTableDescriptorForceCreation.java 8a69a39 bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff,
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087517#comment-13087517
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-19 04:16:44.682053)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
fixed as per ted yu's comments above.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
CHANGES.txt 0478003
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java
d7e43a0
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
5063896
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff,
hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13087518#comment-13087518 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-08-19 04:10:41, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java, line 49 bq. https://reviews.apache.org/r/1214/diff/15/?file=33671#file33671line49 bq. bq. If self is always returned, why do we need the return value here ? it returns a copy of itself, unless it doesn't need deserialization. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1544 --- On 2011-08-19 04:16:44, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-19 04:16:44) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.CHANGES.txt 0478003 bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java c301d1b bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreLAB.java d7e43a0 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java 5063896 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff, hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff, hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff, hbase4027v11.diff, hbase4027v12.1.diff, hbase4027v12.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086117#comment-13086117
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-17 04:16:06.047294)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
refactored, no more casting, cacheable added. multithreaded test that hammers
single key added. general code cleanup stuff as well. heapsize not implemented
completely yet.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java
1ad2ece
src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff,
hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086134#comment-13086134 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1489 --- src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java https://reviews.apache.org/r/1214/#comment3402 Please add javadoc for these methods. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment3403 This shows that Cacheable can reside in on-heap cache. The description for Cacheable should be refined. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3404 HeapSize is covered by Cacheable so is not needed here. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3405 Indentation is incorrect here. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3406 Since HFileBlock implements Cacheable, people may get confused by what 'selfWithoutByteBuffer' means. Please add javadoc. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3407 Please add javadoc for what this method does. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3408 Whitespace. src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java https://reviews.apache.org/r/1214/#comment3409 Indentation. - Ted On 2011-08-17 04:16:06, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-17 04:16:06) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CachedBlock.java 3b130d8 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e2c6c93 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestCachedBlockQueue.java 1ad2ece bq.src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java f0a9832 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13084003#comment-13084003
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-12 08:41:37.017483)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
fixed test failure, as ted yu reported.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13084311#comment-13084311
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-08-12 17:26:40, Todd Lipcon wrote:
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines
101-103
bq. https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line101
bq.
bq. this can race against getBlock() though:
bq.
bq. Thread A: backingMap.get(key) returns object
bq. Thread B: put() returns same object
bq. Thread B: free(object)
bq. Thread A: use object. boom?
bq.
bq. putIfAbsent shouldn't be any slower than put, may as well make use
of it
Ah, gotcha! I see it now. Fixed.
- Li
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1423
---
On 2011-08-12 08:41:37, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-12 08:41:37)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
1338453
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
886c31d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
7a917da
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
7b7bf73
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13084358#comment-13084358
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-12 20:26:21.230751)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Fixed another broken test case. (Didn't reset buffer position before doing
compare) and fixed race.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13084425#comment-13084425
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-08-12 21:52:43, Ted Yu wrote:
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line
38
bq. https://reviews.apache.org/r/1214/diff/11/?file=32400#file32400line38
bq.
bq. Still some white spaces to remove.
Got it.
bq. On 2011-08-12 21:52:43, Ted Yu wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 37
bq. https://reviews.apache.org/r/1214/diff/11/?file=32401#file32401line37
bq.
bq. Incorrect class name.
Doh. Fixed.
- Li
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1430
---
On 2011-08-12 20:26:21, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-12 20:26:21)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
1338453
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
886c31d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
7a917da
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
7b7bf73
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see:
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13084428#comment-13084428
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-12 22:30:10.310736)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
fixed two bugs as per ted yu's reviews.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 7a917da
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.5.diff, hbase4027v11.6.diff, hbase4027v11.7.diff,
hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13082987#comment-13082987
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. CHANGES.txt, line 197
bq. https://reviews.apache.org/r/1214/diff/8/?file=31762#file31762line197
bq.
bq. your diff appears to revert this change. perhaps you need to rebase
on trunk before you take diff against it.
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, lines
24-26
bq. https://reviews.apache.org/r/1214/diff/8/?file=31766#file31766line24
bq.
bq. style:
bq. /**
bq.* ...
bq.*/
bq. public class CacheStates {
bq.
bq. (comment formatting and space before '{')
Fixed
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 58
bq. https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line58
bq.
bq. hrm, is this constructor ever meant to be used? If the off-heap
cache isn't configured, then it should just instantiate LruBlockCache directly,
no?
This is currently taken care of in StoreFile.java, but, good idea, I'll change
that.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 64
bq. https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line64
bq.
bq. does it ever make sense to have offHeapSize onHeapSize? Perhaps we
should have a Preconditions check here?
Its useful for testing, I guess? And 8gb heap with 2gb of offheap 8gb heap
without 2gb of offheap.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 65
bq. https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line65
bq.
bq. hyphenate 'on-heap' and 'off-heap' for clarity
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 67
bq. https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line67
bq.
bq. missing space - bytes ...
The string would end up being: Creating off-heap cache of size 1.9G bytes.
Should it be different?
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 137
bq. https://reviews.apache.org/r/1214/diff/8/?file=31767#file31767line137
bq.
bq. we should add in the heap size used by the accounting and hashmaps
in the off-heap cache as well.
Will take care of next revision.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, lines
118-119
bq. https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line118
bq.
bq. I think there's a bug here if you have multiple users hammering the
same contentBlock -- two people can get to rewind() at the same time. You
probably need synchronized(contentBlock) around these two lines. See if you can
add a unit test which puts just one block in the cache and starts several
threads which hammer it - I bet you eventually one of the blocks comes back
returned as all 0x
changed it to returnBlock.put(contentBlock.duplicate()) instead.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line
176
bq. https://reviews.apache.org/r/1214/diff/8/?file=31773#file31773line176
bq.
bq. this is a little gross, since the APIs imply that you can cache
anything, but the implementation only supports HFileBlock.
bq.
bq. In the HFile code review I'd suggested adding a new interface like
CacheableBlock that would expose any APIs you need. We should consider doing
that now?
Definitely. Will do. At that point, might also change DoubleBlockCache to not
cache things that aren't CacheableBlocks.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line
190
bq. https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line190
bq.
bq. maybe rename to getOccupiedSize?
Done.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line
148
bq. https://reviews.apache.org/r/1214/diff/8/?file=31771#file31771line148
bq.
bq. this.size() is in units of bytes, not blocks
Fixed.
bq. On 2011-08-09 22:05:13, Todd Lipcon wrote:
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java,
line 45
bq. https://reviews.apache.org/r/1214/diff/8/?file=31780#file31780line45
bq.
bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13082998#comment-13082998
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-11 07:52:55.019423)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Fixed as per ted/todd's reviews.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 1338453
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java f22fb6e
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 7b7bf73
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
hbase4027v11.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13082679#comment-13082679
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-08-10 01:44:22, Ted Yu wrote:
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line
18
bq. https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line18
bq.
bq. I think this was intended for use by hbase-env.sh
Yup, it's used by StoreFile.java now. We now specify the size of the off heap
cache as a proportion of total direct memory size.
long offHeapCacheSize = (long)
(conf.getFloat(hbase.offheapcachepercentage, (float) 0.95) *
DirectMemoryUtils.getDirectMemorySize());
- Li
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1371
---
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.CHANGES.txt e9c0478
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
aa09b7d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
88aa652
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
886c31d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
86652c0
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
94c8bb4
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13082693#comment-13082693
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1392
---
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
https://reviews.apache.org/r/1214/#comment3179
I believe this was a typo.
You meant 1 (for bytes), right ?
- Ted
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.CHANGES.txt e9c0478
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
aa09b7d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
88aa652
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
886c31d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
86652c0
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
94c8bb4
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13082713#comment-13082713
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-08-10 21:28:51, Ted Yu wrote:
bq. src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java, line
21
bq. https://reviews.apache.org/r/1214/diff/8/?file=31777#file31777line21
bq.
bq. I believe this was a typo.
bq. You meant 1 (for bytes), right ?
Good catch. Fixed.
- Li
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1392
---
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.CHANGES.txt e9c0478
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
aa09b7d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
88aa652
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
886c31d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
86652c0
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
94c8bb4
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13082835#comment-13082835 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-07-30 02:57:26, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java, line 41 bq. https://reviews.apache.org/r/1214/diff/3/?file=28543#file28543line41 bq. bq. I don't see SkipList being used in this class. Fixed. It got changed to an unsynchronized TreeMap, as we aren't supporting online cache resizing. bq. On 2011-07-30 02:57:26, Ted Yu wrote: bq. conf/hbase-env.sh, line 40 bq. https://reviews.apache.org/r/1214/diff/3/?file=28534#file28534line40 bq. bq. Still some whitespaces. Fixed. bq. On 2011-07-30 02:57:26, Ted Yu wrote: bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java, line 31 bq. https://reviews.apache.org/r/1214/diff/3/?file=28549#file28549line31 bq. bq. This doesn't match the actual test. Fixed. bq. On 2011-07-30 02:57:26, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java, line 31 bq. https://reviews.apache.org/r/1214/diff/3/?file=28542#file28542line31 bq. bq. Doesn't match actual name of class. Fixed. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1238 --- On 2011-08-09 20:39:55, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-08-09 20:39:55) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.CHANGES.txt e9c0478 bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION bq. src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf, HBase-4027.pdf,
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13081498#comment-13081498 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/ --- (Updated 2011-08-09 07:52:09.666709) Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. Changes --- Rebased onto 3857. Configuration added as well. Summary --- Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. This addresses bug HBase-4027. https://issues.apache.org/jira/browse/HBase-4027 Diffs (updated) - CHANGES.txt 7c6f592 conf/hbase-env.sh 2d55d27 pom.xml 0f24681 src/docbkx/book.xml 2c19cef src/docbkx/configuration.xml 3595e76 src/docbkx/developer.xml a3e22ea src/docbkx/performance.xml d8e104f src/docbkx/troubleshooting.xml 9f93cd9 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/Delayable.java 56f0650 src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java f223e46 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 6fb1da7 src/main/java/org/apache/hadoop/hbase/mapred/Driver.java d38956c src/main/java/org/apache/hadoop/hbase/mapreduce/Driver.java dda4241 src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ff05df8 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0 src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java 8e87c83 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 887f736 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java 9d87fa5 src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java 151c90b src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALObserver.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java 8e1f6e0 src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java dcd0937 src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java edca7f4 src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 0729410 src/main/java/org/apache/hadoop/hbase/util/PoolMap.java ebf3078 src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java a0bef34 src/main/ruby/hbase/admin.rb 4460d6e src/site/xdoc/index.xml 8e1b531 src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java 0b21c6c src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6d83b00 src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java b4c407b src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALActionsListener.java dc43eb2 src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALObserver.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java 381ac90 src/test/java/org/apache/hadoop/hbase/util/TestPoolMap.java 2c565d7 Diff: https://reviews.apache.org/r/1214/diff Testing --- Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. Thanks, Li Enable direct byte
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13081519#comment-13081519
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-09 08:36:02.175711)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Fixed diff. Rebase on 3857 works!
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.diff,
hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13081902#comment-13081902
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-09 20:39:55.719673)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Fixed compilation error. Corrected package location of cachetestutils.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
CHANGES.txt e9c0478
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa09b7d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 88aa652
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 886c31d
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 86652c0
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 94c8bb4
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 4387170
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v6.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13081962#comment-13081962
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1363
---
CHANGES.txt
https://reviews.apache.org/r/1214/#comment3063
your diff appears to revert this change. perhaps you need to rebase on
trunk before you take diff against it.
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
https://reviews.apache.org/r/1214/#comment3064
style:
/**
* ...
*/
public class CacheStates {
(comment formatting and space before '{')
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1214/#comment3068
hrm, is this constructor ever meant to be used? If the off-heap cache isn't
configured, then it should just instantiate LruBlockCache directly, no?
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1214/#comment3069
does it ever make sense to have offHeapSize onHeapSize? Perhaps we should
have a Preconditions check here?
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1214/#comment3065
hyphenate 'on-heap' and 'off-heap' for clarity
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1214/#comment3066
missing space - bytes ...
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1214/#comment3067
same as above
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1214/#comment3070
we should add in the heap size used by the accounting and hashmaps in the
off-heap cache as well.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
https://reviews.apache.org/r/1214/#comment3071
vertically collapse this - one line per param
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
https://reviews.apache.org/r/1214/#comment3072
when you check up front here, you end up doing two lookups in backingmap.
Since this is just a safety check, you could instead check the return value of
put() below. Something like:
ByteBuffer storedBlock = ...allloc
... fill it in...
ByteBuffer alreadyCached = backingMap.put(blockName, storedBlock);
if (alreadyCached != null) {
// we didn't insert the new one, so free it and throw an exception
backingStore.free(storedBlock);
throw new RuntimeException(already cached x);
}
make sense?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
https://reviews.apache.org/r/1214/#comment3073
I think there's a bug here if you have multiple users hammering the same
contentBlock -- two people can get to rewind() at the same time. You probably
need synchronized(contentBlock) around these two lines. See if you can add a
unit test which puts just one block in the cache and starts several threads
which hammer it - I bet you eventually one of the blocks comes back returned as
all 0x
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
https://reviews.apache.org/r/1214/#comment3074
this.size() is in units of bytes, not blocks
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
https://reviews.apache.org/r/1214/#comment3075
maybe rename to getOccupiedSize?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
https://reviews.apache.org/r/1214/#comment3076
wrong Log class - should use org.apache.commons.logging.Log
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
https://reviews.apache.org/r/1214/#comment3077
remove extra whitespace
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
https://reviews.apache.org/r/1214/#comment3078
hm, we have 4 different terms for these: buffers, items, chunks, and
blocks. Can we have a terminology that's used consistently throughout?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
https://reviews.apache.org/r/1214/#comment3079
LOG.warn(Shutdown failed!, e); is probably what you want. Also improve
the text of this error message -- eg Unable to deallocate direct memory during
shutdown.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
https://reviews.apache.org/r/1214/#comment3080
getBlock*s*Remaining, right?
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
https://reviews.apache.org/r/1214/#comment3081
incomplete comment here
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13082087#comment-13082087
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1371
---
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
https://reviews.apache.org/r/1214/#comment3155
I think this was intended for use by hbase-env.sh
- Ted
On 2011-08-09 20:39:55, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-08-09 20:39:55)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.CHANGES.txt e9c0478
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
aa09b7d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2d4002c
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java 097dc50
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
88aa652
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
886c31d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
86652c0
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
94c8bb4
bq.src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java e70b0d4
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/HFileBlockCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCacheTestUtils.java
PRE-CREATION
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSizeCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java
4387170
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027-v10.5.diff,
hbase-4027-v10.diff, hbase-4027v10.6.diff, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13075966#comment-13075966
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-01 23:54:08.232314)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Addressed most of Ted Yu/JGray's comments. Will do another patch later. Fixed
build errors. Tested patch/build on fresh pull from trunk.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, hbase-4027v6.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13075983#comment-13075983
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-08-02 00:46:57.567666)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
configuration changed to percentage of MaxDirectMemorySize rather than
specifying size of offHeapCache.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e87eb3e
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 9cc75bb
src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java c4c66e1
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, HBase4027v8.diff, HBase4027v9.diff, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072716#comment-13072716
]
Li Pi commented on HBASE-4027:
--
The bug actually broke an existing test case. I'm not sure why I didn't notice
- I did after trying to rebuild it this morning.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072940#comment-13072940 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1211 --- conf/hbase-env.sh https://reviews.apache.org/r/1214/#comment2675 Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly a bit above the off heap cache size is) src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java https://reviews.apache.org/r/1214/#comment2676 2011 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java https://reviews.apache.org/r/1214/#comment2677 whitespace src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java https://reviews.apache.org/r/1214/#comment2678 license src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java https://reviews.apache.org/r/1214/#comment2679 class comment src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2680 whitespace here and throughout this file src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2681 Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc). If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks) src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2682 longer than 80 chars src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2683 This seems like a behavior that we may not always want. If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?) we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap) src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2684 this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java https://reviews.apache.org/r/1214/#comment2685 huh? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java https://reviews.apache.org/r/1214/#comment2686 line 80 chars src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java https://reviews.apache.org/r/1214/#comment2687 getTotalNumBlocks() and getRemainingNumBlocks() or something? i find the method names a little unclear (or just add some javadoc) src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java https://reviews.apache.org/r/1214/#comment2688 javadoc on these src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java https://reviews.apache.org/r/1214/#comment2689 I'm not totally clear on why the SlabCache contains a bunch of SingleSizeCaches. Why do you need to layer BlockCaches on top of BlockCaches? You'll have one slab per size rather than one cache per size? Can you not pass the right evictor callback in so it goes back to the right slab? src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java https://reviews.apache.org/r/1214/#comment2690 Why these ratios? At the least, this should all be configurable (even if just in code and undocumented). Do we need to always pre-allocate everything and determine the block/slab sizes and all that? The design seems inflexible because it's all determine during construction rather than being adaptive. I'm okay with the first
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073087#comment-13073087 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. conf/hbase-env.sh, line 42 bq. https://reviews.apache.org/r/1214/diff/2/?file=28356#file28356line42 bq. bq. Extra whitespace is obvious on review board. Fixed. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2 bq. https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line2 bq. bq. Year of copyright. Fixed. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 48 bq. https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line48 bq. bq. Long line should wrap. Fixed. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 64 bq. https://reviews.apache.org/r/1214/diff/2/?file=28358#file28358line64 bq. bq. Indentation should be corrected for the if block. Fixed bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 2 bq. https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line2 bq. bq. Copyright missing. Fixed bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 25 bq. https://reviews.apache.org/r/1214/diff/2/?file=28359#file28359line25 bq. bq. This implies that an eviction may involve fewer than one block. bq. If so, this count can be named evictedBlockCount. Fixed. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 100 bq. https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100 bq. bq. Should we add hitCount for offHeapCache ? bq. This can be addressed in a follow-on JIRA. I think the offHeapCache should have a hitcount, but we can deal with it later. Right now, it has the full CacheStats metrics. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115 bq. https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line115 bq. bq. In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts. Agreed. I may be redoing this substantially in a follow on Jira. bq. On 2011-07-29 05:14:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 145 bq. https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line145 bq. bq. Is this a typo ? Yup, and fixed. - Li --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1220 --- On 2011-07-28 23:02:50, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-07-28 23:02:50) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073090#comment-13073090
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-07-30 00:39:48.920025)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Addressed TedYu/JGray's comments.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073097#comment-13073097 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. conf/hbase-env.sh, lines 44-45 bq. https://reviews.apache.org/r/1214/diff/1/?file=28125#file28125line44 bq. bq. Is MaxDirectMemorySize determinable on the running jvm? Could we make the offheapcachesize config as a percentage of the direct memory size like we have for memstore/blockcache today? (default of 0.95 or something would make it so it never really has to be set for most cases... and i'm not sure what exactly a bit above the off heap cache size is) I haven't figured out a way. Just asked StackOverflow - hopefully they'll have an answer. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 2 bq. https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line2 bq. bq. 2011 Fixed. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 1 bq. https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line1 bq. bq. license Fixed. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java, line 41 bq. https://reviews.apache.org/r/1214/diff/1/?file=28127#file28127line41 bq. bq. whitespace Fixed. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java, line 5 bq. https://reviews.apache.org/r/1214/diff/1/?file=28128#file28128line5 bq. bq. class comment Added bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 39 bq. https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line39 bq. bq. Would it make sense to have DoubleBlockCache be more generic? Does it need to be fixed with these two types or could it take two BlockCache's and they are executed in the order they are given in (just need to be clear in doc). bq. bq. If this was generic, it could be reused for various multi-level caches (like an underlying cache with compressed blocks and one above it with uncompressed blocks) It can be made more generic easily. I just haven't done it. Will do though - just create a constructor that takes two BlockCaches? bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 66 bq. https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line66 bq. bq. longer than 80 chars fixed. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, lines 77-79 bq. https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line77 bq. bq. This seems like a behavior that we may not always want. bq. bq. If we made this class generic, could we have some kind of policy we initiate it with? (like default cache in level one, if accessed in level one, cache in level two, etc?) bq. bq. we're going to always be double-storing anything so that the offHeap true capacity is (totalOffHeap - totalOnHeap). in some cases, we might want to cache on heap first and then if evicted we cache off heap, or maybe we want it to work more like the existing LRU (first read goes into off heap, second read upgrades it to the on heap cache and removes from the off heap) I was thinking of sending all initial caches to the off heap cache, giving it an inbuilt scan resistance, but LRUBlockCache already does partitioning to deal with scans. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java, line 115 bq. https://reviews.apache.org/r/1214/diff/1/?file=28129#file28129line115 bq. bq. this is going to make for some weird stats? seems like we may need to actually expose the stats of each underlying cache rather than both? (or both and separate). it's going to be difficult to understand what's happening when the hit and eviction stats cover both. The idea was the combined evicted stats of both, but yeah, CacheStats are implemented for both LruBlockCache and SlabCache, so exposing both might be a good idea. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java, line 74 bq. https://reviews.apache.org/r/1214/diff/1/?file=28132#file28132line74 bq. bq. huh? Just a typo. Fixed. Don't know why that line is there. bq. On 2011-07-29 17:41:50, Jonathan Gray wrote: bq.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073099#comment-13073099
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-07-29 05:14:48, Ted Yu wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 100
bq. https://reviews.apache.org/r/1214/diff/2/?file=28360#file28360line100
bq.
bq. Should we add hitCount for offHeapCache ?
bq. This can be addressed in a follow-on JIRA.
bq.
bq. Li Pi wrote:
bq. I think the offHeapCache should have a hitcount, but we can deal with
it later. Right now, it has the full CacheStats metrics.
What I meant to say, is, that it does have a hitcount. It just isn't displayed
anywhere.
- Li
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1220
---
On 2011-07-30 00:39:48, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-07-30 00:39:48)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
ecab7ca
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
150f54f
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1d5e3fa
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
b600020
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073113#comment-13073113
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1238
---
conf/hbase-env.sh
https://reviews.apache.org/r/1214/#comment2812
Still some whitespaces.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
https://reviews.apache.org/r/1214/#comment2813
Doesn't match actual name of class.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
https://reviews.apache.org/r/1214/#comment2814
I don't see SkipList being used in this class.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
https://reviews.apache.org/r/1214/#comment2815
This doesn't match the actual test.
- Ted
On 2011-07-30 00:39:48, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-07-30 00:39:48)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
ecab7ca
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
150f54f
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1d5e3fa
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
b600020
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, 4027v7.diff, HBase-4027 (1).pdf,
HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072369#comment-13072369
]
Todd Lipcon commented on HBASE-4027:
Nice work and nice write-up. One interesting thing I noticed in the benchmarks
is that the scalability of the FS cache performance is poor - 10 threads looks
like it's only twice as fast as 1 thread on the FS implementations, whereas it
scales much better on the various in-process caches. How large is the dataset
in this test? Is it small enough that all of the threads are contending to read
the same HFile block, and thus hitting the locks in HFile reader?
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072466#comment-13072466
]
Li Pi commented on HBASE-4027:
--
gimme a sec. will do it.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072472#comment-13072472
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-07-28 18:29:02.934543)
Review request for hbase and Li Pi.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072470#comment-13072470
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-07-28 18:29:54.907234)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
Added reviewers. Feel free to review anyways.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072471#comment-13072471
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
Review request for hbase, Todd Lipcon and Li Pi.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072548#comment-13072548
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-07-28 21:39:13.311916)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing (updated)
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, slabcachepatch.diff,
slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff,
slabcachepatchv3.diff, slabcachepatchv4.5.diff, slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072588#comment-13072588
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/
---
(Updated 2011-07-28 23:02:50.734090)
Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray,
and Li Pi.
Changes
---
small bugfix, off by one error in slab.java. declaring things to maximum size
would fail.
Summary
---
Review request - I apparently can't edit tlipcon's earlier posting of my diff,
so creating a new one.
This addresses bug HBase-4027.
https://issues.apache.org/jira/browse/HBase-4027
Diffs (updated)
-
conf/hbase-env.sh 2d55d27
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca
src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
Diff: https://reviews.apache.org/r/1214/diff
Testing
---
Ran benchmarks against it in HBase standalone mode. Wrote test cases for all
classes, multithreaded test cases exist for the cache.
Thanks,
Li
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[ https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072682#comment-13072682 ] jirapos...@reviews.apache.org commented on HBASE-4027: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1214/#review1220 --- conf/hbase-env.sh https://reviews.apache.org/r/1214/#comment2739 Extra whitespace is obvious on review board. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java https://reviews.apache.org/r/1214/#comment2738 Year of copyright. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java https://reviews.apache.org/r/1214/#comment2740 Long line should wrap. src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java https://reviews.apache.org/r/1214/#comment2741 Indentation should be corrected for the if block. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java https://reviews.apache.org/r/1214/#comment2743 Copyright missing. src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java https://reviews.apache.org/r/1214/#comment2744 This implies that an eviction may involve fewer than one block. If so, this count can be named evictedBlockCount. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2745 Should we add hitCount for offHeapCache ? This can be addressed in a follow-on JIRA. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2746 In a follow-on JIRA, we can distinguish between onHeap eviction and offHeap eviction counts. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java https://reviews.apache.org/r/1214/#comment2747 Is this a typo ? - Ted On 2011-07-28 23:02:50, Li Pi wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1214/ bq. --- bq. bq. (Updated 2011-07-28 23:02:50) bq. bq. bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan Gray, and Li Pi. bq. bq. bq. Summary bq. --- bq. bq. Review request - I apparently can't edit tlipcon's earlier posting of my diff, so creating a new one. bq. bq. bq. This addresses bug HBase-4027. bq. https://issues.apache.org/jira/browse/HBase-4027 bq. bq. bq. Diffs bq. - bq. bq.conf/hbase-env.sh 2d55d27 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java ecab7ca bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 150f54f bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1d5e3fa bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020 bq. src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1214/diff bq. bq. bq. Testing bq. --- bq. bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for all classes, multithreaded test cases exist for the cache. bq. bq. bq. Thanks, bq. bq. Li bq. bq. Enable direct byte buffers LruBlockCache Key: HBASE-4027 URL: https://issues.apache.org/jira/browse/HBASE-4027 Project: HBase Issue Type: Improvement Reporter: Jason Rutherglen Assignee: Li Pi Priority: Minor Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff, slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072684#comment-13072684
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1214/#review1221
---
I see you fixed a bug, but you didn't add a new test case! Bug fixes should
include regression tests. Also, the new diff changed some formatting - looks
like some 80char lines now. Will do a full review tomorrow.
- Todd
On 2011-07-28 23:02:50, Li Pi wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1214/
bq. ---
bq.
bq. (Updated 2011-07-28 23:02:50)
bq.
bq.
bq. Review request for hbase, Todd Lipcon, Ted Yu, Michael Stack, Jonathan
Gray, and Li Pi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Review request - I apparently can't edit tlipcon's earlier posting of my
diff, so creating a new one.
bq.
bq.
bq. This addresses bug HBase-4027.
bq. https://issues.apache.org/jira/browse/HBase-4027
bq.
bq.
bq. Diffs
bq. -
bq.
bq.conf/hbase-env.sh 2d55d27
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 509121d
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheTestUtils.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java
ecab7ca
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java
150f54f
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/Slab.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabItemEvictionWatcher.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1d5e3fa
bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
b600020
bq.
src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSingleSlabCache.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlab.java
PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/io/hfile/slab/TestSlabCache.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1214/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran benchmarks against it in HBase standalone mode. Wrote test cases for
all classes, multithreaded test cases exist for the cache.
bq.
bq.
bq. Thanks,
bq.
bq. Li
bq.
bq.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: 4027-v5.diff, HBase-4027.pdf, hbase-4027v6.diff,
slabcachepatch.diff, slabcachepatchv2.diff, slabcachepatchv3.1.diff,
slabcachepatchv3.2.diff, slabcachepatchv3.diff, slabcachepatchv4.5.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070965#comment-13070965
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. pom.xml, line 712
bq. https://reviews.apache.org/r/1191/diff/1/?file=26914#file26914line712
bq.
bq. did you determine that this ConcurrentLinkedHashMap was different
than the one in Guava? I thought it got incorporated into Guava, which we
already depend on.
Changed to MapMaker. Guava has deprecated some of the functions I'm using in
trunk, but hasn't provided a release with a replacement. But we'll deal with
that when the time comes.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 33
bq. https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line33
bq.
bq. punctuation wise, I think it would be easier to read if you
hyphenated on-heap and off-heap. This applies to log messages below as well.
Done
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
lines 52-55
bq. https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line52
bq.
bq. No need to line-break here
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
lines 64-65
bq. https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line64
bq.
bq. consider using StringUtils.humanReadableInt for these sizes.
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 72
bq. https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line72
bq.
bq. @Override
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
lines 77-85
bq. https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line77
bq.
bq. when you're just overriding something from the superclass, no need
for javadoc unless it says something new and exciting. If you feel like you
want to put something there, you can use /** {@inheritDoc} */ to be explicit
that you're inheriting from the superclass.
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 102
bq. https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line102
bq.
bq. I think you should only put-back into the on-heap cache in the case
that the 'caching' parameter is true.
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 128
bq. https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line128
bq.
bq. hrm, the class javadoc says that the statistics should be
cumulative, but this seems to just forward
Fixed. Also refactored CacheStats into a separate class.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
lines 148-166
bq. https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line148
bq.
bq. TODOs
Implemented these.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java,
line 168
bq. https://reviews.apache.org/r/1191/diff/1/?file=26915#file26915line168
bq.
bq. is this code used? seems like dead copy-paste code to me.
It is now.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java, line 1125
bq. https://reviews.apache.org/r/1191/diff/1/?file=26916#file26916line1125
bq.
bq. extraneous debugging left in
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, line 31
bq. https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line31
bq.
bq. I think this is usually called a slab class - I think that name
would be less confusing, since Meta is already used in HBase to refer to
.META.
Fixed.
bq. On 2011-07-25 23:48:42, Todd Lipcon wrote:
bq. src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java, lines 32-34
bq. https://reviews.apache.org/r/1191/diff/1/?file=26917#file26917line32
bq.
bq. unclear what the difference is between the two.
bq.
bq. Is slabs the list of 2GB buffers, and buffers is the list of
actual items that will be allocated? I think the traditional names here are
slabs and items. where each slab holds some number of allocatable items
bq.
bq. Also, rather than // comments, use /** javadoc comments */ before
the vars
Fixed.
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070845#comment-13070845
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1191/
---
Review request for hbase and Todd Lipcon.
Summary
---
Uploading slabcachepatchv4 to review for Li Pi.
This addresses bug HBASE-4027.
https://issues.apache.org/jira/browse/HBASE-4027
Diffs
-
conf/hbase-env.sh 2d55d27
pom.xml 729dc37
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 5963552
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/io/hfile/SlabCache.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java b600020
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSingleSlabCache.java
PRE-CREATION
src/test/java/org/apache/hadoop/hbase/io/hfile/TestSlabCache.java
PRE-CREATION
Diff: https://reviews.apache.org/r/1191/diff
Testing
---
Thanks,
Todd
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070859#comment-13070859
]
stack commented on HBASE-4027:
--
Doc is great.
These could be final:
+ private LruBlockCache onHeapCache;
+ private SlabCache offHeapCache;
Says 'Metrics are the combined size and hits and misses of both caches' but
down in getStats we seem to be getting onheap stats only. Intentional? Same
for heapSize.
Do you want to leave this line in hfile? + LOG.debug(decompressedSize =
+ decompressedSize);
Whats it mean when you say 'An exception will be thrown if the cached data is
larger than the size of the allocated block'?
More notes later.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff,
slabcachepatchv4.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070862#comment-13070862
]
jirapos...@reviews.apache.org commented on HBASE-4027:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1191/#review1182
---
could do with some tests for MetaSlab. also some multi-threaded tests - see
MultithreadedTestUtil, example usage in TestMemStoreLAB
pom.xml
https://reviews.apache.org/r/1191/#comment2484
did you determine that this ConcurrentLinkedHashMap was different than the
one in Guava? I thought it got incorporated into Guava, which we already depend
on.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1191/#comment2485
punctuation wise, I think it would be easier to read if you hyphenated
on-heap and off-heap. This applies to log messages below as well.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1191/#comment2486
No need to line-break here
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1191/#comment2487
consider using StringUtils.humanReadableInt for these sizes.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1191/#comment2488
@Override
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1191/#comment2489
when you're just overriding something from the superclass, no need for
javadoc unless it says something new and exciting. If you feel like you want to
put something there, you can use /** {@inheritDoc} */ to be explicit that
you're inheriting from the superclass.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1191/#comment2490
I think you should only put-back into the on-heap cache in the case that
the 'caching' parameter is true.
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1191/#comment2491
hrm, the class javadoc says that the statistics should be cumulative, but
this seems to just forward
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1191/#comment2492
TODOs
src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java
https://reviews.apache.org/r/1191/#comment2493
is this code used? seems like dead copy-paste code to me.
src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
https://reviews.apache.org/r/1191/#comment2497
extraneous debugging left in
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
https://reviews.apache.org/r/1191/#comment2498
I think this is usually called a slab class - I think that name would be
less confusing, since Meta is already used in HBase to refer to .META.
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
https://reviews.apache.org/r/1191/#comment2499
unclear what the difference is between the two.
Is slabs the list of 2GB buffers, and buffers is the list of actual
items that will be allocated? I think the traditional names here are slabs
and items. where each slab holds some number of allocatable items
Also, rather than // comments, use /** javadoc comments */ before the vars
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
https://reviews.apache.org/r/1191/#comment2500
these vars probably better called maxBlocksPerSlab and maxSlabSize, since
they're upper bounds.
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
https://reviews.apache.org/r/1191/#comment2501
I think this code would be a little easier to understand if you split it
into one loop for the full slabs, and an if statement for the partially full
one. Something like:
int numFullSlabs = numBlocks / maxBlocksPerSlab;
boolean hasPartialSlab = (numBlocks % maxBlocksPerSlab) 0;
for (int i = 0; i numFullSlabs; i++) {
alloc one of maxSlabSize;
addBuffersForSlab(slab);
}
if (hasPartialSlab) {
alloc the partial one
addBuffersForSlab(slab);
}
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
https://reviews.apache.org/r/1191/#comment2502
should be a LOG.warn
src/main/java/org/apache/hadoop/hbase/io/hfile/MetaSlab.java
https://reviews.apache.org/r/1191/#comment2503
shouldn't this class have an alloc() and free() method?
src/main/java/org/apache/hadoop/hbase/io/hfile/SingleSizeCache.java
https://reviews.apache.org/r/1191/#comment2511
shouldn't this implement BlockCache?
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068240#comment-13068240
]
Li Pi commented on HBASE-4027:
--
-And when we cache a block, how we know where to put it? On or off heap? How
you decide where to put it (Looks like you put it off heap always here).
When we catch a block, we throw it into on-heap. Originally, I planned for it
to go off heap, but as the cache partitions in LRUBlockCache gives it scan
resistance already, this seemed unnecessary.
-Any chance of some offheap stats (is this getStats used?)
getStats now works for the offheap cache, at least for eviction count/cache hit
rates.
-We only return heapsize of onheap cache. You think we should not include
offheap?
I figured since offheap wasn't really in the heap, I didn't include it. Not
sure if this was the best option.
-Yeah, a bunch of these state methods go unimplemented. Can we do any of them?
Or is it that they just don't make sense in offheap context?
I took a look, implemented some of them. SlabCache implements everything except
getFreeSize. Since all the space it's given is allocated using
directbytebuffers immediately, theres never any free size. So I have it return
0.
-What is this limit? int blocksPerSlab = Integer.MAX_VALUE / blockSize; Max of
4G per slab?
ByteBuffer positions are addressed using integers. You can only declare one up
to 2gb using ByteBuffer.allocateDirect().
-Should slabsize just be hardcoded as Integer.MAX_VALUE?
Sometimes we want a slab thats less than 2gb. Such as when the size for our
entire cache is smaller than that.
-The kung fu going on in the shutdown of metaslab needs a comment. I think I
know whats going on. Explain what 'cleaner' is.
Yeah, this is how you deallocate directbytebuffers. Added comments, but its
still pretty voodoo. Basically cleaner is a deconstructor for a direct byte
buffer.
-Is there something up w/ the naming in MetaSlab or is it me misreading? We are
iterating up to blocksPerSlab but we are creating slabs, not blocks. I'd think
we'd be making blocks in a slab, not slabs.
If we want to create N blocks of size X, and a slab, can, at max, contain B
blocks, we decrement N by B until N is below B. We create a slab each time and
divide that slab into blocks. If N B and N 0, then we create our final slab
that we use to get space for the remaining blocks.
-You think this is going to happen or how could it happen?
This shouldn't happen unless we have a race condition. But we might, in which
case we should throw an exception. I'll add some comments to this portion.
-Whats going on here (I see this in a few places):
If we get a null pointer exception, that means the cache missed. Therefore we
return null, increment the missed counter.
-The get from backingStore will never return a null here?
As an invariant, it should never. Because if we are running out of buffers, one
should be evicted by the ConcurrentLinkedHashMap when we do a read. On closer
inspection, this can happen in a multithreaded environment, I'll figure out a
way to fix this. (Probably by synchronizing it.)
-Is this a good name for the default blocksize config?
hbase.mapreduce.hfileoutputformat.blocksize The 'outputformat' would seem to
come from mapreduce world (Which seems to be where you found it if I grep src).
SHould we be using DEFAULT_BLOCKSIZE from hfile here instead?
Switched it to DEFAULT_BLOCKSIZE.
Apologies for the first few patches. Apparently I diffed against a different
branch than I had intended. Thus the pom.xml edits and commented out code. That
was for speed testing and benchmarking. I fixed a few more things, and will
continue to search for bugs and add documentation. I'm also implementing better
metrics.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068513#comment-13068513
]
stack commented on HBASE-4027:
--
bq. When we catch a block, we throw it into on-heap.
At one point does stuff go into the offheap cache?
bq. I figured since offheap wasn't really in the heap, I didn't include it. Not
sure if this was the best option.
How would an operator monitor offheap if no stats. What should they do? Its
kinda tough because the cache interface only allows return of one 'size'
bq. Apologies for the first few patches.
Don't apologize. This back and forth is healthy.
Looks like the bulk of my comments above can be addressed by comment in code.
Let us know when you think it ready for new review.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068539#comment-13068539
]
Li Pi commented on HBASE-4027:
--
|At one point does stuff go into the offheap cache?
Err, when we catch a block, it gets thrown into both on-heap and off-heap.
|How would an operator monitor offheap if no stats. What should they do? Its
kinda tough because the cache interface only allows return of one 'size'
Thats an unfortunate problem, seeing as the off heap cache is technically made
of many differently sized caches. I'll have them log info the same was as the
on heap cache does every once in a while.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068544#comment-13068544
]
Todd Lipcon commented on HBASE-4027:
In my opinion we need a servlet or jamon page which can dump the stats of the
various slab classes as well as their hit/miss rates.
Alternatively, a new MetricsContext for the slab info.
But, somehow, we need to expose information similar to memcached's stats
items and stats slabs
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068549#comment-13068549
]
Li Pi commented on HBASE-4027:
--
Yeah, thats what I'm working on as well. Some easy visual way to expose stats.
Time to learn webdev!
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068582#comment-13068582
]
stack commented on HBASE-4027:
--
@Li Related is HBASE-4057?
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068706#comment-13068706
]
Ted Yu commented on HBASE-4027:
---
In TestSlabCache:
{code}
+Block[] blocks = generateRandomBlocks(87, blockSize);
{code}
We'd better cover more cases where cache miss is exercised.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.1.diff, slabcachepatchv3.2.diff, slabcachepatchv3.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067598#comment-13067598
]
Ted Yu commented on HBASE-4027:
---
onHeapCache should be used in getBlock():
{code}
+ /* if ((cachedBlock = onHeapCache.getBlock(blockName, caching)) != null) {
+ return cachedBlock;
+} else */ if ((cachedBlock = offHeapCache.getBlock(blockName, caching)) !=
null) {
+ // onHeapCache.cacheBlock(blockName, cachedBlock);
{code}
Otherwise onHeapCache.cacheBlock() should be commented out in:
{code}
+ public void cacheBlock(String blockName, ByteBuffer buf, boolean inMemory) {
+onHeapCache.cacheBlock(blockName, buf, inMemory);
+offHeapCache.cacheBlock(blockName, buf);
+ }
{code}
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067842#comment-13067842
]
Li Pi commented on HBASE-4027:
--
whoops. I have no idea why thats still commented out.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067930#comment-13067930
]
Li Pi commented on HBASE-4027:
--
I'm gonna upload a PDF of benchmark results, but results are interesting.
Running HBase standalone, with a small enough dataset so that all files are in
local FS cache, the SlabCache substantially outperforms the FS Cache, even if
checksumming is disabled.
Compression is disabled as well.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067992#comment-13067992
]
Ted Yu commented on HBASE-4027:
---
Why is the handling below different from that in cacheBlock(String blockName,
ByteBuffer buf, boolean inMemory) ?
{code}
public void cacheBlock(String blockName, ByteBuffer buf) {
// onHeapCache.cacheBlock(blockName, buf);
{code}
Overall, more javadoc would help others grasp the functionality for each class.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067993#comment-13067993
]
stack commented on HBASE-4027:
--
Patch looks excellent!
You want that change to surefire heap size in pom.xml? Are you saying here
that tests won't pass unless you have 5G of RAM?
In apache code base, you cannot have '@author' tags.
WTF does DoubleBlockCache dooo? (smile). Usually folks will fill in the class
comment on the what/why a class exists. Would be good here; e.g. why is it
called DoubleBlockCache? (Similar for other new classes you've added)
I love this:
{code}
+ private LruBlockCache onHeapCache;
+ private SlabCache offHeapCache;
{code}
Thats sweet.
Remove code rather than comment it out: e.g. +//
onHeapCache.cacheBlock(blockName, buf);
And when we cache a block, how we know where to put it? On or off heap? How
you decide where to put it (Looks like you put it off heap always here).
Any chance of some offheap stats (is this getStats used?)
We only return heapsize of onheap cache. You think we should not include
offheap?
Yeah, a bunch of these state methods go unimplemented. Can we do any of them?
Or is it that they just don't make sense in offheap context?
Do the assign when you declare the blockSize and slabs data members rather than
wait till the constructor runs. Make them final while you are at it. Similar
comment for SingleSizeCache, etc.
What is this limit? int blocksPerSlab = Integer.MAX_VALUE / blockSize; Max of
4G per slab?
Should slabsize just be hardcoded as Integer.MAX_VALUE?
The kung fu going on in the shutdown of metaslab needs a comment. I think I
know whats going on. Explain what 'cleaner' is.
Is there something up w/ the naming in MetaSlab or is it me misreading? We are
iterating up to blocksPerSlab but we are creating slabs, not blocks. I'd think
we'd be making blocks in a slab, not slabs.
You think this is going to happen or how could it happen?
{code}
+if(backingMap.containsKey(blockName)){
+ throw new RuntimeException(Cached an already cached block);
+}
{code}
Whats going on here (I see this in a few places):
{code}
+} catch (Exception e) {
+
+}
{code}
The get from backingStore will never return a null here?
+ contentBlock = backingStore.get(key).getBlock(key, caching);
Is this a good name for the default blocksize config?
hbase.mapreduce.hfileoutputformat.blocksize The 'outputformat' would seem to
come from mapreduce world (Which seems to be where you found it if I grep src).
SHould we be using DEFAULT_BLOCKSIZE from hfile here instead?
There are some copy paste issues with the javadoc you have in your
TestSingleSlabCache.
Hey Li, you get an F in documentation (smile) but the code is great. Good
stuff.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067999#comment-13067999
]
Li Pi commented on HBASE-4027:
--
Thanks!
Theres quite a few bugs I just discovered. I'm fixing them right now. I'm
adding reasonable documentation too now.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068011#comment-13068011
]
Li Pi commented on HBASE-4027:
--
@Ted Yu - thats also a bug. Apparently, I diffed against a testing branch of
this patch. Whoops. This explains the commented out code.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068021#comment-13068021
]
Li Pi commented on HBASE-4027:
--
@Ted, if it NPE's, that means that the cache doesn't contain the required item,
so it's a cache miss.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Attachments: slabcachepatch.diff, slabcachepatchv2.diff,
slabcachepatchv3.diff
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064931#comment-13064931
]
Li Pi commented on HBASE-4027:
--
I have a copy based variant working right now, I'll submit the patch by the end
of this week for that. Benchmarks show roughly 75% of the speed of the existing
LruBlockCache, and around 4x the speed of FS cache reads. Realistically, we'll
be using a two stage cache solution, with the LruBlockCache also holding some
values, so for the majority of reads, it'll be the same speed as the
LruBlockCache.
Currently pursueing a 0 copy solution - going to need to make everything run
off ByteBuffers rather than byte[] though.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064983#comment-13064983
]
stack commented on HBASE-4027:
--
bq. Currently pursueing a 0 copy solution - going to need to make everything
run off ByteBuffers rather than byte[] though.
Go for it. That should fill your summer Li. Good stuff.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Assignee: Li Pi
Priority: Minor
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054639#comment-13054639
]
Jonathan Gray commented on HBASE-4027:
--
In the new HFile v2 over in HBASE-3857 the block cache interface changes from
ByteBuffer to HeapSize. So you can now put anything you want into the cache
that implements HeapSize (there is a new HFileBlock that is used in HFile v2).
One big question is whether you're going to make copies out of the direct byte
buffers on each read of that block, or if you're going to change KeyValue to
use the ByteBuffer interface (or some other) instead of the byte[] directly.
With a DBB you can't get access to an underlying byte[].
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Priority: Minor
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054660#comment-13054660
]
Li Pi commented on HBASE-4027:
--
This would be really useful. I think even making copies out of the direct byte
buffers would confer a substantial performance advantage over the FS cache.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Priority: Minor
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054737#comment-13054737
]
Jason Rutherglen commented on HBASE-4027:
-
{quote}One big question is whether you're going to make copies out of the
direct byte buffers on each read of that block, or if you're going to change
KeyValue to use the ByteBuffer interface (or some other) instead of the byte[]
directly{quote}
Right the {{HFile.Scanner.getKeyValue()}} method is calling {{block.array()}}.
We'd need to track down all {{byte[]}} references, and convert them to
{{ByteBuffer}}. That's more of a separate Jira.
I think converting a direct ByteBuffer to byte[] will generate a fair amount of
garbage, though of a different (smaller and more numerous) kind than the blocks.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Priority: Minor
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4027) Enable direct byte buffers LruBlockCache
[
https://issues.apache.org/jira/browse/HBASE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054747#comment-13054747
]
Jason Rutherglen commented on HBASE-4027:
-
{quote}This would be really useful. I think even making copies out of the
direct byte buffers would confer a substantial performance advantage over the
FS cache.{quote}
The filesystem cache doesn't help because HBase needs quick access to
uncompressed blocks for scanning. For what duration does an uncompressed block
need to be cached? In either case, accessing compressed blocks from the FS
cache will be faster than hitting the disk or network. I am guessing one can
maintain a small'ish block cache, ensure HDFS blocks are local, provide extra
space for the FS cache, and gain in read throughput.
Snappy should decompress fast enough for this to be more viable than
maintaining a large-ish block cache. The problem [today] with a small'ish
block cache is the GC is driven mad.
Enable direct byte buffers LruBlockCache
Key: HBASE-4027
URL: https://issues.apache.org/jira/browse/HBASE-4027
Project: HBase
Issue Type: Improvement
Reporter: Jason Rutherglen
Priority: Minor
Java offers the creation of direct byte buffers which are allocated outside
of the heap.
They need to be manually free'd, which can be accomplished using an
documented {{clean}} method.
The feature will be optional. After implementing, we can benchmark for
differences in speed and garbage collection observances.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
