Add javadoc on how BBIOEngine works (by Anoop Sam John)
Project: http://git-wip-us.apache.org/repos/asf/hbase/repo Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/891569bb Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/891569bb Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/891569bb Branch: refs/heads/hbase-12439 Commit: 891569bbd92f95797033c0706c9de9617228415b Parents: 7073f69 Author: stack <[email protected]> Authored: Tue Mar 22 13:01:13 2016 -0700 Committer: stack <[email protected]> Committed: Tue Mar 22 13:01:13 2016 -0700 ---------------------------------------------------------------------- .../io/hfile/bucket/ByteBufferIOEngine.java | 33 +++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hbase/blob/891569bb/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/ByteBufferIOEngine.java ---------------------------------------------------------------------- diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/ByteBufferIOEngine.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/ByteBufferIOEngine.java index 45ed1ae..63de32c 100644 --- a/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/ByteBufferIOEngine.java +++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/ByteBufferIOEngine.java @@ -31,7 +31,38 @@ import org.apache.hadoop.hbase.util.ByteBufferArray; /** * IO engine that stores data in memory using an array of ByteBuffers - * {@link ByteBufferArray} + * {@link ByteBufferArray}. + * + *<h2>How it Works</h2> + * First, see {@link ByteBufferArray} and how it gives a view across multiple ByteBuffers managed + * by it internally. This class does the physical BB create and the write and read to the + * underlying BBs. So we will create N BBs based on the total BC capacity specified on create + * of the ByteBufferArray. So say we have 10 GB of off heap BucketCache, we will create 2560 such + * BBs inside our ByteBufferArray. + * + * <p>Now the way BucketCache works is that the entire 10 GB is split into diff sized buckets: by + * default from 5 KB to 513 KB. Within each bucket of a particular size, there are + * usually more than one bucket 'block'. The way it is calculate in bucketcache is that the total + * bucketcache size is divided by 4 (hard-coded currently) * max size option. So using defaults, + * buckets will be is 4 * 513kb (the biggest default value) = 2052kb. A bucket of 2052kb at offset + * zero will serve out bucket 'blocks' of 5kb, the next bucket will do the next size up and so on + * up to the maximum (default) of 513kb). + * + * <p>When we write blocks to the bucketcache, we will see which bucket size group it best fits. + * So a 4 KB block size goes to the 5 KB size group. Each of the block writes, writes within its + * appropriate bucket. Though the bucket is '4kb' in size, it will occupy one of the + * 5 KB bucket 'blocks' (even if actual size of the bucket is less). Bucket 'blocks' will not span + * buckets. + * + * <p>But you can see the physical memory under the bucket 'blocks' can be split across the + * underlying backing BBs from ByteBufferArray. All is split into 4 MB sized BBs. + * + * <p>Each Bucket knows its offset in the entire space of BC and when block is written the offset + * arrives at ByteBufferArray and it figures which BB to write to. It may so happen that the entire + * block to be written does not fit a particular backing ByteBufferArray so the remainder goes to + * another BB. See {@link ByteBufferArray#putMultiple(long, int, byte[])}. + +So said all these, when we read a block it may be possible that the bytes of that blocks is physically placed in 2 adjucent BBs. In such case also, we avoid any copy need by having the MBB... */ @InterfaceAudience.Private public class ByteBufferIOEngine implements IOEngine {
