http://git-wip-us.apache.org/repos/asf/hbase/blob/cb77a925/src/main/docbkx/appendix_hfile_format.xml
----------------------------------------------------------------------
diff --git a/src/main/docbkx/appendix_hfile_format.xml
b/src/main/docbkx/appendix_hfile_format.xml
deleted file mode 100644
index ee43031..0000000
--- a/src/main/docbkx/appendix_hfile_format.xml
+++ /dev/null
@@ -1,657 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<appendix version="5.0" xml:id="hfile_format"
- xmlns="http://docbook.org/ns/docbook";
xmlns:xlink="http://www.w3.org/1999/xlink";
- xmlns:xi="http://www.w3.org/2001/XInclude";
xmlns:svg="http://www.w3.org/2000/svg";
- xmlns:m="http://www.w3.org/1998/Math/MathML";
xmlns:html="http://www.w3.org/1999/xhtml";
- xmlns:db="http://docbook.org/ns/docbook";>
- <!--
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
--->
- <title>HFile format</title>
- <para>This appendix describes the evolution of the HFile format.</para>
-
- <section xml:id="hfilev1">
- <title>HBase File Format (version 1)</title>
- <para>As we will be discussing changes to the HFile format, it is useful
to give a short overview of the original (HFile version 1) format.</para>
- <section xml:id="hfilev1.overview">
- <title>Overview of Version 1</title>
- <para>An HFile in version 1 format is structured as follows:</para>
- <figure>
- <title>HFile V1 Format</title>
- <mediaobject>
- <imageobject>
- <imagedata align="center" valign="middle" fileref="hfile.png"/>
- </imageobject>
- <textobject>
- <phrase>HFile Version 1</phrase>
- </textobject>
- <caption><para>Image courtesy of Lars George, <link
-
xlink:href="http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html";
-
>hbase-architecture-101-storage.html</link>.</para></caption>
- </mediaobject>
- </figure>
-
- </section>
- <section><title> Block index format in version 1 </title>
- <para>The block index in version 1 is very straightforward. For each entry,
it contains: </para>
- <orderedlist>
- <listitem>
- <para>Offset (long)</para>
- </listitem>
- <listitem>
- <para>Uncompressed size (int)</para>
- </listitem>
- <listitem>
- <para>Key (a serialized byte array written using
Bytes.writeByteArray) </para>
- <orderedlist>
- <listitem>
- <para>Key length as a variable-length integer (VInt)
- </para>
- </listitem>
- <listitem>
- <para>
- Key bytes
- </para>
- </listitem>
- </orderedlist>
- </listitem>
- </orderedlist>
- <para>The number of entries in the block index is stored in the fixed file
trailer, and has to be passed in to the method that reads the block index. One
of the limitations of the block index in version 1 is that it does not provide
the compressed size of a block, which turns out to be necessary for
decompression. Therefore, the HFile reader has to infer this compressed size
from the offset difference between blocks. We fix this limitation in version 2,
where we store on-disk block size instead of uncompressed size, and get
uncompressed size from the block header.</para>
- </section>
- </section>
- <section xml:id="hfilev2"><title>
- HBase file format with inline blocks (version 2)
- </title>
- <para>Note: this feature was introduced in HBase 0.92</para>
- <section><title>Motivation </title>
- <para>We found it necessary to revise the HFile format after encountering
high memory usage and slow startup times caused by large Bloom filters and
block indexes in the region server. Bloom filters can get as large as 100 MB
per HFile, which adds up to 2 GB when aggregated over 20 regions. Block indexes
can grow as large as 6 GB in aggregate size over the same set of regions. A
region is not considered opened until all of its block index data is loaded.
Large Bloom filters produce a different performance problem: the first get
request that requires a Bloom filter lookup will incur the latency of loading
the entire Bloom filter bit array.</para>
- <para>To speed up region server startup we break Bloom filters and block
indexes into multiple blocks and write those blocks out as they fill up, which
also reduces the HFile writerâs memory footprint. In the Bloom filter case,
âfilling up a blockâ means accumulating enough keys to efficiently utilize
a fixed-size bit array, and in the block index case we accumulate an âindex
blockâ of the desired size. Bloom filter blocks and index blocks (we call
these âinline blocksâ) become interspersed with data blocks, and as a side
effect we can no longer rely on the difference between block offsets to
determine data block length, as it was done in version 1.</para>
- <para>HFile is a low-level file format by design, and it should not deal
with application-specific details such as Bloom filters, which are handled at
StoreFile level. Therefore, we call Bloom filter blocks in an HFile "inline"
blocks. We also supply HFile with an interface to write those inline blocks.
</para>
- <para>Another format modification aimed at reducing the region server
startup time is to use a contiguous âload-on-openâ section that has to be
loaded in memory at the time an HFile is being opened. Currently, as an HFile
opens, there are separate seek operations to read the trailer, data/meta
indexes, and file info. To read the Bloom filter, there are two more seek
operations for its âdataâ and âmetaâ portions. In version 2, we seek
once to read the trailer and seek again to read everything else we need to open
the file from a contiguous block.</para></section>
- <section xml:id="hfilev2.overview">
- <title>Overview of Version 2</title>
- <para>The version of HBase introducing the above features reads both
version 1 and 2 HFiles, but only writes version 2 HFiles. A version 2 HFile is
structured as follows:
- <inlinemediaobject>
- <imageobject>
- <imagedata align="center" valign="middle"
fileref="hfilev2.png" />
- </imageobject>
- <textobject>
- <phrase>HFile Version 2</phrase>
- </textobject>
- </inlinemediaobject>
-
- </para>
- </section>
- <section><title>Unified version 2 block format</title>
- <para>In the version 2 every block in the data section contains the
following fields: </para>
- <orderedlist>
- <listitem>
- <para>8 bytes: Block type, a sequence of bytes equivalent to version
1's "magic records". Supported block types are: </para>
- <orderedlist>
- <listitem>
- <para>DATA â data blocks
- </para>
- </listitem>
- <listitem>
- <para>
- LEAF_INDEX â leaf-level index blocks in a
multi-level-block-index
- </para>
- </listitem>
- <listitem>
- <para>
- BLOOM_CHUNK â Bloom filter chunks
- </para>
- </listitem>
- <listitem>
- <para>
- META â meta blocks (not used for Bloom filters in
version 2 anymore)
- </para>
- </listitem>
- <listitem>
- <para>
- INTERMEDIATE_INDEX â intermediate-level index blocks in
a multi-level blockindex
- </para>
- </listitem>
- <listitem>
- <para>
- ROOT_INDEX â root>level index blocks in a multi>level
block index
- </para>
- </listitem>
- <listitem>
- <para>
- FILE_INFO â the âfile infoâ block, a small
key>value map of metadata
- </para>
- </listitem>
- <listitem>
- <para>
- BLOOM_META â a Bloom filter metadata block in the
load>on>open section
- </para>
- </listitem>
- <listitem>
- <para>
- TRAILER â a fixed>size file trailer. As opposed to the
above, this is not an
- HFile v2 block but a fixed>size (for each HFile version)
data structure
- </para>
- </listitem>
- <listitem>
- <para>
- INDEX_V1 â this block type is only used for legacy
HFile v1 block
- </para>
- </listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Compressed size of the block's data, not including the header
(int).
- </para>
- <para>
-Can be used for skipping the current data block when scanning HFile data.
- </para>
- </listitem>
- <listitem>
- <para>Uncompressed size of the block's data, not including the header
(int)</para>
- <para>
- This is equal to the compressed size if the compression algorithm is NONE
- </para>
- </listitem>
- <listitem>
- <para>File offset of the previous block of the same type (long)</para>
- <para>
- Can be used for seeking to the previous data/index block
- </para>
- </listitem>
- <listitem>
- <para>Compressed data (or uncompressed data if the compression
algorithm is NONE).</para>
- </listitem>
- </orderedlist>
- <para>The above format of blocks is used in the following HFile
sections:</para>
- <orderedlist>
- <listitem>
- <para>Scanned block section. The section is named so because it
contains all data blocks that need to be read when an HFile is scanned
sequentially. Â Also contains leaf block index and Bloom chunk blocks. </para>
- </listitem>
- <listitem>
- <para>Non-scanned block section. This section still contains
unified-format v2 blocks but it does not have to be read when doing a
sequential scan. This section contains âmetaâ blocks and intermediate-level
index blocks.
- </para>
- </listitem>
- </orderedlist>
- <para>We are supporting âmetaâ blocks in version 2 the same way they
were supported in version 1, even though we do not store Bloom filter data in
these blocks anymore. </para></section>
-
-<section><title> Block index in version 2</title>
- <para>There are three types of block indexes in HFile version 2, stored in
two different formats (root and non-root): </para>
- <orderedlist>
- <listitem>
- <para>Data index â version 2 multi-level block index, consisting
of:</para>
- <orderedlist>
- <listitem>
- <para>
- Version 2 root index, stored in the data block index section of the file
- </para>
- </listitem>
- <listitem>
- <para>
-Optionally, version 2 intermediate levels, stored in the non%root format in
the data index section of the file. Intermediate levels can only be present
if leaf level blocks are present
- </para>
- </listitem>
- <listitem>
- <para>
-Optionally, version 2 leaf levels, stored in the non%root format inline with
data blocks
- </para>
- </listitem>
- </orderedlist>
- </listitem>
- <listitem>
- <para>Meta index â version 2 root index format only, stored in the
meta index section of the file</para>
- </listitem>
- <listitem>
- <para>Bloom index â version 2 root index format only, stored in the
âload-on-openâ section as part of Bloom filter metadata.</para>
- </listitem>
- </orderedlist></section>
-<section><title>
- Root block index format in version 2</title>
- <para>This format applies to:</para>
- <orderedlist>
- <listitem>
- <para>Root level of the version 2 data index</para>
- </listitem>
- <listitem>
- <para>Entire meta and Bloom indexes in version 2, which are always
single-level. </para>
- </listitem>
- </orderedlist>
- <para>A version 2 root index block is a sequence of entries of the
following format, similar to entries of a version 1 block index, but storing
on-disk size instead of uncompressed size. </para>
- <orderedlist>
- <listitem>
- <para>Offset (long) </para>
- <para>
-This offset may point to a data block or to a deeper>level index block.
- </para>
- </listitem>
- <listitem>
- <para>On-disk size (int) </para>
- </listitem>
- <listitem>
- <para>Key (a serialized byte array stored using Bytes.writeByteArray)
</para>
- <orderedlist>
- <listitem>
- <para>Key (VInt)
- </para>
- </listitem>
- <listitem>
- <para>Key bytes
- </para>
- </listitem>
- </orderedlist>
- </listitem>
- </orderedlist>
- <para>A single-level version 2 block index consists of just a single root
index block. To read a root index block of version 2, one needs to know the
number of entries. For the data index and the meta index the number of entries
is stored in the trailer, and for the Bloom index it is stored in the compound
Bloom filter metadata.</para>
-
- <para>For a multi-level block index we also store the following fields in
the root index block in the load-on-open section of the HFile, in addition to
the data structure described above:</para>
- <orderedlist>
- <listitem>
- <para>Middle leaf index block offset</para>
- </listitem>
- <listitem>
- <para>Middle leaf block on-disk size (meaning the leaf index block
containing the reference to the âmiddleâ data block of the file) </para>
- </listitem>
- <listitem>
- <para>The index of the mid-key (defined below) in the middle
leaf-level block.</para>
- </listitem>
- </orderedlist>
- <para/>
- <para>These additional fields are used to efficiently retrieve the mid-key
of the HFile used in HFile splits, which we define as the first key of the
block with a zero-based index of (n â 1) / 2, if the total number of blocks
in the HFile is n. This definition is consistent with how the mid-key was
determined in HFile version 1, and is reasonable in general, because blocks are
likely to be the same size on average, but we donât have any estimates on
individual key/value pair sizes. </para>
- <para/>
- <para>When writing a version 2 HFile, the total number of data blocks
pointed to by every leaf-level index block is kept track of. When we finish
writing and the total number of leaf-level blocks is determined, it is clear
which leaf-level block contains the mid-key, and the fields listed above are
computed. Â When reading the HFile and the mid-key is requested, we retrieve
the middle leaf index block (potentially from the block cache) and get the
mid-key value from the appropriate position inside that leaf
block.</para></section>
-<section><title>
- Non-root block index format in version 2</title>
- <para>This format applies to intermediate-level and leaf index blocks of a
version 2 multi-level data block index. Every non-root index block is
structured as follows. </para>
- <orderedlist>
- <listitem>
- <para>numEntries: the number of entries (int). </para>
- </listitem>
- <listitem>
- <para>entryOffsets: the âsecondary indexâ of offsets of entries
in the block, to facilitate a quick binary search on the key (numEntries + 1
int values). The last value is the total length of all entries in this index
block. For example, in a non-root index block with entry sizes 60, 80, 50 the
âsecondary indexâ will contain the following int array: {0, 60, 140,
190}.</para>
- </listitem>
- <listitem>
- <para>Entries. Each entry contains: </para>
- <orderedlist>
- <listitem>
- <para>
-Offset of the block referenced by this entry in the file (long)
- </para>
- </listitem>
- <listitem>
- <para>
-On>disk size of the referenced block (int)
- </para>
- </listitem>
- <listitem>
- <para>
-Key. The length can be calculated from entryOffsets.
- </para>
- </listitem>
- </orderedlist>
-
- </listitem>
- </orderedlist></section><section><title>
- Bloom filters in version 2</title>
- <para>In contrast with version 1, in a version 2 HFile Bloom filter
metadata is stored in the load-on-open section of the HFile for quick startup.
</para>
- <orderedlist>
- <listitem>
- <para>A compound Bloom filter. </para>
- <orderedlist>
- <listitem>
- <para>
- Bloom filter version = 3 (int). There used to be a DynamicByteBloomFilter
class that had the Bloom filter version number 2
- </para>
- </listitem>
- <listitem>
- <para>
-The total byte size of all compound Bloom filter chunks (long)
- </para>
- </listitem>
- <listitem>
- <para>
- Number of hash functions (int
- </para>
- </listitem>
- <listitem>
- <para>
-Type of hash functions (int)
- </para>
- </listitem>
- <listitem>
- <para>
-The total key count inserted into the Bloom filter (long)
- </para>
- </listitem>
- <listitem>
- <para>
-The maximum total number of keys in the Bloom filter (long)
- </para>
- </listitem>
- <listitem>
- <para>
-The number of chunks (int)
- </para>
- </listitem>
- <listitem>
- <para>
-Comparator class used for Bloom filter keys, a UTF>8 encoded string stored
using Bytes.writeByteArray
- </para>
- </listitem>
- <listitem>
- <para>
- Bloom block index in the version 2 root block index format
- </para>
- </listitem>
- </orderedlist>
- </listitem>
- </orderedlist></section><section><title>File Info format in versions 1 and
2</title>
- <para>The file info block is a serialized <link
xlink:href="http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/io/HbaseMapWritable.html";>HbaseMapWritable</link>
(essentially a map from byte arrays to byte arrays) with the following keys,
among others. StoreFile-level logic adds more keys to this.</para>
- <informaltable frame="all">
- <tgroup cols="2"><tbody><row>
- <entry>
- <para>hfile.LASTKEY </para>
- </entry>
- <entry>
- <para>The last key of the file (byte array) </para>
- </entry>
- </row>
- <row>
- <entry>
- <para>hfile.AVG_KEY_LEN </para>
- </entry>
- <entry>
- <para>The average key length in the file (int) </para>
- </entry>
- </row>
- <row>
- <entry>
- <para>hfile.AVG_VALUE_LEN </para>
- </entry>
- <entry>
- <para>The average value length in the file (int) </para>
- </entry>
- </row></tbody></tgroup>
- </informaltable>
- <para>File info format did not change in version 2. However, we moved the
file info to the final section of the file, which can be loaded as one block at
the time the HFile is being opened. Also, we do not store comparator in the
version 2 file info anymore. Instead, we store it in the fixed file trailer.
This is because we need to know the comparator at the time of parsing the
load-on-open section of the HFile.</para></section><section><title>
- Fixed file trailer format differences between versions 1 and 2</title>
- <para>The following table shows common and different fields between fixed
file trailers in versions 1 and 2. Note that the size of the trailer is
different depending on the version, so it is âfixedâ only within one
version. However, the version is always stored as the last four-byte integer in
the file. </para>
- <para/>
- <informaltable frame="all">
- <tgroup cols="2">
-<colspec colname='c1'/>
-<colspec colname='c2'/>
-<tbody>
- <row>
- <entry>
- <para>Version 1 </para>
- </entry>
- <entry>
- <para>Version 2 </para>
- </entry>
- </row>
- <row>
- <entry align="center" namest="c1" nameend="c2">
- <para>File info offset (long) </para>
- </entry>
- </row>
- <row>
- <entry>
- <para>Data index offset (long) </para>
- </entry>
- <entry>
- <para>loadOnOpenOffset (long)</para>
- <para><emphasis>The offset of the section that we need toload
when opening the file.</emphasis></para>
- </entry>
- </row>
- <row>
- <entry align="center" namest="c1" nameend="c2">
- <para>Number of data index entries (int) </para>
- </entry>
- </row>
- <row>
- <entry>
- <para>metaIndexOffset (long)</para>
- <para>This field is not being used by the version 1 reader, so
we removed it from version 2.</para>
- </entry>
- <entry>
- <para>uncompressedDataIndexSize (long)</para>
- <para>The total uncompressed size of the whole data block
index, including root-level, intermediate-level, and leaf-level blocks.</para>
- </entry>
- </row>
- <row>
- <entry namest="c1" nameend="c2" align="center">
- <para>Number of meta index entries (int) </para>
- </entry>
- </row>
- <row>
- <entry namest="c1" nameend="c2" align="center">
- <para>Total uncompressed bytes (long) </para>
- </entry>
- </row>
- <row>
- <entry>
- <para>numEntries (int) </para>
- </entry>
- <entry>
- <para>numEntries (long) </para>
- </entry>
- </row>
- <row>
- <entry namest="c1" nameend="c2" align="center">
- <para>Compression codec: 0 = LZO, 1 = GZ, 2 = NONE (int) </para>
- </entry>
- </row>
- <row>
- <entry>
- <para></para>
- </entry>
- <entry>
- <para>The number of levels in the data block index (int) </para>
- </entry>
- </row>
- <row>
- <entry>
- <para></para>
- </entry>
- <entry>
- <para>firstDataBlockOffset (long)</para>
- <para>The offset of the first first data block. Used when
scanning. </para>
- </entry>
- </row>
- <row>
- <entry>
- <para></para>
- </entry>
- <entry>
- <para>lastDataBlockEnd (long)</para>
- <para>The offset of the first byte after the last key/value
data block. We don't need to go beyond this offset when scanning. </para>
- </entry>
- </row>
- <row>
- <entry>
- <para>Version: 1 (int) </para>
- </entry>
- <entry>
- <para>Version: 2 (int) </para>
- </entry>
- </row></tbody></tgroup>
- </informaltable>
- <para/></section>
- <section><title>getShortMidpointKey(an optimization for data index
block)</title>
- <para>Note: this optimization was introduced in HBase 0.95+</para>
- <para>HFiles contain many blocks that contain a range of sorted Cells.
Each cell has a key. To save IO when reading Cells, the HFile also has an index
that maps a Cell's start key to the offset of the beginning of a particular
block. Prior to this optimization, HBase would use the key of the first cell in
each data block as the index key.</para>
- <para>In HBASE-7845, we generate a new key that is lexicographically
larger than the last key of the previous block and lexicographically equal or
smaller than the start key of the current block. While actual keys can
potentially be very long, this "fake key" or "virtual key" can be much shorter.
For example, if the stop key of previous block is "the quick brown fox", the
start key of current block is "the who", we could use "the r" as our virtual
key in our hfile index.</para>
- <para>There are two benefits to this:</para>
- <itemizedlist>
- <listitem><para>having shorter keys reduces the hfile index size,
(allowing us to keep more indexes in memory), and</para></listitem>
- <listitem><para>using something closer to the end key of the previous
block allows us to avoid a potential extra IO when the target key lives in
between the "virtual key" and the key of the first element in the target
block.</para></listitem>
- </itemizedlist>
- <para>This optimization (implemented by the getShortMidpointKey method)
is inspired by LevelDB's ByteWiseComparatorImpl::FindShortestSeparator() and
FindShortSuccessor().</para>
- </section>
- </section>
- <section xml:id="hfilev3">
- <title>HBase File Format with Security Enhancements (version 3)</title>
- <para>Note: this feature was introduced in HBase 0.98</para>
- <section xml:id="hfilev3.motivation">
- <title>Motivation </title>
- <para>
- Version 3 of HFile makes changes needed to ease management of
encryption at rest and
- cell-level metadata (which in turn is needed for cell-level ACLs and
cell-level visibility
- labels). For more information see <xref
linkend="hbase.encryption.server"/>,
- <xref linkend="hbase.tags"/>, <xref
linkend="hbase.accesscontrol.configuration"/>, and
- <xref linkend="hbase.visibility.labels"/>.
- </para>
- </section>
- <section xml:id="hfilev3.overview">
- <title>Overview</title>
- <para>
- The version of HBase introducing the above features reads HFiles in
versions 1, 2, and 3 but
- only writes version 3 HFiles. Version 3 HFiles are structured the same
as version 2 HFiles.
- For more information see <xref linkend="hfilev2.overview"/>.
- </para>
- </section>
- <section xml:id="hvilev3.infoblock">
- <title>File Info Block in Version 3</title>
- <para>
- Version 3 added two additional pieces of information to the reserved
keys in the file info
- block.
- <informaltable frame="all">
- <tgroup cols="2">
- <tbody>
- <row>
- <entry>
- <para>hfile.MAX_TAGS_LEN</para>
- </entry>
- <entry>
- <para>
- The maximum number of bytes needed to store the
serialized tags for any single
- cell in this hfile (int)
- </para>
- </entry>
- </row>
- <row>
- <entry>
- <para>hfile.TAGS_COMPRESSED</para>
- </entry>
- <entry>
- <para>Does the block encoder for this hfile compress tags?
(boolean)</para>
- <para>
- Should only be present if
<classname>hfile.MAX_TAGS_LEN</classname> is also
- present.
- </para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </para>
- <para>
- When reading a Version 3 HFile the presence of
<classname>MAX_TAGS_LEN</classname> is used
- to determine how to deserialize the cells within a data block.
Therefore, consumers must
- read the file's info block prior to reading any data blocks.
- </para>
- <para>
- When writing a Version 3 HFile, HBase will always include
<classname>MAX_TAGS_LEN
- </classname> when flushing the memstore to underlying filesystem and
when using prefix tree
- encoding for data blocks, as described in <xref
linkend="compression"/>. When compacting
- extant files, the default writer will omit
<classname>MAX_TAGS_LEN</classname> if all of the
- files selected do not themselves contain any cells with tags. See
- <xref linkend="compaction"/> for details on the compaction file
selection algorithm.
- </para>
- </section>
- <section xml:id="hfilev3.datablock">
- <title>Data Blocks in Version 3</title>
- <para>
- Within an HFile, HBase cells are stored in data blocks as a sequence
of KeyValues (see <xref
- linkend="hfilev1.overview"/>, or <link xlink:href=
-
"http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html";>Lars
George's
- excellent introduction to HBase Storage</link>). In version 3, these
KeyValue optionally
- will include a set of 0 or more tags:
- <informaltable frame="all">
- <tgroup cols="2">
- <colspec colname='c1'/>
- <colspec colname='c2'/>
- <tbody>
- <row>
- <entry>
- <para>Version 1 & 2</para>
- <para>Version 3 without MAX_TAGS_LEN</para>
- </entry>
- <entry><para>Version 3 with MAX_TAGS_LEN</para></entry>
- </row>
- <row>
- <entry align="center" namest="c1" nameend="c2">
- <para>Key Length (4 bytes)</para>
- </entry>
- </row>
- <row>
- <entry align="center" namest="c1" nameend="c2">
- <para>Value Length (4 bytes)</para>
- </entry>
- </row>
- <row>
- <entry align="center" namest="c1" nameend="c2">
- <para>Key bytes (variable)</para>
- </entry>
- </row>
- <row>
- <entry align="center" namest="c1" nameend="c2">
- <para>Value bytes (variable)</para>
- </entry>
- </row>
- <row>
- <entry align="center" namest="c2" nameend="c2">
- <para>Tags Length (2 bytes)</para>
- </entry>
- </row>
- <row>
- <entry align="center" namest="c2" nameend="c2">
- <para>Tags bytes (variable)</para>
- </entry>
- </row>
- </tbody>
- </tgroup>
- </informaltable>
- </para>
- <para>
- If the info block for a given HFile contains an entry for
- <classname>MAX_TAGS_LEN</classname> each cell will have the length of
that cell's tags
- included, even if that length is zero. The actual tags are stored as a
sequence of tag
- length (2 bytes), tag type (1 byte), tag bytes (variable). The format
an individual tag's
- bytes depends on the tag type.
- </para>
- <para>
- Note that the dependence on the contents of the info block implies
that prior to reading
- any data blocks you must first process a file's info block. It also
implies that prior to
- writing a data block you must know if the file's info block will
include
- <classname>MAX_TAGS_LEN</classname>.
- </para>
- </section>
- <section xml:id="hfilev3.fixedtrailer">
- <title>Fixed File Trailer in Version 3</title>
- <para>
- The fixed file trailers written with HFile version 3 are always
serialized with protocol
- buffers. Additionally, it adds an optional field to the version 2
protocol buffer named
- encryption_key. If HBase is configured to encrypt HFiles this field
will store a data
- encryption key for this particular HFile, encrypted with the current
cluster master key
- using AES. For more information see <xref
linkend="hbase.encryption.server"/>.
- </para>
- </section>
- </section>
-</appendix>
