Charles Connell created HBASE-29135:
---------------------------------------
Summary: ZStandard decompression can operate directly on ByteBuffs
Key: HBASE-29135
URL: https://issues.apache.org/jira/browse/HBASE-29135
Project: HBase
Issue Type: Improvement
Reporter: Charles Connell
Assignee: Charles Connell
Attachments: create-decompression-stream-zstd.html
I've been thinking about ways to improve HBase's performance when reading
HFiles, and I believe there is significant opportunity. I look at many
RegionServer profile flamegraphs of my company's servers. A pattern that I've
discovered is that object allocation in a very hot code path is a performance
killer. The HFile decoding code makes some effort to avoid this, but it isn't
totally successful.
Each time a block is decoded in HFileBlockDefaultDecodingContext, a new
DecompressorStream is allocated and used. This is a lot of allocation, and the
use of the streaming pattern requires copying every byte to be decompressed
more times than necessary. Each byte is copied from a ByteBuff into a byte[],
then decompressed, then copied back to a ByteBuff. For decompressors like
org.apache.hadoop.hbase.io.compress.zstd.ZstdDecompressor that only operate on
direct memory, two additional copies are introduced to move from a byte[] to a
direct NIO ByteBuffer, then back to a byte[].
Aside from the copies inherent in the decompression algorithm, the necessity of
copying from an compressed buffer to an uncompressed buffer, all of these other
copies can be avoided without sacrificing functionality. Along the way, we'll
also avoid allocating objects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
