[
https://issues.apache.org/jira/browse/HBASE-27049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538566#comment-17538566
]
Andrew Kyle Purtell edited comment on HBASE-27049 at 5/18/22 4:14 AM:
----------------------------------------------------------------------
We have to care about file compatibility and so if making the change here any
changes to the binary format of the file must be analyzed and addressed.
Unfortunately it will not enough just to do a configuration switch if there is
a change to the binary format of the compression because our users may do
rolling upgrades, or rollback, or offline hfile writes and bulk loads. There
are many scenarios where there will be mixed version readers and writers.
So making the change you propose is fine, but there must also be a plan for
evolution of the hfile format (and WAL format too, for the value compression
option) if the binary format of the compression will change.
It might also be possible to read and write the same effective format with the
new implementation by maintaining the details. Read the impl code of the Hadoop
block compression streams and you will see what I mean. If so obviously there
will be no file format issues.
was (Author: apurtell):
We have to care about file compatibility and so if making the change here any
changes to the binary format of the file must be analyzed and addressed.
Unfortunately it will not enough just to do a configuration switch if there is
a change to the binary format of the compression because our users may do
rolling upgrades, or rollback, or offline hfile writes and bulk loads. There
are many scenarios where there will be mixed version readers and writers.
So making the change you propose is fine, but there must also be a plan for
evolution of the hfile format (and WAL format too, for the value compression
option) if the binary format of the compression will change. Read the impl code
of the Hadoop block compression streams and you will see what I mean.
> Decrease memory copy when decompress data
> -----------------------------------------
>
> Key: HBASE-27049
> URL: https://issues.apache.org/jira/browse/HBASE-27049
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: chenfengge
> Priority: Minor
>
> HBase RegionServer use createDecompressionStream in class
> org.apache.hadoop.hbase.io.compress.Compression, which cause extra memory
> copy during decompression.We can offer interface for block decompression,
> like "void decompress(ByteBuff src, ByteBuff dst);", and offer default
> implementation for all algorithms.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
