[
https://issues.apache.org/jira/browse/HBASE-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15174292#comment-15174292
]
deepankar commented on HBASE-15362:
-----------------------------------
I was trying to set {{io.compression.codec.lzo.buffersize}}, the reason for
setting this is to decrease the buffer size allocated in
BlockDecompressorStream, it defaults to 64 kb and if you are missing cache
frequently then this was causing some significant amount of garbage.
Since you brought this up does it make sense that we should recognize that some
(almost all in the current case) of the codecs does not need to create the
above stream and directly send the calls to codec or the Decompressor class
exposed by the codec, it will avoid this 64 kb allocations across the board and
the decompressors are recycled so the buffers inside them are not a problem.
Emulating the code inside the BlockDecompressorStream, the only problem is it
cannot work with byte arrays, we special cased this internal and the code
turned out be very small, pasting this part here
{code}
// Code similar to the code in BlockDecompressionStream#decompress
Decompressor decompressor = null;
try {
decompressor = compressAlgo.getDecompressor();
int curInputOffset = inputOffset, curOutputOffset = destOffset;
int originalBlockSize = Bytes.toInt(inputArray, curInputOffset, 4);
int noUncompressedBytes = 0;
curInputOffset += 4;
// Iterate until you finish the input
while (curInputOffset < (inputOffset + compressedSize)
&& noUncompressedBytes < uncompressedSize) {
if (decompressor.needsInput()) {
int blockSize = Bytes.toInt(inputArray, curInputOffset, 4);
curInputOffset += 4;
decompressor.setInput(inputArray, curInputOffset, blockSize);
curInputOffset += blockSize;
}
int n = decompressor.decompress(dest, curOutputOffset,
uncompressedSize - noUncompressedBytes);
noUncompressedBytes += n;
curOutputOffset += n;
if (n == 0) {
if (decompressor.finished() || decompressor.needsDictionary()) {
if (noUncompressedBytes >= originalBlockSize) {
throw new IOException("uncompressed bytes >= orginal size");
}
}
}
}
if (noUncompressedBytes != uncompressedSize) {
throw new IOException("Premature EOF from the input bytes");
}
} finally {
if (decompressor != null) {
compressAlgo.returnDecompressor(decompressor);
}
}
{code}
> Compression Algorithm does not respect config params from hbase-site
> --------------------------------------------------------------------
>
> Key: HBASE-15362
> URL: https://issues.apache.org/jira/browse/HBASE-15362
> Project: HBase
> Issue Type: Bug
> Reporter: deepankar
> Assignee: deepankar
> Priority: Trivial
> Attachments: HBASE-15362.patch
>
>
> Compression creates conf using new Configuration() and this leads to it not
> respecting the confs set in hbase-site, fixing it is trivial using
> HBaseConfiguration.create()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
