[
https://issues.apache.org/jira/browse/HADOOP-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630441#action_12630441
]
Hong Tang commented on HADOOP-4162:
-----------------------------------
Thanks for the information. The problem was found in the development of TFIle
(Hadoop-3315). The way we use Hadoop Compression in TFile is to take each
compression block as a separate compression stream (each block writes conclude
with compressor.finish()). It makes no assumption of any internals of
compression algorithm. The tests show both LZOP and LZO work fine.
Also, based on the information you provided, it seems that existence of
LzopDecompressor is to read lzop compressed data. So I changed to use LZO
instead of LZOP internally for TFile now.
> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
> -----------------------------------------------------------------------------
>
> Key: HADOOP-4162
> URL: https://issues.apache.org/jira/browse/HADOOP-4162
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.18.0
> Reporter: Hong Tang
> Assignee: Arun C Murthy
> Fix For: 0.19.0
>
> Attachments: HADOOP-4162_0_20080911.patch
>
>
> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
> I investigated the code, the reason seems to be the following:
> LzopCodec inherits from LzoCodec. The getDecompressorType() method is
> supposed to return the concrete Decompressor class type the specific Codec
> class creates. In this case, LzopCodec creates LzopDecompressors and should
> return LzopDecompressor.class. But instead, it uses the getDecompressorType()
> method defined in the parent and returns LzoDecompressor.class.
> This leads to CodecPool unable to properly recycle the decompressors.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
