[
https://issues.apache.org/jira/browse/MAPREDUCE-2258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980882#action_12980882
]
Hong Tang commented on MAPREDUCE-2258:
--------------------------------------
Such pattern would in general affect all CompressionCodec's and is similar to a
bug I filed earlier: HADOOP-4195.
On the other hand, as explained by Chris D in HADOOP-4162, LzopCodec cannot be
safely reused in Hadoop, and thus the problem you described actually should not
happen. In fact, repeatedly getting LzopCodec from CodecPool is likely get you
into OOM.
> IFile reader closes stream and compressor in wrong order
> --------------------------------------------------------
>
> Key: MAPREDUCE-2258
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2258
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task
> Affects Versions: 0.20.4, 0.22.0
> Reporter: Todd Lipcon
> Fix For: 0.22.0
>
>
> In IFile.Reader.close(), we return the decompressor to the pool and then call
> close() on the input stream. This is backwards and causes a rare race in the
> case of LzopCodec, since LzopInputStream makes a few calls on the
> decompressor object inside close(). If another thread pulls the decompressor
> out of the pool and starts to use it in the meantime, the first thread's
> close() will cause the second thread to potentially miss pieces of data.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
