[
https://issues.apache.org/jira/browse/MAPREDUCE-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405234#comment-13405234
]
Ankit Kamboj commented on MAPREDUCE-4354:
-----------------------------------------
This makes sense. The reinit for LZO should only reset buffers instead of
recreating them.
So, I have removed the 'if' condition for GZIP in CodecPool and instead changed
reinit logic in LZOCompressor.
Please find attached the corresponding patch.
> Performance improvement with compressor object reinit restriction
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-4354
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4354
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance
> Affects Versions: 0.20.205.0
> Reporter: Ankit Kamboj
> Priority: Minor
> Labels: performance
> Fix For: 0.20.205.0
>
> Attachments: codec_reinit_diff
>
>
> HADOOP-5879 patch aimed at picking the conf (instead of default) settings for
> GzipCodec. It also involved re-initializing the recycled compressor object.
> On our performance tests, this re-initialization led to performance
> degradation of 15% for LzoCodec because re-initialization for Lzo involves
> reallocation of buffers. LzoCodec takes the initial settings from config so
> it is not necessary to re-initialize it. This patch checks for the codec
> class and calls reinit only if the codec class is Gzip. This led to
> significant performance improvement of 15% for LzoCodec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
