[
https://issues.apache.org/jira/browse/HADOOP-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897084#action_12897084
]
Alex Rovner commented on HADOOP-4162:
-------------------------------------
We are using 0.20 and LZO can be used for map compression. In which case the
same issue appears:
[2010-08-10 18:19:18,459] INFO (JvmMetrics.java:71) - Initializing JVM Metrics
with processName=SHUFFLE, sessionId=
[2010-08-10 18:19:18,729] INFO (GPLNativeCodeLoader.java:34) - Loaded native
gpl library
[2010-08-10 18:19:18,739] INFO (LzoCodec.java:65) - Successfully loaded &
initialized native-lzo library [hadoop-lzo rev fatal: Not a git repository]
[2010-08-10 18:19:18,793] INFO (ReduceTask.java:1014) - ShuffleRamManager:
MemoryLimit=358881696, MaxSingleShuffleLimit=89720424
[2010-08-10 18:19:18,836] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,839] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,841] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,842] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,844] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,845] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,847] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,849] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,851] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,852] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,854] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,855] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,857] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,859] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,863] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,865] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,867] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,881] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,885] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,886] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,888] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,889] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,891] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,892] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,894] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,895] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,897] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,899] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,901] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,902] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,904] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,905] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,907] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,908] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,910] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,912] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,914] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,915] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,917] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,918] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,920] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,921] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,923] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,924] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,926] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,928] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,930] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,931] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,933] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,934] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,936] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,938] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,939] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,941] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,942] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,943] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,945] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,946] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,947] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,948] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,949] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,950] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,951] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,953] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,954] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,955] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,956] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,957] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,958] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,960] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,961] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,962] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,963] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,964] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,964] INFO (CodecPool.java:121) - Got brand-new
decompressor
[2010-08-10 18:19:18,970] INFO (ReduceTask.java:2468) -
attempt_201008101538_0280_r_000000_0 Thread started: Thread for merging on-disk
files
[2010-08-10 18:19:18,971] INFO (ReduceTask.java:2473) -
attempt_201008101538_0280_r_000000_0 Thread waiting: Thread for merging on-disk
files
> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
> -----------------------------------------------------------------------------
>
> Key: HADOOP-4162
> URL: https://issues.apache.org/jira/browse/HADOOP-4162
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 0.18.0
> Reporter: Hong Tang
> Assignee: Arun C Murthy
> Attachments: HADOOP-4162_0_20080911.patch
>
>
> CodecPool.getDecompressor(LzopCodec) always creates a brand-new decompressor.
> I investigated the code, the reason seems to be the following:
> LzopCodec inherits from LzoCodec. The getDecompressorType() method is
> supposed to return the concrete Decompressor class type the specific Codec
> class creates. In this case, LzopCodec creates LzopDecompressors and should
> return LzopDecompressor.class. But instead, it uses the getDecompressorType()
> method defined in the parent and returns LzoDecompressor.class.
> This leads to CodecPool unable to properly recycle the decompressors.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
