[
https://issues.apache.org/jira/browse/HADOOP-11334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225556#comment-14225556
]
Eric Yang commented on HADOOP-11334:
------------------------------------
-Dio.native.lib.available=false and
-Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.GzipCodec
are conditions that opposed each other. There is no pure Java only
implementation of GzipCodec compressor. There is a pure Java only
decompressor. To solve this issue, we have two options:
1. Either implement a compressor codec for GzipCodec that does not depend on
zlib
2. The jobclient needs to validate the compressor and compression related flags
to safe guard contradictions and spaw off reducer with properly validated
decompressor.
Either options are possible.
> Mapreduce Job Failed due to failure fetching mapper output on the reduce side
> -----------------------------------------------------------------------------
>
> Key: HADOOP-11334
> URL: https://issues.apache.org/jira/browse/HADOOP-11334
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
> Affects Versions: 2.4.1
> Reporter: Jinghui Wang
>
> Running terasort with the following options hadoop jar
> hadoop-mapreduce-examples.jar terasort *-Dio.native.lib.available=false
> -Dmapreduce.map.output.compress=true
> -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.GzipCodec*
> /tmp/tera-in /tmp/tera-out
> The job failed with the reducer failed to fetching the output from mappers
> (see the following stacktrace). The problem is that in JIRA MAPREDUCE-1784,
> it added support to handle null compressors to default to non-compressed
> output. In this case, when the *io.native.lib.available* is set to true, the
> compressor will be null. However, the decompressor has a Java implementation,
> so when the reducer tries to read the mapper output, it uses the
> decompressor, but the output does not have the Gzip header.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
