Temporary map-ouput files don't use Avro format. There is a custom format which
should be compressed if you set mapred.compress.map.output.
Arun
On Jan 12, 2012, at 8:08 AM, Frank Grimes wrote:
> I tried conf.setBoolean("mapred.compress.map.output", true); but it didn't
> seem to work.
>
> Also, since I'm using the Avro mapred APIs, maybe there's something Avro
> specific to get it enabled?
> Should I ask on the Avro mailing lists?
>
> Thanks,
>
> Frank Grimes
>
>
> On 2012-01-12, at 10:49 AM, bejoy.had...@gmail.com wrote:
>
>> Hi Frank
>> Is map output compression enabled?
>>
>> The config param would be like
>> mapred.map.output.compress=true
>> (It is from my memory, Please cross check)
>>
>> ------Original Message------
>> From: Frank Grimes
>> To: hdfs-user@hadoop.apache.org
>> ReplyTo: hdfs-user@hadoop.apache.org
>> Subject: Can spill to disk be in compressed format to reduce I/O?
>> Sent: Jan 12, 2012 21:10
>>
>> Hi All,
>>
>> We're trying to speed up an M/R job which combines multiple .avro files.
>> We've noticed that when it spills to disk, it's in uncompressed format.
>> Is there a way to make it spill temporary segments as .avro with Deflate
>> compression?
>>
>> Thanks,
>>
>> Frank Grimes
>>
>> Regards
>> Bejoy K S
>
