Have you considered deflate or bzip?
- Tim.
________________________________________
From: Marek Miglinski [mmiglin...@seven.com]
Sent: Thursday, June 14, 2012 1:39 AM
To: mapreduce-user@hadoop.apache.org
Subject: codec compression ratio
When procession 65billion records and using LZO or Snappy codecs, disk IO is at
100% because mappers are spilling all the time, but CPU is at 40%. Is there a
setting where I can raise compression ratio for map/reduce internal temp data
(for LZO or Snappy)? So that I can raise effort on CPU and lower IO? Google
didn't gave any ideas...
Thanks.
Marek M.
The information contained in this email is intended only for the personal and
confidential use of the recipient(s) named above. The information and any
attached documents contained in this message may be Exar confidential and/or
legally privileged. If you are not the intended recipient, you are hereby
notified that any review, use, dissemination or reproduction of this message is
strictly prohibited and may be unlawful. If you have received this
communication in error, please notify us immediately by return email and delete
the original message.
