[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066431#comment-14066431
 ] 

Todd Lipcon commented on MAPREDUCE-5962:
----------------------------------------

Realized for this to be effective we also need to implement the Checksum 
interface with the native code. Currently the native code only supports the 
"chunked sums" verification used by HDFS, and doesn't implement the java 
Checksum.update interface that IFile uses. Will hold off on this patch for the 
time being.

> Support CRC32C in IFile
> -----------------------
>
>                 Key: MAPREDUCE-5962
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5962
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: performance, task
>    Affects Versions: 2.5.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: mapreduce-5962.txt, mapreduce-5962.txt
>
>
> Currently, the IFile format used by the MR shuffle checksums all data using 
> the zlib CRC32 polynomial. If we allow use of CRC32C instead, we can get a 
> large reduction in CPU usage by leveraging the native hardware CRC32C 
> implementation (approx half a second of CPU time savings per GB checksummed).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to