[
https://issues.apache.org/jira/browse/HADOOP-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063609#comment-14063609
]
Todd Lipcon commented on HADOOP-10778:
--------------------------------------
BTW, what clock rate is your laptop? The best non-hardware CRC32 implementation
I'm aware of (http://code.google.com/p/crcutil/) goes 1 cycle per byte. So, my
guess is that your 2.3GB/sec is probably from a 2.3 (or so) Ghz machine using
an algorithm like crcutil's. Perhaps Apple patched their zlib to use this
algorithm instead of the very simple implementation in the upstream zlib?
> Use NativeCrc32 only if it is faster
> ------------------------------------
>
> Key: HADOOP-10778
> URL: https://issues.apache.org/jira/browse/HADOOP-10778
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Tsz Wo Nicholas Sze
> Attachments: c10778_20140702.patch
>
>
> From the benchmark post in [this
> comment|https://issues.apache.org/jira/browse/HDFS-6560?focusedCommentId=14044060&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14044060],
> NativeCrc32 is slower than java.util.zip.CRC32 for Java 7 and above when
> bytesPerChecksum > 512.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
