[
https://issues.apache.org/jira/browse/HDFS-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Tianyi updated HDFS-10331:
-----------------------------
Description:
In java8, performance of intrinsic CRC32 has been dramatically improved.
See: https://bugs.openjdk.java.net/browse/JDK-7088419
I carried an in-memory benchmark of throughput, on a server with two E5-2630 v2
cpus, results:
(single threaded)
java7 java.util.zip.CRC32: 0.81GB/s
hdfs DataChecksum, native: 1.46GB/s
java8 java.util.zip.CRC32: 2.39GB/s
hdfs DataChecksum, CRC32 on java8: 2.39GB/s
IMHO I think we could either:
A) provide a configuration for user to switch CRC32 implementations;
or B) On java8 or above, always use intrinsic CRC32.
was:
In java8, performance of intrinsic CRC32 has been dramatically improved.
See: https://bugs.openjdk.java.net/browse/JDK-7088419
I carried an in-memory benchmark of throughput, on a server with two E5-2630 v2
cpus, results:
java7 java.util.zip.CRC32: 0.81GB/s
hdfs DataChecksum, native: 1.46GB/s
java8 java.util.zip.CRC32: 2.39GB/s
hdfs DataChecksum, CRC32 on java8: 2.39GB/s
IMHO I think we could either:
A) provide a configuration for user to switch CRC32 implementations;
or B) On java8 or above, always use intrinsic CRC32.
> Use java.util.zip.CRC32 for java8 or above in libhadoop
> -------------------------------------------------------
>
> Key: HDFS-10331
> URL: https://issues.apache.org/jira/browse/HDFS-10331
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, hdfs, hdfs-client
> Affects Versions: 2.6.0
> Reporter: He Tianyi
>
> In java8, performance of intrinsic CRC32 has been dramatically improved.
> See: https://bugs.openjdk.java.net/browse/JDK-7088419
> I carried an in-memory benchmark of throughput, on a server with two E5-2630
> v2 cpus, results:
> (single threaded)
> java7 java.util.zip.CRC32: 0.81GB/s
> hdfs DataChecksum, native: 1.46GB/s
> java8 java.util.zip.CRC32: 2.39GB/s
> hdfs DataChecksum, CRC32 on java8: 2.39GB/s
> IMHO I think we could either:
> A) provide a configuration for user to switch CRC32 implementations;
> or B) On java8 or above, always use intrinsic CRC32.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
