[
https://issues.apache.org/jira/browse/HADOOP-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063577#comment-14063577
]
Todd Lipcon commented on HADOOP-10778:
--------------------------------------
Sure, here's the whole table (Native is always faster):
Direct Buffer Performance Table (bpc = byte-per-crc in MB/sec; #T = #Theads)
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 32 | 1 | 422.8 | 802.0 | 89.7% | 1908.2 | 351.3% | 137.9% |
| 32 | 2 | 316.7 | 622.1 | 96.4% | 1853.9 | 485.3% | 198.0% |
| 32 | 4 | 285.8 | 516.9 | 80.9% | 1806.3 | 532.1% | 249.5% |
| 32 | 8 | 171.5 | 304.2 | 77.3% | 1052.5 | 513.5% | 246.0% |
| 32 | 16 | 95.8 | 189.7 | 97.9% | 640.3 | 568.1% | 237.6% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 64 | 1 | 604.4 | 1034.8 | 71.2% | 1795.6 | 197.1% | 73.5% |
| 64 | 2 | 506.4 | 865.6 | 70.9% | 1752.2 | 246.0% | 102.4% |
| 64 | 4 | 530.5 | 923.6 | 74.1% | 1642.4 | 209.6% | 77.8% |
| 64 | 8 | 300.5 | 468.2 | 55.8% | 1222.6 | 306.8% | 161.1% |
| 64 | 16 | 163.2 | 284.6 | 74.4% | 669.8 | 310.4% | 135.3% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 128 | 1 | 782.8 | 1184.3 | 51.3% | 1694.8 | 116.5% | 43.1% |
| 128 | 2 | 759.5 | 1147.2 | 51.1% | 1690.3 | 122.6% | 47.3% |
| 128 | 4 | 737.2 | 1110.1 | 50.6% | 1634.1 | 121.7% | 47.2% |
| 128 | 8 | 488.2 | 602.1 | 23.3% | 1317.1 | 169.8% | 118.8% |
| 128 | 16 | 281.2 | 375.9 | 33.7% | 794.8 | 182.6% | 111.4% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 256 | 1 | 921.1 | 1242.1 | 34.9% | 1711.7 | 85.8% | 37.8% |
| 256 | 2 | 890.4 | 1231.2 | 38.3% | 1618.5 | 81.8% | 31.5% |
| 256 | 4 | 868.9 | 1162.0 | 33.7% | 1597.3 | 83.8% | 37.5% |
| 256 | 8 | 651.1 | 673.0 | 3.4% | 1321.4 | 102.9% | 96.3% |
| 256 | 16 | 369.6 | 397.5 | 7.5% | 730.2 | 97.6% | 83.7% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 512 | 1 | 993.0 | 1292.9 | 30.2% | 1653.6 | 66.5% | 27.9% |
| 512 | 2 | 926.7 | 1274.4 | 37.5% | 1635.4 | 76.5% | 28.3% |
| 512 | 4 | 918.0 | 1217.7 | 32.6% | 1589.3 | 73.1% | 30.5% |
| 512 | 8 | 778.0 | 711.6 | -8.5% | 1365.8 | 75.5% | 91.9% |
| 512 | 16 | 457.2 | 406.2 | -11.2% | 790.6 | 72.9% | 94.6% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 1024 | 1 | 1016.0 | 1303.2 | 28.3% | 1673.8 | 64.7% | 28.4% |
| 1024 | 2 | 987.0 | 1264.9 | 28.2% | 1637.3 | 65.9% | 29.4% |
| 1024 | 4 | 956.7 | 1239.4 | 29.5% | 1593.3 | 66.5% | 28.6% |
| 1024 | 8 | 837.3 | 716.4 | -14.4% | 1336.8 | 59.7% | 86.6% |
| 1024 | 16 | 522.1 | 389.0 | -25.5% | 948.5 | 81.7% | 143.8% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 2048 | 1 | 1006.9 | 1271.3 | 26.3% | 1695.3 | 68.4% | 33.4% |
| 2048 | 2 | 989.1 | 1234.1 | 24.8% | 1652.3 | 67.1% | 33.9% |
| 2048 | 4 | 959.1 | 1200.1 | 25.1% | 1581.7 | 64.9% | 31.8% |
| 2048 | 8 | 875.4 | 713.4 | -18.5% | 1422.8 | 62.5% | 99.4% |
| 2048 | 16 | 507.5 | 402.2 | -20.7% | 923.7 | 82.0% | 129.6% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 4096 | 1 | 1025.0 | 1251.6 | 22.1% | 1691.7 | 65.0% | 35.2% |
| 4096 | 2 | 990.7 | 1219.8 | 23.1% | 1630.2 | 64.6% | 33.6% |
| 4096 | 4 | 961.5 | 1178.9 | 22.6% | 1582.7 | 64.6% | 34.3% |
| 4096 | 8 | 891.1 | 714.0 | -19.9% | 1429.4 | 60.4% | 100.2% |
| 4096 | 16 | 529.1 | 405.3 | -23.4% | 862.6 | 63.0% | 112.8% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 8192 | 1 | 1015.4 | 1290.6 | 27.1% | 1685.9 | 66.0% | 30.6% |
| 8192 | 2 | 1007.6 | 1266.1 | 25.7% | 1619.9 | 60.8% | 27.9% |
| 8192 | 4 | 981.5 | 1208.1 | 23.1% | 1602.3 | 63.3% | 32.6% |
| 8192 | 8 | 903.9 | 711.7 | -21.3% | 1426.6 | 57.8% | 100.4% |
| 8192 | 16 | 523.9 | 412.6 | -21.2% | 812.4 | 55.1% | 96.9% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 16384 | 1 | 1027.2 | 1319.0 | 28.4% | 1656.1 | 61.2% | 25.6% |
| 16384 | 2 | 1009.8 | 1259.3 | 24.7% | 1647.4 | 63.1% | 30.8% |
| 16384 | 4 | 977.9 | 1222.2 | 25.0% | 1593.4 | 62.9% | 30.4% |
| 16384 | 8 | 902.6 | 709.6 | -21.4% | 1424.7 | 57.8% | 100.8% |
| 16384 | 16 | 528.8 | 380.3 | -28.1% | 878.3 | 66.1% | 130.9% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 32768 | 1 | 1040.7 | 1315.2 | 26.4% | 1692.0 | 62.6% | 28.7% |
| 32768 | 2 | 1021.2 | 1265.5 | 23.9% | 1634.0 | 60.0% | 29.1% |
| 32768 | 4 | 995.8 | 1210.8 | 21.6% | 1597.4 | 60.4% | 31.9% |
| 32768 | 8 | 909.1 | 711.5 | -21.7% | 1437.2 | 58.1% | 102.0% |
| 32768 | 16 | 493.0 | 428.2 | -13.2% | 856.7 | 73.8% | 100.1% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 65536 | 1 | 1038.0 | 1318.6 | 27.0% | 1685.7 | 62.4% | 27.8% |
| 65536 | 2 | 1025.2 | 1282.0 | 25.1% | 1639.9 | 60.0% | 27.9% |
| 65536 | 4 | 996.7 | 1212.4 | 21.6% | 1602.5 | 60.8% | 32.2% |
| 65536 | 8 | 914.6 | 714.4 | -21.9% | 1437.1 | 57.1% | 101.2% |
| 65536 | 16 | 517.0 | 402.2 | -22.2% | 908.1 | 75.6% | 125.8% |
Elapsed 101.2s
> Use NativeCrc32 only if it is faster
> ------------------------------------
>
> Key: HADOOP-10778
> URL: https://issues.apache.org/jira/browse/HADOOP-10778
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Reporter: Tsz Wo Nicholas Sze
> Assignee: Tsz Wo Nicholas Sze
> Attachments: c10778_20140702.patch
>
>
> From the benchmark post in [this
> comment|https://issues.apache.org/jira/browse/HDFS-6560?focusedCommentId=14044060&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14044060],
> NativeCrc32 is slower than java.util.zip.CRC32 for Java 7 and above when
> bytesPerChecksum > 512.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
