[
https://issues.apache.org/jira/browse/HDFS-6560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044060#comment-14044060
]
Tsz Wo Nicholas Sze commented on HDFS-6560:
-------------------------------------------
For Java 7, NativeCrc32 with direct buffer is faster than zip.CRC32 for
byte-per-crc < 512 but slower than it for byte-per-crc > 512. For byte-per-crc
== 512 (which is an important case), their performances are similar.
{noformat}
java.version = 1.7.0_60
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.7.0_60-b19
java.vm.version = 24.60-b09
java.vm.vendor = Oracle Corporation
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.7
java.specification.version = 1.7
os.arch = x86_64
os.name = Mac OS X
os.version = 10.9.3
DATA_LENGTH = 67108864
TRIALS = 10
{noformat}
Performance Table (bpc is byte-per-crc in MB/sec; #T = #Theads)
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 32 | 1 | 237.7 | 789.8 | 232.3% | 1624.2 | 583.3% | 105.7% |
| 32 | 2 | 207.6 | 604.5 | 191.2% | 1608.3 | 674.8% | 166.1% |
| 32 | 4 | 179.8 | 609.8 | 239.2% | 1387.8 | 671.9% | 127.6% |
| 32 | 8 | 163.4 | 356.8 | 118.3% | 910.4 | 457.1% | 155.1% |
| 32 | 16 | 81.6 | 183.7 | 125.0% | 490.9 | 501.4% | 167.3% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 64 | 1 | 423.7 | 1027.0 | 142.4% | 1654.4 | 290.4% | 61.1% |
| 64 | 2 | 417.7 | 1031.8 | 147.0% | 1640.1 | 292.7% | 59.0% |
| 64 | 4 | 366.0 | 693.8 | 89.5% | 1381.7 | 277.5% | 99.2% |
| 64 | 8 | 280.2 | 443.5 | 58.3% | 1046.8 | 273.5% | 136.0% |
| 64 | 16 | 143.3 | 233.0 | 62.6% | 556.3 | 288.2% | 138.8% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 128 | 1 | 716.1 | 1229.9 | 71.7% | 1628.6 | 127.4% | 32.4% |
| 128 | 2 | 703.0 | 1221.4 | 73.7% | 1610.0 | 129.0% | 31.8% |
| 128 | 4 | 708.1 | 998.7 | 41.0% | 1408.1 | 98.8% | 41.0% |
| 128 | 8 | 503.3 | 583.7 | 16.0% | 1059.4 | 110.5% | 81.5% |
| 128 | 16 | 259.6 | 316.4 | 21.9% | 610.3 | 135.2% | 92.9% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 256 | 1 | 1217.4 | 1346.6 | 10.6% | 1554.3 | 27.7% | 15.4% |
| 256 | 2 | 1186.3 | 1339.0 | 12.9% | 1556.6 | 31.2% | 16.3% |
| 256 | 4 | 1094.9 | 1102.9 | 0.7% | 1389.3 | 26.9% | 26.0% |
| 256 | 8 | 768.3 | 656.8 | -14.5% | 1109.4 | 44.4% | 68.9% |
| 256 | 16 | 394.6 | 358.7 | -9.1% | 597.8 | 51.5% | 66.7% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 512 | 1 | 1632.0 | 1391.3 | -14.7% | 1548.1 | -5.1% | 11.3% |
| 512 | 2 | 1608.9 | 1377.8 | -14.4% | 1550.1 | -3.7% | 12.5% |
| 512 | 4 | 1465.2 | 1092.6 | -25.4% | 1420.8 | -3.0% | 30.0% |
| 512 | 8 | 1027.7 | 721.7 | -29.8% | 1124.4 | 9.4% | 55.8% |
| 512 | 16 | 551.6 | 397.9 | -27.9% | 628.2 | 13.9% | 57.9% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 1024 | 1 | 1980.3 | 1411.7 | -28.7% | 1570.7 | -20.7% | 11.3% |
| 1024 | 2 | 1909.4 | 1396.7 | -26.9% | 1534.7 | -19.6% | 9.9% |
| 1024 | 4 | 1747.4 | 1159.9 | -33.6% | 1426.2 | -18.4% | 23.0% |
| 1024 | 8 | 1245.6 | 752.7 | -39.6% | 1149.8 | -7.7% | 52.8% |
| 1024 | 16 | 660.6 | 380.2 | -42.4% | 618.1 | -6.4% | 62.6% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 2048 | 1 | 2140.4 | 1390.2 | -35.0% | 1570.3 | -26.6% | 13.0% |
| 2048 | 2 | 2126.5 | 1374.5 | -35.4% | 1538.9 | -27.6% | 12.0% |
| 2048 | 4 | 1769.0 | 1132.9 | -36.0% | 1411.5 | -20.2% | 24.6% |
| 2048 | 8 | 1358.6 | 754.8 | -44.4% | 1207.0 | -11.2% | 59.9% |
| 2048 | 16 | 749.4 | 394.4 | -47.4% | 639.9 | -14.6% | 62.2% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 4096 | 1 | 2325.5 | 1427.0 | -38.6% | 1531.2 | -34.2% | 7.3% |
| 4096 | 2 | 2199.7 | 1375.1 | -37.5% | 1524.4 | -30.7% | 10.9% |
| 4096 | 4 | 1927.3 | 1103.8 | -42.7% | 1412.7 | -26.7% | 28.0% |
| 4096 | 8 | 1427.1 | 773.2 | -45.8% | 1206.2 | -15.5% | 56.0% |
| 4096 | 16 | 761.0 | 401.3 | -47.3% | 632.6 | -16.9% | 57.6% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 8192 | 1 | 2364.7 | 1431.6 | -39.5% | 1566.2 | -33.8% | 9.4% |
| 8192 | 2 | 2266.2 | 1382.1 | -39.0% | 1540.8 | -32.0% | 11.5% |
| 8192 | 4 | 1995.2 | 1175.2 | -41.1% | 1401.0 | -29.8% | 19.2% |
| 8192 | 8 | 1483.5 | 770.8 | -48.0% | 1144.6 | -22.8% | 48.5% |
| 8192 | 16 | 787.0 | 395.5 | -49.7% | 644.0 | -18.2% | 62.8% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 16384 | 1 | 2428.3 | 1423.9 | -41.4% | 1548.6 | -36.2% | 8.8% |
| 16384 | 2 | 2305.4 | 1381.9 | -40.1% | 1498.3 | -35.0% | 8.4% |
| 16384 | 4 | 1952.3 | 1133.4 | -41.9% | 1410.6 | -27.7% | 24.5% |
| 16384 | 8 | 1488.3 | 770.4 | -48.2% | 1227.3 | -17.5% | 59.3% |
| 16384 | 16 | 831.4 | 398.6 | -52.1% | 639.4 | -23.1% | 60.4% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 32768 | 1 | 2434.5 | 1420.6 | -41.6% | 1554.1 | -36.2% | 9.4% |
| 32768 | 2 | 2390.1 | 1396.7 | -41.6% | 1541.7 | -35.5% | 10.4% |
| 32768 | 4 | 2041.5 | 1122.9 | -45.0% | 1394.5 | -31.7% | 24.2% |
| 32768 | 8 | 1516.1 | 776.3 | -48.8% | 1188.1 | -21.6% | 53.1% |
| 32768 | 16 | 815.8 | 400.2 | -50.9% | 650.8 | -20.2% | 62.6% |
| bpc | #T || Zip || PureJava | % diff || Native | % diff | % diff |
| 65536 | 1 | 2349.2 | 1427.2 | -39.2% | 1539.3 | -34.5% | 7.9% |
| 65536 | 2 | 2361.0 | 1391.0 | -41.1% | 1519.3 | -35.7% | 9.2% |
| 65536 | 4 | 2010.3 | 1152.3 | -42.7% | 1413.2 | -29.7% | 22.6% |
| 65536 | 8 | 1530.8 | 777.1 | -49.2% | 1226.2 | -19.9% | 57.8% |
| 65536 | 16 | 831.0 | 386.4 | -53.5% | 615.1 | -26.0% | 59.2% |
Elapsed 182.0s
> Byte array native checksumming on DN side
> -----------------------------------------
>
> Key: HDFS-6560
> URL: https://issues.apache.org/jira/browse/HDFS-6560
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs-client, performance
> Reporter: James Thomas
> Assignee: Todd Lipcon
> Attachments: HDFS-3528.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
