[
https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------
Attachment: c6166_20090810.patch
>> I think there may be some improvement possible in the 8_8 case in how the 9
>> XORs at the end are done. ...
>Yeah, we should try.
c6166_20090810.patch: tried various xor schemes for 8_8 and 16_16.
Seems Crc32_8_8d is the best choice. I will generate a patch to replace
current PureJavaCrc32 unless someone would like to run some benchmarks.
- Linux
java.version = 1.6.0_15
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_15-b03
java.vm.version = 14.1-b02
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp
Performance Table (The unit is MB/sec)
|| Num Bytes || CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b ||
Crc32_8_8c || Crc32_8_8d || Crc32_16_16 || Crc32_16_16b || Crc32_16_16c ||
Crc32_16_16d ||
| 8 | 49.617 | 186.277 | 245.369 | 258.905 |
229.094 | 246.095 | 217.684 | 209.102 | 225.104 |
216.537 |
| 16 | 83.794 | 260.552 | 254.068 | 253.335 |
240.414 | 269.863 | 268.293 | 280.991 | 248.725 |
272.588 |
| 32 | 129.851 | 311.725 | 319.791 | 317.253 |
300.914 | 325.145 | 294.224 | 306.928 | 294.640 |
294.838 |
| 64 | 175.566 | 348.701 | 357.292 | 359.353 |
345.997 | 370.976 | 325.410 | 349.629 | 343.592 |
346.502 |
| 128 | 217.394 | 369.562 | 386.217 | 384.014 |
371.929 | 392.099 | 339.749 | 382.753 | 378.737 |
381.048 |
| 256 | 246.060 | 381.159 | 403.724 | 401.239 |
389.683 | 403.455 | 346.794 | 394.901 | 397.287 |
398.093 |
| 512 | 261.941 | 385.598 | 412.253 | 405.512 |
395.790 | 412.212 | 350.763 | 409.029 | 408.213 |
407.630 |
| 1024 | 271.043 | 390.318 | 408.391 | 408.592 |
398.981 | 413.389 | 352.213 | 414.423 | 414.213 |
413.736 |
| 2048 | 275.942 | 391.870 | 412.933 | 411.371 |
401.784 | 417.020 | 351.803 | 418.063 | 416.982 |
415.269 |
| 4096 | 280.018 | 393.816 | 412.645 | 411.781 |
402.632 | 418.353 | 353.348 | 420.783 | 416.263 |
416.360 |
| 8192 | 281.432 | 394.699 | 414.998 | 410.618 |
401.145 | 417.090 | 351.428 | 420.008 | 415.680 |
415.281 |
| 16384 | 279.658 | 391.788 | 413.657 | 411.491 |
402.747 | 418.249 | 349.888 | 419.431 | 414.399 |
414.312 |
| 32768 | 279.788 | 391.665 | 408.767 | 410.097 |
403.396 | 417.542 | 349.473 | 418.467 | 409.388 |
412.525 |
| 65536 | 280.168 | 391.557 | 411.674 | 410.447 |
404.971 | 419.271 | 350.824 | 419.866 | 412.099 |
413.847 |
| 131072 | 281.196 | 393.222 | 411.687 | 411.539 |
404.931 | 418.101 | 350.692 | 418.924 | 412.873 |
413.763 |
| 262144 | 281.874 | 392.158 | 411.668 | 411.660 |
405.089 | 419.301 | 350.236 | 418.776 | 412.855 |
413.918 |
| 524288 | 281.905 | 392.967 | 411.713 | 410.524 |
404.851 | 418.481 | 350.890 | 419.835 | 412.881 |
412.677 |
| 1048576 | 281.864 | 393.155 | 411.589 | 410.697 |
405.352 | 417.905 | 350.737 | 419.861 | 412.734 |
413.808 |
| 2097152 | 281.269 | 392.916 | 410.495 | 411.322 |
405.298 | 419.084 | 349.765 | 419.540 | 412.517 |
413.538 |
| 4194304 | 280.904 | 388.513 | 408.313 | 410.383 |
404.432 | 418.168 | 349.982 | 418.628 | 411.404 |
412.493 |
| 8388608 | 279.946 | 389.306 | 407.485 | 407.420 |
401.088 | 415.597 | 347.824 | 415.479 | 408.459 |
409.290 |
| 16777216 | 279.517 | 388.903 | 407.068 | 406.760 |
400.404 | 415.171 | 347.468 | 415.047 | 408.089 |
408.576 |
- Windows
java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1
Performance Table (The unit is MB/sec)
|| Num Bytes || CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b ||
Crc32_8_8c || Crc32_8_8d || Crc32_16_16 || Crc32_16_16b || Crc32_16_16c ||
Crc32_16_16d ||
| 8 | 30.582 | 165.684 | 222.712 | 239.031 |
192.301 | 192.855 | 121.520 | 118.156 | 121.212 |
119.522 |
| 16 | 54.720 | 215.567 | 237.607 | 232.193 |
230.172 | 215.288 | 269.855 | 279.729 | 254.891 |
251.049 |
| 32 | 93.370 | 250.110 | 290.161 | 300.234 |
272.414 | 266.170 | 288.042 | 293.170 | 273.936 |
278.459 |
| 64 | 142.510 | 271.193 | 317.292 | 331.031 |
293.492 | 298.169 | 314.142 | 310.036 | 309.833 |
315.505 |
| 128 | 193.336 | 285.876 | 340.127 | 355.663 |
302.081 | 319.978 | 330.662 | 318.949 | 326.466 |
333.227 |
| 256 | 237.137 | 292.450 | 349.322 | 366.198 |
308.711 | 328.690 | 335.694 | 323.855 | 328.766 |
342.473 |
| 512 | 266.200 | 293.008 | 352.708 | 370.587 |
307.753 | 330.876 | 335.443 | 320.305 | 338.866 |
342.883 |
| 1024 | 275.926 | 292.009 | 347.310 | 367.795 |
305.459 | 332.800 | 328.240 | 291.745 | 270.544 |
338.249 |
| 2048 | 285.404 | 282.665 | 338.805 | 356.377 |
300.794 | 333.204 | 334.669 | 317.972 | 327.810 |
347.768 |
| 4096 | 295.626 | 296.351 | 353.792 | 374.937 |
310.589 | 337.225 | 338.851 | 323.519 | 335.347 |
349.135 |
| 8192 | 299.106 | 297.335 | 352.159 | 371.486 |
310.073 | 334.584 | 336.462 | 320.744 | 340.072 |
343.119 |
| 16384 | 298.202 | 295.121 | 353.141 | 371.399 |
311.208 | 347.209 | 340.819 | 321.550 | 341.870 |
349.779 |
| 32768 | 301.017 | 294.772 | 355.933 | 374.572 |
311.367 | 354.163 | 335.798 | 320.324 | 344.780 |
351.859 |
| 65536 | 305.349 | 297.090 | 355.641 | 377.599 |
310.232 | 354.429 | 337.837 | 324.306 | 345.244 |
350.304 |
| 131072 | 305.814 | 297.094 | 356.518 | 380.588 |
310.183 | 354.515 | 339.324 | 319.657 | 344.429 |
347.836 |
| 262144 | 303.674 | 294.812 | 352.616 | 377.256 |
306.132 | 349.959 | 330.275 | 319.694 | 339.153 |
347.561 |
| 524288 | 306.440 | 296.660 | 352.876 | 371.992 |
308.378 | 353.454 | 339.735 | 322.835 | 344.267 |
350.561 |
| 1048576 | 303.807 | 297.499 | 351.199 | 375.630 |
307.785 | 350.496 | 338.899 | 321.673 | 342.487 |
350.763 |
| 2097152 | 302.701 | 295.785 | 352.618 | 376.966 |
306.529 | 351.045 | 335.405 | 318.857 | 340.192 |
349.454 |
| 4194304 | 300.318 | 293.945 | 345.822 | 373.705 |
306.829 | 348.913 | 334.297 | 318.580 | 339.454 |
346.533 |
| 8388608 | 299.847 | 293.259 | 348.667 | 373.015 |
305.414 | 338.622 | 333.332 | 317.337 | 339.085 |
346.144 |
| 16777216 | 298.157 | 293.100 | 349.116 | 372.502 |
306.273 | 348.192 | 334.259 | 316.192 | 336.934 |
347.545 |
> Improve PureJavaCrc32
> ---------------------
>
> Key: HADOOP-6166
> URL: https://issues.apache.org/jira/browse/HADOOP-6166
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt,
> c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch,
> c6166_20090728.patch, c6166_20090810.patch
>
>
> Got some ideas to improve CRC32 calculation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
