[
https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------
Attachment: c6166_20090811.patch
c6166_20090811.patch: added Crc32_8_8e and deleted some old classes.
> We probably want to have Todd's concurrency test from HADOOP-5318 run to make
> sure the larger lookup table doesn't slow things down under concurrency.
Todd, could you help running the test?
> We might also want to try the old four at a time code for 4 <= len < 8.
What do you mean exactly?
> We should also confirm the results on one of the other systems we tested in
> the past. I won't be able to do that for a couple days, but it should be easy
> then.
Hope you could find some time to do it soon.
> How many other variants did you try? Intel's C code does some strange things
> to group CRC's by 3. ...
Many others. You know, there are many combinations. I also tried different
table sizes as shown before.
Included Crc32_8_8e below. Crc32_8_8d still seems the best choice.
- java.version = 1.6.0_15
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_15-b03
java.vm.version = 14.1-b02
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp
Performance Table (The unit is MB/sec)
|| Num Bytes || CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b ||
Crc32_8_8c || Crc32_8_8d || Crc32_8_8e || Crc32_16_16 || Crc32_16_16b ||
Crc32_16_16c || Crc32_16_16d ||
| 1 | 7.554 | 71.591 | 79.205 | 103.936 |
80.965 | 79.185 | 80.812 | 84.215 | 80.733 |
80.732 | 84.860 |
| 2 | 14.768 | 104.753 | 110.837 | 110.771 |
110.798 | 115.222 | 110.820 | 119.278 | 110.844 |
110.836 | 119.207 |
| 4 | 27.150 | 177.142 | 119.587 | 114.780 |
120.051 | 128.626 | 117.005 | 125.412 | 115.439 |
120.409 | 125.486 |
| 8 | 49.921 | 193.631 | 239.096 | 238.506 |
228.868 | 248.667 | 244.446 | 217.270 | 209.170 |
209.149 | 215.505 |
| 16 | 83.886 | 259.453 | 261.683 | 254.016 |
240.835 | 258.250 | 247.261 | 267.441 | 279.439 |
276.864 | 271.520 |
| 32 | 128.960 | 312.202 | 323.233 | 319.924 |
301.175 | 322.962 | 320.275 | 295.428 | 302.402 |
294.960 | 295.872 |
| 64 | 177.529 | 349.882 | 362.037 | 364.918 |
347.768 | 365.013 | 358.567 | 326.303 | 352.837 |
345.517 | 347.880 |
| 128 | 217.577 | 370.920 | 387.254 | 386.534 |
372.472 | 391.180 | 386.159 | 339.850 | 383.044 |
373.716 | 381.058 |
| 256 | 245.685 | 382.041 | 403.693 | 401.793 |
390.270 | 406.037 | 402.777 | 347.134 | 400.956 |
388.941 | 399.331 |
| 512 | 263.143 | 385.666 | 411.380 | 407.443 |
396.506 | 413.556 | 411.803 | 350.844 | 411.293 |
397.335 | 409.011 |
| 1024 | 271.941 | 390.055 | 406.949 | 406.830 |
399.163 | 415.963 | 424.095 | 352.875 | 417.516 |
401.168 | 414.229 |
| 2048 | 276.881 | 392.684 | 412.759 | 411.123 |
402.143 | 418.730 | 424.565 | 353.111 | 418.855 |
402.870 | 415.696 |
| 4096 | 279.541 | 393.738 | 418.844 | 413.271 |
403.645 | 419.106 | 424.975 | 353.582 | 421.574 |
403.834 | 417.058 |
| 8192 | 280.308 | 392.859 | 417.096 | 412.115 |
403.015 | 417.989 | 422.254 | 352.011 | 419.664 |
402.378 | 415.385 |
| 16384 | 280.420 | 393.006 | 415.629 | 409.566 |
403.498 | 418.107 | 420.676 | 350.423 | 418.062 |
401.090 | 412.468 |
| 32768 | 280.488 | 392.172 | 410.737 | 411.129 |
403.873 | 417.434 | 413.839 | 349.526 | 418.633 |
401.736 | 412.341 |
| 65536 | 281.809 | 393.565 | 411.490 | 412.331 |
405.347 | 419.190 | 414.866 | 350.781 | 418.473 |
401.637 | 412.297 |
| 131072 | 281.994 | 393.634 | 411.533 | 412.281 |
405.387 | 419.620 | 414.802 | 350.869 | 418.350 |
401.634 | 412.346 |
| 262144 | 282.106 | 393.732 | 411.339 | 411.562 |
404.172 | 416.299 | 413.072 | 349.416 | 416.190 |
400.395 | 410.680 |
| 524288 | 281.194 | 392.195 | 409.839 | 410.759 |
403.954 | 417.950 | 413.178 | 350.104 | 418.596 |
401.175 | 412.137 |
| 1048576 | 282.160 | 393.700 | 411.316 | 412.339 |
405.165 | 419.694 | 414.608 | 351.029 | 416.836 |
401.927 | 412.122 |
| 2097152 | 281.584 | 393.264 | 410.322 | 405.048 |
402.028 | 414.893 | 406.574 | 328.270 | 410.558 |
393.239 | 403.796 |
| 4194304 | 274.062 | 385.751 | 409.093 | 408.604 |
402.332 | 419.176 | 412.265 | 350.039 | 415.385 |
400.332 | 410.738 |
| 8388608 | 279.870 | 385.793 | 406.788 | 407.685 |
400.830 | 415.537 | 409.612 | 344.396 | 412.684 |
397.534 | 406.201 |
| 16777216 | 279.585 | 389.025 | 405.902 | 407.222 |
400.414 | 414.680 | 409.184 | 346.930 | 413.039 |
396.057 | 406.953 |
- java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1
Performance Table (The unit is MB/sec)
|| Num Bytes || CRC32 || PureJavaCrc32 || Crc32_8_8 || Crc32_8_8b ||
Crc32_8_8c || Crc32_8_8d || Crc32_8_8e || Crc32_16_16 || Crc32_16_16b ||
Crc32_16_16c || Crc32_16_16d ||
| 1 | 4.669 | 62.483 | 67.995 | 64.360 |
64.700 | 64.430 | 51.386 | 61.391 | 62.175 |
68.944 | 62.271 |
| 2 | 9.959 | 79.342 | 88.566 | 79.881 |
91.326 | 88.695 | 81.976 | 89.846 | 83.984 |
91.166 | 84.918 |
| 4 | 19.448 | 136.996 | 119.941 | 129.939 |
120.992 | 119.898 | 132.349 | 118.682 | 116.076 |
122.505 | 114.090 |
| 8 | 36.518 | 168.433 | 240.284 | 219.896 |
213.865 | 212.471 | 209.605 | 115.325 | 135.040 |
145.009 | 139.316 |
| 16 | 66.157 | 233.146 | 272.227 | 258.220 |
250.948 | 245.771 | 253.216 | 313.174 | 321.156 |
289.904 | 285.301 |
| 32 | 111.982 | 283.081 | 327.881 | 326.083 |
311.988 | 300.416 | 299.799 | 331.685 | 338.723 |
316.987 | 319.500 |
| 64 | 169.448 | 315.802 | 364.829 | 368.059 |
348.219 | 340.445 | 323.045 | 360.273 | 357.086 |
352.497 | 359.832 |
| 128 | 227.468 | 335.743 | 384.770 | 394.813 |
373.775 | 368.234 | 338.247 | 379.686 | 363.607 |
372.900 | 382.183 |
| 256 | 274.901 | 348.319 | 395.674 | 410.081 |
386.566 | 388.908 | 345.215 | 385.142 | 368.920 |
383.446 | 392.874 |
| 512 | 307.317 | 353.626 | 401.379 | 422.212 |
392.543 | 399.756 | 348.747 | 391.548 | 372.486 |
389.674 | 400.035 |
| 1024 | 325.610 | 358.707 | 408.000 | 424.190 |
390.858 | 386.952 | 349.863 | 392.606 | 373.485 |
393.192 | 403.376 |
| 2048 | 337.261 | 361.642 | 410.563 | 429.318 |
395.794 | 388.887 | 351.070 | 392.865 | 373.458 |
394.564 | 404.084 |
| 4096 | 346.026 | 363.077 | 408.908 | 432.169 |
402.087 | 394.633 | 351.254 | 393.764 | 372.779 |
395.751 | 405.531 |
| 8192 | 348.807 | 363.550 | 407.264 | 431.891 |
409.774 | 400.138 | 350.242 | 393.582 | 373.939 |
394.498 | 405.981 |
| 16384 | 350.030 | 361.773 | 407.574 | 433.393 |
413.786 | 399.135 | 352.191 | 391.875 | 371.887 |
396.563 | 404.991 |
| 32768 | 350.552 | 360.131 | 408.495 | 428.463 |
413.372 | 406.904 | 352.049 | 390.723 | 371.160 |
393.841 | 403.526 |
| 65536 | 350.624 | 359.343 | 407.846 | 427.477 |
411.844 | 407.847 | 351.866 | 388.578 | 371.861 |
394.314 | 403.752 |
| 131072 | 351.383 | 358.687 | 407.854 | 426.513 |
411.453 | 407.655 | 352.480 | 388.961 | 372.343 |
394.181 | 401.394 |
| 262144 | 351.508 | 359.392 | 406.755 | 427.817 |
413.534 | 407.467 | 352.457 | 389.811 | 370.843 |
396.034 | 404.051 |
| 524288 | 350.264 | 358.659 | 407.698 | 427.750 |
411.986 | 408.091 | 351.678 | 390.126 | 371.458 |
395.884 | 404.374 |
| 1048576 | 351.092 | 358.452 | 406.796 | 426.634 |
413.797 | 408.162 | 351.430 | 389.716 | 372.157 |
394.826 | 403.300 |
| 2097152 | 349.731 | 357.048 | 407.095 | 426.922 |
411.427 | 405.970 | 351.830 | 388.308 | 371.514 |
394.656 | 401.170 |
| 4194304 | 345.948 | 354.513 | 401.117 | 423.269 |
408.442 | 401.112 | 348.512 | 384.834 | 365.403 |
390.539 | 399.911 |
| 8388608 | 343.162 | 350.602 | 400.009 | 415.757 |
403.662 | 399.569 | 344.909 | 382.301 | 365.704 |
386.523 | 397.156 |
| 16777216 | 343.989 | 351.502 | 400.036 | 417.893 |
403.717 | 400.511 | 345.505 | 381.932 | 363.491 |
385.044 | 393.422 |
> Improve PureJavaCrc32
> ---------------------
>
> Key: HADOOP-6166
> URL: https://issues.apache.org/jira/browse/HADOOP-6166
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt,
> c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch,
> c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch
>
>
> Got some ideas to improve CRC32 calculation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
