[
https://issues.apache.org/jira/browse/HADOOP-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HADOOP-6166:
-------------------------------------------
Attachment: c6166_20090819.patch
> I tried this before but it did not help. ...
I should be more clear: it did help for the 4-byte case but it hurt the
performance in some other cases.
c6166_20090819.patch: added Crc32_8_8b2 and Crc32_8_8d2. See whether anyone
wants to play with them.
Here is my results.
- java.version = 1.6.0_15
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_15-b03
java.vm.version = 14.1-b02
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = amd64
os.name = Linux
os.version = 2.6.9-55.ELsmp
Performance Table (The unit is MB/sec)
|| Num Bytes || CRC32 || PureJavaCrc32 || Crc32_8_8b || Crc32_8_8b2 ||
Crc32_8_8d || Crc32_8_8d2 ||
| 1 | 7.236 | 106.903 | 58.534 | 61.106 |
109.742 | 67.673 |
| 2 | 14.605 | 104.665 | 104.573 | 104.952 |
115.005 | 102.944 |
| 4 | 27.566 | 177.502 | 223.599 | 228.648 |
131.890 | 218.530 |
| 8 | 49.708 | 193.198 | 196.794 | 175.006 |
251.469 | 204.796 |
| 16 | 83.804 | 259.008 | 234.906 | 221.473 |
258.878 | 236.937 |
| 32 | 128.918 | 313.157 | 299.622 | 273.306 |
322.913 | 300.376 |
| 64 | 176.797 | 349.738 | 339.404 | 320.543 |
365.522 | 348.328 |
| 128 | 217.087 | 370.589 | 362.759 | 353.613 |
390.209 | 384.627 |
| 256 | 244.869 | 381.630 | 376.096 | 369.157 |
405.767 | 401.649 |
| 512 | 261.282 | 385.554 | 382.974 | 378.841 |
413.286 | 411.211 |
| 1024 | 271.205 | 390.170 | 384.064 | 384.586 |
415.817 | 413.505 |
| 2048 | 276.269 | 392.448 | 386.215 | 387.585 |
418.559 | 417.466 |
| 4096 | 279.280 | 393.668 | 387.571 | 388.814 |
418.777 | 419.686 |
| 8192 | 280.938 | 394.176 | 388.399 | 389.576 |
420.204 | 420.571 |
| 16384 | 281.385 | 393.164 | 389.518 | 388.856 |
420.236 | 420.392 |
| 32768 | 281.442 | 391.935 | 389.115 | 387.306 |
419.440 | 417.362 |
| 65536 | 281.605 | 391.986 | 389.124 | 387.297 |
419.405 | 417.123 |
| 131072 | 281.334 | 393.221 | 389.093 | 384.810 |
417.836 | 415.334 |
| 262144 | 280.363 | 391.701 | 386.141 | 385.937 |
417.907 | 415.452 |
| 524288 | 280.373 | 391.727 | 387.646 | 385.911 |
417.851 | 415.510 |
| 1048576 | 280.295 | 391.585 | 387.535 | 385.745 |
417.587 | 414.222 |
| 2097152 | 280.866 | 391.217 | 387.413 | 385.603 |
417.605 | 414.120 |
| 4194304 | 280.273 | 390.247 | 387.650 | 385.971 |
418.284 | 414.302 |
| 8388608 | 279.799 | 388.960 | 385.093 | 383.170 |
414.179 | 412.095 |
| 16777216 | 279.473 | 388.359 | 384.629 | 382.737 |
413.717 | 411.523 |
- java.version = 1.6.0_14
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_14-b08
java.vm.version = 14.0-b16
java.vm.vendor = Sun Microsystems Inc.
java.vm.name = Java HotSpot(TM) Server VM
java.vm.specification.version = 1.0
java.specification.version = 1.6
os.arch = x86
os.name = Windows XP
os.version = 5.1
Performance Table (The unit is MB/sec)
|| Num Bytes || CRC32 || PureJavaCrc32 || Crc32_8_8b || Crc32_8_8b2 ||
Crc32_8_8d || Crc32_8_8d2 ||
| 1 | 3.744 | 49.901 | 47.314 | 49.920 |
51.063 | 45.306 |
| 2 | 7.397 | 78.714 | 64.757 | 63.904 |
69.411 | 69.742 |
| 4 | 14.129 | 118.892 | 116.559 | 122.589 |
100.686 | 120.545 |
| 8 | 27.169 | 141.788 | 178.545 | 179.537 |
177.252 | 176.913 |
| 16 | 48.864 | 197.479 | 199.851 | 206.187 |
205.112 | 197.922 |
| 32 | 87.972 | 242.980 | 252.241 | 243.120 |
248.911 | 230.985 |
| 64 | 128.459 | 262.461 | 272.747 | 284.252 |
276.611 | 264.473 |
| 128 | 178.257 | 265.661 | 305.895 | 303.167 |
293.562 | 286.069 |
| 256 | 213.740 | 280.286 | 307.318 | 324.065 |
296.782 | 297.760 |
| 512 | 247.829 | 287.490 | 334.125 | 332.208 |
317.107 | 310.138 |
| 1024 | 266.095 | 289.453 | 343.593 | 340.496 |
322.506 | 330.703 |
| 2048 | 284.693 | 286.926 | 328.546 | 338.577 |
323.831 | 324.033 |
| 4096 | 292.326 | 304.553 | 354.299 | 351.903 |
332.555 | 334.845 |
| 8192 | 276.416 | 286.101 | 343.856 | 336.716 |
327.936 | 317.006 |
| 16384 | 284.944 | 292.414 | 338.566 | 333.038 |
323.622 | 314.212 |
| 32768 | 275.270 | 292.939 | 339.562 | 332.072 |
332.398 | 294.426 |
| 65536 | 271.285 | 294.269 | 336.092 | 339.905 |
327.122 | 306.583 |
| 131072 | 288.692 | 290.441 | 337.711 | 312.378 |
240.516 | 299.113 |
| 262144 | 288.789 | 290.830 | 342.317 | 331.144 |
322.446 | 312.372 |
| 524288 | 277.527 | 288.056 | 337.654 | 329.802 |
326.141 | 301.740 |
| 1048576 | 278.305 | 288.149 | 332.875 | 328.402 |
320.754 | 300.196 |
| 2097152 | 278.992 | 291.338 | 324.239 | 325.872 |
324.888 | 294.238 |
| 4194304 | 282.476 | 281.645 | 331.034 | 332.346 |
318.677 | 296.442 |
| 8388608 | 283.086 | 287.335 | 328.317 | 331.580 |
325.669 | 306.567 |
| 16777216 | 290.031 | 296.875 | 339.540 | 332.101 |
323.918 | 307.348 |
> Improve PureJavaCrc32
> ---------------------
>
> Key: HADOOP-6166
> URL: https://issues.apache.org/jira/browse/HADOOP-6166
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Reporter: Tsz Wo (Nicholas), SZE
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: c6166_20090722.patch, c6166_20090722_benchmark_32VM.txt,
> c6166_20090722_benchmark_64VM.txt, c6166_20090727.patch,
> c6166_20090728.patch, c6166_20090810.patch, c6166_20090811.patch,
> c6166_20090819.patch, graph.r, graph.r, Rplots-laptop.pdf,
> Rplots-nehalem32.pdf, Rplots-nehalem64.pdf, Rplots.pdf, Rplots.pdf, Rplots.pdf
>
>
> Got some ideas to improve CRC32 calculation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
