[
https://issues.apache.org/jira/browse/HADOOP-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16048982#comment-16048982
]
LiXin Ge commented on HADOOP-14424:
-----------------------------------
Thanks for your advice, [~iwasakims].
I'm done to combine the two performance functions into one test, some interface
and functions was modified to support both Crc32 and Crc32C.
It's only print diff to one of the results now and a part of the output of
Crc32PerformanceTest is:
Direct Buffer():
{quote}
...
| bpc | #T || Zip || PureJava | % diff || PureJavaC | % diff || Native
| % diff || NativeC | % diff |
| 32 | 1 | 154.5 | 357.2 | 131.2% | 349.2 | -2.2% | 783.3
| 124.3% | 3146.0 | 301.6% |
| 32 | 2 | 154.3 | 354.0 | 129.4% | 350.2 | -1.1% | 794.8
| 126.9% | 1710.1 | 115.2% |
| 32 | 4 | 98.4 | 294.0 | 198.8% | 339.6 | 15.5% | 782.2
| 130.3% | 2615.0 | 234.3% |
| 32 | 8 | 112.1 | 231.7 | 106.6% | 199.8 | -13.7% | 708.4
| 254.5% | 1833.0 | 158.7% |
| 32 | 16 | 65.0 | 137.2 | 111.2% | 144.2 | 5.1% | 442.9
| 207.1% | 1107.7 | 150.1% |
...
{quote}
Non-direct Buffer:
{quote}
...
| bpc | #T || Zip || PureJava | % diff || PureJavaC | % diff || NativeC
| % diff |
| 32 | 1 | 161.3 | 402.0 | 149.3% | 396.9 | -1.3% | 3028.9
| 663.1% |
| 32 | 2 | 158.3 | 403.1 | 154.6% | 396.4 | -1.7% | 1697.1
| 328.1% |
| 32 | 4 | 156.8 | 402.7 | 156.8% | 394.7 | -2.0% | 1898.1
| 380.9% |
| 32 | 8 | 157.4 | 402.8 | 155.9% | 392.7 | -2.5% | 1897.4
| 383.2% |
| 32 | 16 | 91.4 | 224.1 | 145.2% | 219.9 | -1.9% | 1202.8
| 446.9% |
...
{quote}
> Add CRC32C performance test.
> ----------------------------
>
> Key: HADOOP-14424
> URL: https://issues.apache.org/jira/browse/HADOOP-14424
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
> Affects Versions: 3.0.0-alpha2
> Reporter: LiXin Ge
> Assignee: LiXin Ge
> Priority: Minor
> Labels: test
> Attachments: HADOOP-14424.002.patch, HADOOP-14424.patch
>
>
> The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new
> test to compare Crc32C chunked verification implementations.
> This test is based on Crc32PerformanceTest, what I have done in this test is:
> 1.CRC32C performance test.
> 2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it
> from this test.
> 3.User can choose either direct buffer or non-directly buffer to run this
> test manually.
> 4.Take use of verifyChunkedSumsByteArray for native to support non-directly
> native test.
> The test result in my environment is:
> [root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest
> java.version = 1.8.0_111
> java.runtime.name = Java(TM) SE Runtime Environment
> java.runtime.version = 1.8.0_111-b14
> java.vm.version = 25.111-b14
> java.vm.vendor = Oracle Corporation
> java.vm.name = Java HotSpot(TM) 64-Bit Server VM
> java.vm.specification.version = 1.8
> java.specification.version = 1.8
> os.arch = amd64
> os.name = Linux
> os.version = 2.6.33.20
> Data Length = 64 MB
> Trials = 5
> Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads)
> | bpc | #T || PureJava || Native | % diff |
> | 32 | 1 | 394.0 | 4156.2 | 954.9% |
> | 32 | 2 | 400.5 | 3679.7 | 818.7% |
> | 32 | 4 | 401.8 | 2657.0 | 561.3% |
> | 32 | 8 | 389.1 | 1633.8 | 319.9% |
> | 32 | 16 | 222.2 | 1116.3 | 402.5% |
> | bpc | #T || PureJava || Native | % diff |
> | 64 | 1 | 465.0 | 5931.0 | 1175.5% |
> | 64 | 2 | 468.8 | 1839.2 | 292.3% |
> | 64 | 4 | 460.4 | 2968.3 | 544.7% |
> | 64 | 8 | 452.4 | 1925.7 | 325.6% |
> | 64 | 16 | 246.9 | 1291.8 | 423.3% |
> | bpc | #T || PureJava || Native | % diff |
> | 128 | 1 | 522.0 | 6147.8 | 1077.6% |
> | 128 | 2 | 366.0 | 4758.5 | 1200.2% |
> | 128 | 4 | 307.8 | 3265.1 | 960.8% |
> | 128 | 8 | 283.6 | 2092.2 | 637.6% |
> | 128 | 16 | 219.9 | 1226.1 | 457.6% |
> | bpc | #T || PureJava || Native | % diff |
> | 256 | 1 | 550.7 | 3177.6 | 477.0% |
> | 256 | 2 | 538.6 | 1933.2 | 258.9% |
> | 256 | 4 | 427.2 | 3278.1 | 667.3% |
> | 256 | 8 | 420.8 | 2272.3 | 440.0% |
> | 256 | 16 | 294.0 | 1311.2 | 346.0% |
> | bpc | #T || PureJava || Native | % diff |
> | 512 | 1 | 553.4 | 3690.4 | 566.9% |
> | 512 | 2 | 455.6 | 4974.1 | 991.7% |
> | 512 | 4 | 494.2 | 3406.4 | 589.2% |
> | 512 | 8 | 431.4 | 2257.0 | 423.2% |
> | 512 | 16 | 316.3 | 1272.0 | 302.2% |
> | bpc | #T || PureJava || Native | % diff |
> | 1024 | 1 | 566.1 | 3520.0 | 521.8% |
> | 1024 | 2 | 508.7 | 4437.4 | 772.3% |
> | 1024 | 4 | 520.7 | 3422.6 | 557.4% |
> | 1024 | 8 | 501.8 | 2124.7 | 323.4% |
> | 1024 | 16 | 340.6 | 1305.0 | 283.2% |
> | bpc | #T || PureJava || Native | % diff |
> | 2048 | 1 | 535.1 | 5438.5 | 916.4% |
> | 2048 | 2 | 537.3 | 4668.3 | 768.8% |
> | 2048 | 4 | 529.2 | 2417.2 | 356.7% |
> | 2048 | 8 | 485.1 | 2249.8 | 363.8% |
> | 2048 | 16 | 334.3 | 1265.7 | 278.6% |
> | bpc | #T || PureJava || Native | % diff |
> | 4096 | 1 | 563.0 | 7264.0 | 1190.1% |
> | 4096 | 2 | 538.8 | 5681.4 | 954.4% |
> | 4096 | 4 | 528.9 | 3107.6 | 487.5% |
> | 4096 | 8 | 521.8 | 2168.9 | 315.6% |
> | 4096 | 16 | 336.5 | 1256.5 | 273.4% |
> | bpc | #T || PureJava || Native | % diff |
> | 8192 | 1 | 521.1 | 6842.4 | 1213.1% |
> | 8192 | 2 | 543.0 | 5510.2 | 914.8% |
> | 8192 | 4 | 526.1 | 3759.5 | 614.6% |
> | 8192 | 8 | 480.1 | 2254.1 | 369.6% |
> | 8192 | 16 | 336.4 | 1213.3 | 260.6% |
> | bpc | #T || PureJava || Native | % diff |
> | 16384 | 1 | 521.7 | 3695.7 | 608.3% |
> | 16384 | 2 | 540.8 | 5610.5 | 937.4% |
> | 16384 | 4 | 539.1 | 3313.0 | 514.6% |
> | 16384 | 8 | 518.7 | 2198.4 | 323.8% |
> | 16384 | 16 | 337.2 | 1237.8 | 267.1% |
> | bpc | #T || PureJava || Native | % diff |
> | 32768 | 1 | 532.1 | 3696.0 | 594.6% |
> | 32768 | 2 | 541.4 | 4647.4 | 758.4% |
> | 32768 | 4 | 536.6 | 2325.1 | 333.3% |
> | 32768 | 8 | 518.8 | 2036.9 | 292.6% |
> | 32768 | 16 | 337.8 | 1232.0 | 264.7% |
> | bpc | #T || PureJava || Native | % diff |
> | 65536 | 1 | 527.7 | 8240.0 | 1461.4% |
> | 65536 | 2 | 561.9 | 5472.3 | 873.9% |
> | 65536 | 4 | 536.5 | 3725.2 | 594.4% |
> | 65536 | 8 | 482.8 | 2040.2 | 322.5% |
> | 65536 | 16 | 335.9 | 1240.4 | 269.3% |
> Elapsed 77.4s
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
