[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14584577#comment-14584577 ] Andrew Pinski commented on HADOOP-11660: On aarch64 a crc32 takes 3 exec cycles, but it only has one crc unit, so if we did the same on aarch64 and had say crc32 w0, w0, x3 crc32 w1, w1, x4 crc32 w2, w2, x5 The second crc32 w1, w1, x4 would have to wait for the 1st crc to complete and the 3rd would have to wait for the 2nd, taking 9 cycles in any case, so there is no benefit to pipelining. This is not true on some AARCH64 processors. For ThunderX, this is definitely not true. crc32 (32bits) is fully pipelined and the next one will issue right away. Though the latency of those instructions are 4 cycles long. So this will only take 6 cycles on ThunderX. Thanks, Andrew Pinski Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14584776#comment-14584776 ] Edward Nevill commented on HADOOP-11660: Hi Andrew, Yes. Please see my comments of 4/Mar/15. The version of the CRC patch checked into Hadoop does support pipelining. On A57 without pipelining the raw CRC speed was 4.5X better. With pipelining it was 11X better. Regards, Ed. Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388409#comment-14388409 ] Hudson commented on HADOOP-11660: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #149 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/149/]) HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 architecture (Edward Nevill via Colin P. McCabe) (cmccabe: rev d9ac5ee2c4dcd4a108ca892af501618caaea450c) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c * hadoop-common-project/hadoop-common/src/CMakeLists.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java * hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388429#comment-14388429 ] Hudson commented on HADOOP-11660: - SUCCESS: Integrated in Hadoop-Yarn-trunk #883 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/883/]) HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 architecture (Edward Nevill via Colin P. McCabe) (cmccabe: rev d9ac5ee2c4dcd4a108ca892af501618caaea450c) * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c * hadoop-common-project/hadoop-common/src/CMakeLists.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java * hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c * hadoop-common-project/hadoop-common/CHANGES.txt Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388600#comment-14388600 ] Hudson commented on HADOOP-11660: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2081 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2081/]) HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 architecture (Edward Nevill via Colin P. McCabe) (cmccabe: rev d9ac5ee2c4dcd4a108ca892af501618caaea450c) * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c * hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c * hadoop-common-project/hadoop-common/src/CMakeLists.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388622#comment-14388622 ] Hudson commented on HADOOP-11660: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #140 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/140/]) HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 architecture (Edward Nevill via Colin P. McCabe) (cmccabe: rev d9ac5ee2c4dcd4a108ca892af501618caaea450c) * hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c * hadoop-common-project/hadoop-common/src/CMakeLists.txt Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389142#comment-14389142 ] Hudson commented on HADOOP-11660: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2099 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2099/]) HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 architecture (Edward Nevill via Colin P. McCabe) (cmccabe: rev d9ac5ee2c4dcd4a108ca892af501618caaea450c) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java * hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/CMakeLists.txt * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389075#comment-14389075 ] Hudson commented on HADOOP-11660: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #149 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/149/]) HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 architecture (Edward Nevill via Colin P. McCabe) (cmccabe: rev d9ac5ee2c4dcd4a108ca892af501618caaea450c) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java * hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c * hadoop-common-project/hadoop-common/src/CMakeLists.txt * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c * hadoop-common-project/hadoop-common/CHANGES.txt Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387384#comment-14387384 ] Colin Patrick McCabe commented on HADOOP-11660: --- +1. Thanks, [~enevill]. Just a quick note for the future, we usually leave around the old patch when attaching a new patch. Then we have files like jira-11660.001.patch, jira-11660.002.patch, etc. attached so we can see the evolution of the patch over time. Committed to 2.8. Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 3.0.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387420#comment-14387420 ] Hudson commented on HADOOP-11660: - SUCCESS: Integrated in Hadoop-trunk-Commit #7469 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7469/]) HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 architecture (Edward Nevill via Colin P. McCabe) (cmccabe: rev d9ac5ee2c4dcd4a108ca892af501618caaea450c) * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_aarch64.c * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestNativeCrc32.java * hadoop-common-project/hadoop-common/src/main/native/src/test/org/apache/hadoop/util/test_bulk_crc32.c * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32.c * hadoop-common-project/hadoop-common/src/CMakeLists.txt * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/bulk_crc32_x86.c Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
