[jira] [Updated] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-11660: -- Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: 3.0.0, 2.8.0 (was: 3.0.0) Status: Resolved (was: Patch Available) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 3.0.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-11660: -- Target Version/s: 2.8.0 (was: 3.0.0, 2.8.0) Affects Version/s: (was: 3.0.0) 2.8.0 Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 2.8.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Fix For: 2.8.0 Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
[ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-11660: -- Summary: Add support for hardware crc of HDFS checksums on ARM aarch64 architecture (was: Add support for hardware crc on ARM aarch64 architecture) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture -- Key: HADOOP-11660 URL: https://issues.apache.org/jira/browse/HADOOP-11660 Project: Hadoop Common Issue Type: Improvement Components: native Affects Versions: 3.0.0 Environment: ARM aarch64 development platform Reporter: Edward Nevill Assignee: Edward Nevill Priority: Minor Labels: performance Attachments: jira-11660.patch Original Estimate: 48h Remaining Estimate: 48h This patch adds support for hardware crc for ARM's new 64 bit architecture The patch is completely conditionalized on __aarch64__ I have only added support for the non pipelined version as I benchmarked the pipelined version on aarch64 and it showed no performance improvement. The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported on ARM aarch64 hardwre. To benchmark this I modified the test_bulk_crc32 test to print out the time taken to CRC a 1MB dataset 1000 times. Before: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55 After: CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57 So this represents a 5X performance improvement on raw CRC calculation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
