[
https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096269#comment-14096269
]
Todd Lipcon commented on HDFS-6561:
-----------------------------------
Hey James. I looked over this patch and wrote/ran some performance tests.
At first, I was a little concerned that the way you consolidated the code for
bulk_crc32 would slow things down. In particular, there's now a new branch per
chunk to determine whether to store or verify the CRC. I was worried that, if
this branch were mispredicted, we'd pay an extra 15-20 cycles for every
512-byte chunk (which at 0.13 cycles/byte only takes ~66 cycles). That would
represent a close to 20% performance regression.
So, I wrote a simple test which approximates exactly the HDFS usage of these
APIs -- ie 512 byte chunks and a reasonable amount of data. In this test, I
found that the above concern was unwarranted - probably because the branch
prediction unit does a very good job with the simple branch pattern here. I'll
attach a version of your patch which includes the benchmark that I wrote in
case anyone else wants to run it.
Here are my average timings for 512MB of 512-byte-chunked checksums (on my
Intel(R) Core(TM) i7-4700MQ CPU @ 2.40GHz)
*Before*:
Calculate: 401275us (1.28GB/sec)
Verify: 41184us (12.43GB/sec
*After*:
Calculate: 41808us (12.25GB/sec)
Verify: 41604us (12.31GB/sec)
These seem to match earlier results you've posted elsewhere - just wanted to
confirm on my machine and make sure that the existing verify code path didn't
regress due to the new functionality.
For ease of review, I also think it makes sense to split this patch up a little
further, and make this JIRA only do the changes to the checksumming code to
allow for native calculation. The changes to FSOutputSummer, DFSOutputStream,
etc, are a bit more complex and probably should be reviewed separately. I took
the liberty of removing those chunks from the patch as I was testing it, so
I'll upload that here and you can take a look.
Given the above, I only reviewed the portion related to checksumming and didn't
yet look in detail at the outputsummer, etc, changes.
> Byte array native checksumming on client side
> ---------------------------------------------
>
> Key: HDFS-6561
> URL: https://issues.apache.org/jira/browse/HDFS-6561
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, hdfs-client, performance
> Reporter: James Thomas
> Assignee: James Thomas
> Attachments: HDFS-6561.2.patch, HDFS-6561.3.patch, HDFS-6561.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
