[
https://issues.apache.org/jira/browse/HDFS-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052119#comment-13052119
]
Todd Lipcon commented on HDFS-2080:
-----------------------------------
bq. Did you compare the performance of "software" version with zlib?
zlib's implementation iirc is the straightforward byte-by-byte algorithm,
whereas the "software" implementation here is the "slicing-by-8" algorithm
which generally performs much better. I didn't do a rigorous comparison, though
I think I did notice a speedup when I switched from zlib to this implementation.
bq. Although it's not in the patch, I am sure you have play with it. Is there
anything you found useful in making this work?
I did some hacking here:
https://github.com/toddlipcon/cpp-dfsclient/blob/master/test_readblock.cc
See the read_packet() function and the crc32cHardware64_3parallel(...) code.
This code does run faster than the "naive" non-pipelined implementation, though
I didn't do a rigorous benchmark here either.
I figure it would be best to post the patch above before going all-out on
optimization.
A few other notes on the patch:
- a few unit tests are failing because of bugs in the tests (eg not creating a
socket with an associated Channel, or assuming read() will always return the
requested size)
- the use of native byte buffers could cause a leak - we need some kind of
pooling/buffer reuse here to avoid the native memory leak
Sadly this project is "for fun" for me at the moment so I probably won't be
able to circle back for a little while. I will try to post a patch which
addresses some of the above bugs though tonight.
> Speed up DFS read path
> ----------------------
>
> Key: HDFS-2080
> URL: https://issues.apache.org/jira/browse/HDFS-2080
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2080.txt
>
>
> I've developed a series of patches that speeds up the HDFS read path by a
> factor of about 2.5x (~300M/sec to ~800M/sec for localhost reading from
> buffer cache) and also will make it easier to allow for advanced users (eg
> hbase) to skip a buffer copy.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
