Ruby deserializer speed improvements
------------------------------------
Key: THRIFT-1189
URL: https://issues.apache.org/jira/browse/THRIFT-1189
Project: Thrift
Issue Type: Improvement
Components: Ruby - Library
Affects Versions: 0.6.1
Environment: OS X 10.6 i686 / Linux x86_64
Ruby 1.8.7-p334 / Ruby 1.9.2-p180
Reporter: Ilya Maykov
I have a patch to the Ruby libraries that greatly increases deserializer speed.
We've been running our production systems at Ooyala with this patch for weeks
but it was previously just a standalone file which changed some methods inside
Thrift code and we loaded it after requiring thrift. Over the weekend, I ported
it into a proper patch against the 0.6.1 tag and would like to commit it back.
I originally wrote this while trying to speed up some code we have that has to
deserialize a lot of thrift objects. I ran it under ruby-prof and noticed that
a huge amount of time was spent inside thrift deserialization code. Digging
deeper still, I saw a lot of time spent in String allocation and copy methods.
It turns out that there are several low-hanging fruit:
1) XProtocol#read_byte() methods end up calling read_all(1), getting back a
string of size 1, and converting it to a byte. This is an unnecessary string
alloc + copy that's pretty easy to get around. The patch does this by adding a
read_byte method to the XTransport classes. The transports that have buffering
of some kind (BufferedTransport, FramedTransport, MemoryBufferTransport) can
look up the byte, convert to unsigned, and return it without doing the extra
alloc + copy.
2) the BaseProtocol#read_all() method always allocates an empty buffer string,
reads bytes from the underlying transport, then appends the result to the
buffer. This extra string alloc + copy is also removed in my patch as it's not
needed.
3) Thrift::Struct#hash() is inefficient - it allocates an array and copies all
struct fields into it. Replaced with logic copied from Apache's Java
HashCodeBuilder class.
I've built a gem locally (i gave it version number 0.6.1.1) and wrote a simple
benchmark to test the changes. Here are the results:
|| Benchmark/Platform || r1.8.7-p334/thrift-0.6.0 || r1.8.7-p334/thrift-0.6.1.1
|| r1.9.2-p180/thrift-0.6.0 || r1.9.2-p180/thrift-0.6.1.1 ||
| Deserialization: BinaryProtocol | 16.01 | 10.23 | 8.73 | 5.58 |
| Deserialization: BinaryProtocolAccelerated | 11.46 | 6.31 | 5.73 | 3.64 |
| Deserialization: CompactProtocol | 11.56 | 3.73 | 5.70 | 2.61 |
| Hashing | 1.5 | 1.23 | 0.95 | 0.49 |
I will be attaching the patch and benchmark code shortly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
