[
https://issues.apache.org/jira/browse/THRIFT-401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12688839#action_12688839
]
Tyler Kovacs commented on THRIFT-401:
-------------------------------------
Just wanted to confirm that I had been working with an older revision (754551).
Trunk still uses slice! in the read method of FramedTransport so the call
trees that we observed in the older release should still apply to trunk. BTW,
I'm still running ruby 1.8.6 and slice! may have been optimized in ruby 1.9.
There's a new patch attached in "svn diff" format. It's shorter than the
original since the Transport classes in trunk don't implement borrow and
consume! methods. I don't know enough about Thrift to understand the
significance of that.
BTW, I'm having trouble running tests from the tree so I have yet to verify
that I didn't break everything.
# ruby spec/transport_spec.rb
/usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in
`gem_original_require': no such file to load --
./spec/../debug_proto_test/gen-rb/Srv (LoadError)
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in
`require'
from ./spec/spec_helper.rb:38
from spec/transport_spec.rb:1:in `require'
from spec/transport_spec.rb:1
> ruby transport class read buffer very slow due to usage of slice!
> -----------------------------------------------------------------
>
> Key: THRIFT-401
> URL: https://issues.apache.org/jira/browse/THRIFT-401
> Project: Thrift
> Issue Type: Improvement
> Components: Library (Ruby)
> Environment: # uname -a
> Linux zvm.local 2.6.9-78.0.1.ELsmp #1 SMP Tue Aug 5 11:02:47 EDT 2008 i686
> i686 i386 GNU/Linux
> Reporter: Tyler Kovacs
> Priority: Minor
> Attachments: after.png, before.png,
> remove_slice_from_framed_transport.diff
>
>
> We use Thrift as a cross-language transport for Hypertable - an open-source
> distributed database. While profiling queries with large response using the
> ruby Thrift libraries, we discovered that the majority of time was spent in
> thrift/transport.rb. Specifically, the slice! method, which is used to
> manage the read buffer (@rbuf) was responsible for almost all latency.
> We tried an alternative implementation that showed 300x speedup in our tests.
> Instead of repeatedly calling slice! to alter @rbuf (which apparently is
> extremely expensive), we maintain an offset counter (@rpos) which starts at
> zero and is incremented by sz each time we read from @rbuf. Before and after
> screenshots from kcachegrind are attached.
> I'll copy the monkey patch that we use within the description below - and
> I'll try to assemble a patch later today.
> module Thrift
> class FramedTransport < Transport
> def initialize(transport, read=true, write=true)
> @transport = transport
> @rbuf = ''
> @wbuf = ''
> @read = read
> @write = write
> @rpos = 0
> end
> def read(sz)
> return @transport.read(sz) unless @read
> return '' if sz <= 0
> read_frame if @rpos >= @rbuf.length
> @rpos += sz
> @rb...@rpos - sz, sz] || ''
> end
> def borrow(requested_length = 0)
> read_frame if @rpos >= @rbuf.length
> # there isn't any more coming, so if it's not enough, it's an error.
> raise EOFError if requested_length > (@rbuf.length - @rpos)
> @rb...@rpos, requested_length]
> end
> def consume!(size)
> @rpos += size
> @rb...@rpos - size, size]
> end
> private
> def read_frame
> sz = @transport.read_all(4).unpack('N').first
> @rpos = 0
> @rbuf = @transport.read_all(sz)
> end
> end
> end
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
