Awesome! Really great! This is starting to become _really_ fun! ... maybe it's time to do a hprof on the RDoc stuff?
/O At 09:29 2006-06-16, you wrote: >So we all know our friend Ola has contributed some great pieces of work >over the past several months. As another JRubyist put it, "Ola is a f'n >machine". StringIO, YAML, and Zlib are now mostly Java and far, far faster >than they were. These contributions have made RubyGems far, far faster. > From the original install time on this Opteron 2.6G of almost an hour >(sans rdoc, mind you), we get this: > >real 5m35.341s >user 0m36.379s >sys 0m5.195s > >Impressive, by any measure. Ok, so I don't remember whether it was exactly >an hour to install before, but it was bloody long enough to hit the >drive-in for lunch. Grotesquely slow. Ola's changes alone dropped that >time down to comparatively nothing. I went through the pains this evening >of applying all his patches, cleaning up conflicts between them (since I >didn't have a complete picture), and tidying up a few minor things. Then I >ran a full Rails install. The time was impressive, no doubt about it. >However, something really, really bothered me. In the time between gem >download and gem install, the machine was sitting there, doing *nothing*. >No network traffic, 0% CPU utilization. Zippo. > >Then it hit me. > >You've all heard the story about the timeout library. net/http calls >net/protocol which reads in 1024-byte chunks from the stream with a >timeout thread for each one. I cried about this on my blog too, since it >seemed to be the source of a terrible performance problem. It turns out >that once again, Java has already optimized our troubles away, in this >case the cost of spinning up hundreds of threads. > >Our NIO-based IO handler had a tiny flaw when it came to Socket IO: it was >blocking. That meant that for the last read from the stream, the last few >bytes of a gem file, it would sit and wait to fill up net/protocol's >1024-byte buffer. I am no NIO expert, but channels appear to behave >somewhat differently when doing File versus Socket IO. At the end of a >File, the stream is empty and read(ByteBuffer) will immediately return -1. >With a Socket, however, if there are not enough bytes to fill the buffer >and the socket is still open, it will block. Such is the case here. The >HTTP get of a gem seems to leave the channel open. I'm not sure if this is >intentional in RubyGems (for retrieving multiple gems) or a bug >(potentially in JRuby) but leaving the Socket in blocking mode resulted in >that last chunk of bytes waiting until timeout kicked in. > >My fix (in the form of a patch to IOHandlerNIO) may or may not be correct; >like I said, I'm no NIO expert. However, it seems to work well enough: > >Attempting local installation of 'rails' >Local gem file not found: rails*.gem >Attempting remote installation of 'rails' >Updating Gem source index for: ><http://gems.rubyforge.org>http://gems.rubyforge.org >Successfully installed rails-1.1.2 >Successfully installed rake-0.7.1 >Successfully installed activesupport-1.3.1 >Successfully installed activerecord-1.14.2 >Successfully installed actionpack-1.12.1 >Successfully installed actionmailer-1.2.1 >Successfully installed actionwebservice-1.1.2 > >real 1m3.751s >user 0m53.817s >sys 0m5.722s > >For sake of comparison, Ruby installs Rails and its dependencies in about >25 seconds on my system. It's probably safe to say that we're only 2x >slower for non-rdoc gem installations, which seems pretty good to me. > >The NIO patch alone is attached, as is a complete patch with everything >changed on my system, including: > >- All Ola's stuff (stringio, zlib, and signal) >- My unrelated simplifications to Thread.critical= (not as deterministic >as before but less likely or unlikely to deadlock...needs a bit more testing) >- My NIO fix > >It will be nice to say that not only have we gotten RubyGems working, but >it's been sped up by perhaps an order of magnitude since we first started. > >-- >Charles Oliver Nutter @ <http://headius.blogspot.com>headius.blogspot.com >JRuby Developer @ <http://jruby.sourceforge.net>jruby.sourceforge.net >Application Architect @ <http://www.ventera.com>www.ventera.com >Content-Type: text/x-patch; name=fast_gems_complete.patch; charset=ISO-8859-1 >X-Attachment-Id: f_eoi7pv7y >Content-Disposition: attachment; filename="fast_gems_complete.patch" > >Content-Type: text/x-patch; name=nio_nonblocking_socket.patch; > charset=ISO-8859-1 >X-Attachment-Id: f_eoi7q336 >Content-Disposition: attachment; filename="nio_nonblocking_socket.patch" > >Content-Type: text/x-patch; name=thread_critical_simplification.patch; > charset=ISO-8859-1 >X-Attachment-Id: f_eoi7qxff >Content-Disposition: attachment; > filename="thread_critical_simplification.patch" > > >_______________________________________________ >Jruby-devel mailing list >Jruby-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/jruby-devel _______________________________________________ Jruby-devel mailing list Jruby-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jruby-devel
