On 6/16/06, Charles O Nutter <
[EMAIL PROTECTED]> wrote:
So we all know our friend Ola has contributed some great pieces of work over the past several months. As another JRubyist put it, "Ola is a f'n machine". StringIO, YAML, and Zlib are now mostly Java and far, far faster than they were. These contributions have made RubyGems far, far faster. From the original install time on this Opteron 2.6G of almost an hour (sans rdoc, mind you), we get this:
real 5m35.341s
user 0m36.379s
sys 0m5.195s
Impressive, by any measure. Ok, so I don't remember whether it was exactly an hour to install before, but it was bloody long enough to hit the drive-in for lunch. Grotesquely slow. Ola's changes alone dropped that time down to comparatively nothing. I went through the pains this evening of applying all his patches, cleaning up conflicts between them (since I didn't have a complete picture), and tidying up a few minor things. Then I ran a full Rails install. The time was impressive, no doubt about it. However, something really, really bothered me. In the time between gem download and gem install, the machine was sitting there, doing *nothing*. No network traffic, 0% CPU utilization. Zippo.
Then it hit me.
You've all heard the story about the timeout library. net/http calls net/protocol which reads in 1024-byte chunks from the stream with a timeout thread for each one. I cried about this on my blog too, since it seemed to be the source of a terrible performance problem. It turns out that once again, Java has already optimized our troubles away, in this case the cost of spinning up hundreds of threads.
Our NIO-based IO handler had a tiny flaw when it came to Socket IO: it was blocking. That meant that for the last read from the stream, the last few bytes of a gem file, it would sit and wait to fill up net/protocol's 1024-byte buffer. I am no NIO expert, but channels appear to behave somewhat differently when doing File versus Socket IO. At the end of a File, the stream is empty and read(ByteBuffer) will immediately return -1. With a Socket, however, if there are not enough bytes to fill the buffer and the socket is still open, it will block. Such is the case here. The HTTP get of a gem seems to leave the channel open. I'm not sure if this is intentional in RubyGems (for retrieving multiple gems) or a bug (potentially in JRuby) but leaving the Socket in blocking mode resulted in that last chunk of bytes waiting until timeout kicked in.
My fix (in the form of a patch to IOHandlerNIO) may or may not be correct; like I said, I'm no NIO expert. However, it seems to work well enough:
Attempting local installation of 'rails'
Local gem file not found: rails*.gem
Attempting remote installation of 'rails'
Updating Gem source index for: http://gems.rubyforge.org
Successfully installed rails-1.1.2
Successfully installed rake-0.7.1
Successfully installed activesupport-1.3.1
Successfully installed activerecord-1.14.2
Successfully installed actionpack-1.12.1
Successfully installed actionmailer-1.2.1
Successfully installed actionwebservice-1.1.2
real 1m3.751s
user 0m53.817s
sys 0m5.722s
For sake of comparison, Ruby installs Rails and its dependencies in about 25 seconds on my system. It's probably safe to say that we're only 2x slower for non-rdoc gem installations, which seems pretty good to me.
The NIO patch alone is attached, as is a complete patch with everything changed on my system, including:
- All Ola's stuff (stringio, zlib, and signal)
- My unrelated simplifications to Thread.critical= (not as deterministic as before but less likely or unlikely to deadlock...needs a bit more testing)
- My NIO fix
It will be nice to say that not only have we gotten RubyGems working, but it's been sped up by perhaps an order of magnitude since we first started.
--
Charles Oliver Nutter @ headius.blogspot.com
JRuby Developer @ jruby.sourceforge.net
Application Architect @ www.ventera.com
--
Charles Oliver Nutter @ headius.blogspot.com
JRuby Developer @ jruby.sourceforge.net
Application Architect @ www.ventera.com
_______________________________________________ Jruby-devel mailing list Jruby-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jruby-devel
