Hi, I've started a new project that uses Mongrel. It basically lets you defer requests to it until other Mongrels in your (Rails) pool becomes free.
Rubyforge project page: http://rubyforge.org/projects/qrp/ gems, tarballs and git repo in case they haven't hit the mirrors yet: http://bogomips.org/ruby/ I should also add that nginx 0.6.7 or later is required for the "backup" feature I mention below in the README: Queueing Reverse Proxy (qrp) Ever pick the wrong line at the checkout counters in a crowded store? This is what happens to HTTP requests when you mix a multi-threaded Mongrel with Rails, which is single-threaded. qrp aims to be the simplest (worse-is-better) solution and have the lowest (adverse) impact to an existing setup. Background: An existing Rails site running Mongrel with nginx proxying to them. Unlike Apache, nginx fully buffers HTTP requests from clients before passing them off to Mongrels, which allows Mongrels to dedicate more cycles to running Rails itself. Problem: Rails is single-threaded; this is (probably) not easily fixable. By default, Mongrel will accept and queue requests while Rails is handling another request. Some Rails actions will take longer than others; and sometimes several seconds can be required to respond to an HTTP request. This problem is exacerbated if the Rails application queries third-party servers for information. Any queued requests inside Mongrel running Rails must wait until a slow Rails action has finished before they can run. If another Mongrel in the pool becomes free, then the requests that got queued behind a still-busy Mongrel would still be stuck and unable to get to the free Mongrel. Disabling concurrency on the Rails Mongrel (with "num_processors: 1" in the config)[1] will cause clients to be rejected outright and users will see 502 (Bad Gateway) errors. Bad Gateway errors getting returned to clients are bad, a slightly slower site is still better than a broken site. The developers also lack the resources to migrate to thread-safe platform (such as Merb or Waves) at the moment. Solution: Disable concurrency in Mongrels running Rails is part of the solution. Then setup a qrp or two as a backup member in your nginx configuration. Connections will normally go directly from nginx to Rails Mongrels (as before). However if all your regular Mongrels are busy, *then* nginx will send requests to the backup qrp instance(s). Once a request gets to qrp, qrp will retry the all the members in a given pool until a connection can be made and a response is returned. This avoids extra data copies of requests for the common (non-busy) case, and requires few changes to any existing infrastructure. Having fail_timeout=0 in the nginx config for every member of the Rails pool will allow nginx to immediately re-add a Rails Mongrel to the pool once the Rails Mongrel has finished processing. --- highlights of the nginx config: upstream mongrel { server 0:3000 fail_timeout=0; # Rails server 0:3001 fail_timeout=0; # Rails server 0:3002 fail_timeout=0; # Rails server 0:3003 fail_timeout=0; # Rails server 0:3500 backup; # qrp server 0:3501 backup; # qrp } --- highlights of the qrp config: # same Rails upstreams as in the nginx config upstreams: - 0:3000 - 0:3001 - 0:3002 - 0:3003 # ... --- highlight of the mongrel config[1]: num_processors: 1 Other existing solutions (and why I chose qrp): Fair-proxy balancer patch for nginx - this can keep new connections away from busy Mongrels, but if all (concurrency-disabled) Mongrels in your pool get busy, then you'll still get 502 errors. HAProxy - This will queue requests for you, but only if it makes all the connections to the backends itself. This means you cannot make other HTTP connections to the backends without confusing HAProxy; which (IMHO) defeats the purpose of using HTTP over a custom protocol. Swiftiply - admittedly I haven't tried it. It seems to require changes to our current infrastructure in deployment and monitoring tools. Additionally, the extra layer between nginx and Mongrel hurts performance for _every_ request, not just those that get unlucky. This also seems to take away the flexibility of being able to talk to any individual Mongrel process using plain HTTP. Footnotes: [1] - The current version of mongrel (1.1.3) does not handle the -n/--num-procs command-line option, and hence the current mongrel_cluster (1.0.5) is broken with it: http://mongrel.rubyforge.org/ticket/14 A better solution would be to use mongrel_cow_cluster (also a development of mine) as it handles the "num_processors:" directive correctly in the config file and also supports rolling restarts. /EOF -- Eric Wong _______________________________________________ Mongrel-users mailing list Mongrel-users@rubyforge.org http://rubyforge.org/mailman/listinfo/mongrel-users
