Hi, I have been having some problems with my Merb server lately and it is giving me a few mysterious errors.
I'm running a ten mongrel cluster and sometimes the webserver returns a "HTTP Error 500 Internal server error" to my c++ client which uploads files to the merb server using libcurl. What would happen is that the server would return a 500 internal server error yet it would still successfully save the file on the server, which leads to duplication file uploads since my client tries to re upload the file. After doing some debuging this is the error that my server gives randomly: merb : worker (port 4000) ~ Thread#join: deadlock 0xb674ebac - mutual join(0xb6746cf4) - (ThreadError) /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/dispatch/dispatcher.rb:100:in `synchronize' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/dispatch/dispatcher.rb:100:in `dispatch_action' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/dispatch/dispatcher.rb:74:in `handle' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/dispatch/dispatcher.rb:36:in `handle' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/rack/application.rb:17:in `call' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/rack/middleware/static.rb:28:in `call' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/rack/handler/mongrel.rb:82:in `process' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:159:in `process_client' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:158:in `each' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:158:in `process_client' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `run' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `initialize' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `new' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `run' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:268:in `initialize' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:268:in `new' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:268:in `run' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/rack/adapter/mongrel.rb:38:in `start_server' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/rack/adapter/abstract.rb:235:in `start_at_port' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/rack/adapter/abstract.rb:85:in `start' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/server.rb:174:in `bootup' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/server.rb:42:in `start' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core.rb:170:in `start' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/bin/merb:11 /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/bin/merb:19:in `load' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/bin/merb:19 This appear to only happen when the server is receiving a large number of requests. Lets say, 20-40 clients uploading one file each from a pool of 400 files. After looking at why this might be happening, I found something interesting in the dispatcher used to route requests to the controllers. The threads are being joined while the mutex locking is suppose to be off. I have it turned off in the init.rb file ( /config/ init.rb ). Merb::Config.use do |c| c[:use_mutex] = false end I found a ticket for this that says that the current work around is to use Merb::Dispatcher.use_mutex = false. [ https://merb.lighthouseapp.com/projects/7433/tickets/1174-merbconfiguse_mutex-issue ]. After I did this, I started to receive another problem. Instead of getting deadlocks on the threads, this is the new error i received: Thread #<Thread:0xb6753044 sleep> is too old, killing. Mon Nov 09 12:23:05 -0800 2009: Error reading HTTP body: #<Mongrel::TimeoutError: Mongrel timed out this thread: too many open files> /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:221:in `read_socket' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel/http_request.rb:77:in `read_body' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel/http_request.rb:55:in `initialize' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:149:in `new' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:149:in `process_client' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `run' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `initialize' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `new' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `run' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:268:in `initialize' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:268:in `new' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:268:in `run' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/rack/adapter/mongrel.rb:38:in `start_server' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/rack/adapter/abstract.rb:235:in `start_at_port' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/rack/adapter/abstract.rb:85:in `start' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/server.rb:174:in `bootup' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core/server.rb:42:in `start' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/lib/merb-core.rb:170:in `start' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.8.1/bin/merb:11 /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/bin/merb:19:in `load' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/bin/merb:19 So now I am under the impression that merb is creating threads, but is not closing the file descriptors that is using, which leads to my client failing to be able to even connect to the server and upload files. Up to this point, I have been running merb 1.0.8.1 so I figured maybe I would upgrade merb and see what happens. After upgrading merb to 1.0.15, I discovered that the mutex locking is still being read incorrectly from the init.rb file so I had to use the same workaround as before. In merb 1.0.15 when I have mutex locking ON my threads still get into deadlocks. After turning off mutex locking, a new error is now popping up in merb instead of too many file descriptors being open. After upgrading merb to 1.0.15 and turning off mutex locks, this is the error: Tue Nov 24 15:14:31 -0800 2009: Error reading HTTP body: #<RuntimeError: Socket read returned insufficient data: 193> /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel/http_request.rb:107:in `read_socket' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel/http_request.rb:77:in `read_body' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel/http_request.rb:55:in `initialize' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:149:in `new' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:149:in `process_client' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `run' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `initialize' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `new' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:285:in `run' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:268:in `initialize' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:268:in `new' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/ mongrel-1.1.5/lib/mongrel.rb:268:in `run' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.15/lib/merb-core/rack/adapter/mongrel.rb:38:in `start_server' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.15/lib/merb-core/rack/adapter/abstract.rb:296:in `start_at_port' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.15/lib/merb-core/rack/adapter/abstract.rb:128:in `start' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.15/lib/merb-core/server.rb:174:in `bootup' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.15/lib/merb-core/server.rb:42:in `start' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.15/lib/merb-core.rb:173:in `start' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/gems/merb- core-1.0.15/bin/merb:11 /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/bin/merb:19:in `load' /soft/fw/tools/ruby/ruby-1.8.6-p369-rhel3/lib/ruby/gems/bin/merb:19 Now after spending a few days with playing around with all this I am not sure how to proceed. Is this just a limitation of Merb and do I just have to increase the size of the mongrel cluster to lets say 15-20 from 10? Or is this a know issue and there is a plan to fix it in the near future? Thanks, Bane -- You received this message because you are subscribed to the Google Groups "merb" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/merb?hl=en.
