On Jan 10, 1:02 pm, Emily Dresner <[email protected]> wrote:
> Hi --
>
> I have run into an issue with Sequel that has me quite stumped and I'm
> wondering if anyone can possibly lend a hand.
>
> I am running a web service under rack with ruby 1.9.2 using Sequel as
> the database layer and the mysql2 adapter to the MySQL database. When
> I write code in the method calls for the service using Sequel to send
> direct calls to the database using the database connect object, I have
> no problems at all. Sequel is, in this case, measurably faster than
> Active Record. However, in some places, I'm using the Sequel ORM and
> when I use the ORM Sequel seems to be crashing catastrophically on a
> mutex issue.
>
> The test setup is to run the service using unicorn with two worker
> processes. Then I flood the worker processes with a simple
> apachebench (ab) call. Around the 50th HTTP transaction, the threads
> simply freeze and never respond again. They must be killed with a
> judicious kill -9. I have hooked strace up to the worker processes
> and used tcpdump to look at the traffic between client to server and
> server to database. It looks like the connection is getting lost in
> the connection pool during the transaction between the Sequel code and
> the database. I can see Sequel sending a request to the MySQL adapter
> and the MySQL adapter sending a write() down the pipe to the database
> and then... crickets.
It's odd that you would see any threading related lockups with
unicorn, since unicorn is single threaded. If you are using unicorn,
you should probably be using Sequel's single threaded connection pool.
Also, no matter which database you are using, if you are using
multiple processes, make sure you disconnect the database connection
before forking. This is fairly easy in unicorn since it has a
before_fork hook. I think in passenger you can add it to the end of
config.ru. Not sure about thin, as I don't have much experience with
it. I don't think that would cause the problems you are experiencing,
but it is good general advice.
> To make matters more interesting, I'm running the code in an OpenVZ
> virtualized container. I can consistently reproduce the issue using
> OpenVZ to a MySQL database in a bunch of configurations -- different
> http server (passenger, thin, etc.), different OSes (Ubuntu, Debian,
> CentOS), and different adapters (MySQL, MySQL2). I have been able to
> eliminate the database as the source of the issue. And even beyond
> that, when I run the same code using the Sequel ORM on a bare metal
> system, I do not see the mutex hang issue. It only occurs under
> virtualization.
If it is consistently reproducible under virtualization and not
reproducible outside of virtualization, my jump to conclusions mat
leads me to believe it may be due to virtualization-related bugs. :)
It's possible Sequel triggers these bugs where ActiveRecord does not.
Of course, it's also possible that there are race conditions in Sequel
that are very unlikely to be hit on real hardware but more likely to
be hit in virtualization scenarios.
If the problem is due to Sequel's use of threading, it would be
database independent, because the only code in Sequel that really
deals with threading is the connection pool, and that is database
independent.
> I am currently going through every line of code in the Sequel gem and
> I do see some threading and synchronization code in the connection
> pool. What worries me is, as I move all our code off the ORM and to
> straight DB calls through Sequel, I'll run into these issues further
> on as I get closer to a go-live date and not have time to course-
> correct.
>
> Does anyone have any insight? Any test I might have overlooked? I do
> not want to be forced to move back to Active Record but putting
> systems into production that might not be stable worries me as well.
I've spent many hours going over Sequel's connection pool code, and
I'm not aware of any threading issues with it. I don't want to imply
there are none, as multi-threaded code is very difficult to prove
correct. If you have a reproducible test-case that you can share, I
would be very interested in trying it myself. I don't have any
experience with OpenVZ, but I do have a couple of VirtualBox-based
virtual machines that I use for Sequel testing.
If you can't share the code and are looking for additional ways to
debug the issue:
class Sequel::ThreadedConnectionPool
attr_accessor :mutex
end
m = DB.pool.mutex
def m.synchronize
tc = Thread.current.object_id
some_logger.debug("Before Outside #{tc}")
r = super do
some_logger.debug("Before Inside #{tc}")
yield
some_logger.debug("After Inside #{tc}")
end
some_logger.debug("After Outside #{tc}")
r
end
That should give you a log that allows you to see the order of events
leading up to the crash.
Jeremy
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/sequel-talk?hl=en.
