On Thu, 19 Jan 2006, Kent Johnson wrote:
> In your original desing were you sharing a connection between threads?
> That could cause trouble. But if each connection has its own thread and
> you are using transactions and isolation levels appropriately, they
> shouldn't stomp on each other.
Hi Kent and Bernard,
It's possible that MySQLdb was linked against the non-thread-safe
mysqlclient (rather than mysqlclient_r) library; that would probably cause
havoc.
> > So the solution was to start some sort of queue server in a separate
> > thread. This queue would consist of a list, and each time the program
> > would want to perform a MySQL operation, it would add it to the queue.
We may want to use the Queue module here instead of a list; the way the
program is described now sounds like the server is busy-spinning when it
looks for new things to do. The thing that makes busy-spinning slightly
not-nice is that it consumes CPU regardless if the system's doing anything
or not.
But a more serious problem than busy-waiting is one of thread-safety: the
object that's used to communicate between two threads --- a shared list
--- might be unreliable if the list methods aren't thread-safe. And I'm
not certain that all the list methods are thread-safe.
That is, the problem with mutual exclusion may have just been pushed up
the program's structure, from the MySQL queries up to the shared queue
list access. *grin*
The synchronized Queue described in:
http://www.python.org/doc/lib/module-Queue.html
is designed to be a reliable communication medium between threads, and I
strongly recommend you look at it, especially because you've seen
first-hand the weird things that can happen with threads. *grin*
Here's a small example that shows what a different Queue can make:
############################################
from threading import Thread
class Server:
def __init__(self):
self.queue = []
def acceptJob(self, query):
self.queue.append((query,))
def shutdownOnIdle(self):
self.queue.append("QUIT!")
def jobLoop(self):
while True:
print "looping"
if len(self.queue) > 0:
nextJob = self.queue.pop(0)
if nextJob == "QUIT!":
return
print "I should do", nextJob
def startServer(self):
Thread(target=self.jobLoop).start()
if __name__ == '__main__':
server = Server()
server.startServer()
server.acceptJob("a")
server.acceptJob("b")
server.acceptJob("c")
server.shutdownOnIdle()
############################################
Running this will show just how much work a busy-waiting thread does.
But if we change a few methods to use Queues instead of lists:
######
from threading import Thread
from Queue import Queue
class Server:
def __init__(self):
self.queue = Queue()
def acceptJob(self, query):
self.queue.put((query,))
def shutdownOnIdle(self):
self.queue.put("QUIT!")
def jobLoop(self):
while True:
print "looping"
nextJob = self.queue.get()
if nextJob == "QUIT!":
return
print "I should do", nextJob
def startServer(self):
Thread(target=self.jobLoop).start()
if __name__ == '__main__':
server = Server()
server.startServer()
server.acceptJob("a")
server.acceptJob("b")
server.acceptJob("c")
server.shutdownOnIdle()
######
and compare the output of this to the original implementation, it should
be clearer why Queues are cool. *grin*
Does this make sense? I hope this helps!
_______________________________________________
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
