Hi Danny,
So have written a little test script.
The fact is that I want to be able to manage the same queue from
separate threads. Below is an example of what my real program is
doing:
from threading import Thread
from Queue import Queue
import time
class SQLServer:
def __init__( self ):
self.queueinput = Queue()
self.queueoutput = Queue()
def query( self, sInput ):
# Puts an item to the input queue
self.queueinput.put( sInput )
# Wait for input queue to output result
while True:
# Get output from output queue
sOutput = self.queueoutput.get()
# Return result to caller
return sOutput
def shutdownOnIdle( self ):
self.queueinput.put( "QUIT!" )
# SERVER
def serveSqlQueue( self ):
while True:
# Get a job from the input queue
sQuery = self.queueinput.get()
# Check if server has to quit
if sQuery == "QUIT!": return
print 'Server does %s' % sQuery
self.queueoutput.put( sQuery )
# START THE SERVER
def startServer( self ):
Thread( target = self.serveSqlQueue ).start()
def separateCaller():
i = 1000
while True:
print '--Separate caller asks %i' % i
oSeparateOutput = server.query( i )
print '--Separate caller got %s' % str(oSeparateOutput)
i += 1
if i == 1004:
break
if __name__ == '__main__':
# Instantiate the server class
server = SQLServer()
# Start the server
server.startServer()
# Start the separate thread
Thread( target = separateCaller ).start()
print 'Main thread asks %s' % 'a'
oMainOutput = server.query( 'a' )
print 'Main thread got %s' % str(oMainOutput)
print 'Main thread asks %s' % 'b'
oMainOutput = server.query( 'b' )
print 'Main thread got %s' % str(oMainOutput)
print 'Main thread asks %s' % 'c'
oMainOutput = server.query( 'c' )
print 'Main thread got %s' % str(oMainOutput)
server.shutdownOnIdle()
When I run this, I get this output:
Main thread asks a
--Separate caller asks 1000
Server does 1000
--Separate caller got 1000
--Separate caller asks 1001
Server does a
--Separate caller got a
--Separate caller asks 1002
Server does 1001
--Separate caller got 1001
--Separate caller asks 1003
Server does 1002
--Separate caller got 1002
Server does 1003
Main thread got 1003
Main thread asks b
Server does b
Main thread got b
Main thread asks c
Server does c
Main thread got c
As you can see, the main thread and the separateCaller function have
mixed results. How can I "synchronize" the queues?
Thanks
Bernard
On 1/19/06, Bernard Lebel <[EMAIL PROTECTED]> wrote:
> Thanks a lot Danny,
>
> That certainly does make sense. I'll look into implementing the Queue
> approach in my program tomorrow. I remember you recommending me this
> module as well not long ago, although in a different discussion (where
> I suspected problem with file access from multiple thread, but I guess
> it's pretty much a similar problem, isn't it? :-)
>
>
> Thanks again
> Bernard
>
>
>
> On 1/19/06, Danny Yoo <[EMAIL PROTECTED]> wrote:
> >
> >
> > On Thu, 19 Jan 2006, Kent Johnson wrote:
> >
> >
> > > In your original desing were you sharing a connection between threads?
> > > That could cause trouble. But if each connection has its own thread and
> > > you are using transactions and isolation levels appropriately, they
> > > shouldn't stomp on each other.
> >
> > Hi Kent and Bernard,
> >
> > It's possible that MySQLdb was linked against the non-thread-safe
> > mysqlclient (rather than mysqlclient_r) library; that would probably cause
> > havoc.
> >
> >
> > > > So the solution was to start some sort of queue server in a separate
> > > > thread. This queue would consist of a list, and each time the program
> > > > would want to perform a MySQL operation, it would add it to the queue.
> >
> > We may want to use the Queue module here instead of a list; the way the
> > program is described now sounds like the server is busy-spinning when it
> > looks for new things to do. The thing that makes busy-spinning slightly
> > not-nice is that it consumes CPU regardless if the system's doing anything
> > or not.
> >
> > But a more serious problem than busy-waiting is one of thread-safety: the
> > object that's used to communicate between two threads --- a shared list
> > --- might be unreliable if the list methods aren't thread-safe. And I'm
> > not certain that all the list methods are thread-safe.
> >
> > That is, the problem with mutual exclusion may have just been pushed up
> > the program's structure, from the MySQL queries up to the shared queue
> > list access. *grin*
> >
> >
> > The synchronized Queue described in:
> >
> > http://www.python.org/doc/lib/module-Queue.html
> >
> > is designed to be a reliable communication medium between threads, and I
> > strongly recommend you look at it, especially because you've seen
> > first-hand the weird things that can happen with threads. *grin*
> >
> >
> > Here's a small example that shows what a different Queue can make:
> >
> > ############################################
> > from threading import Thread
> >
> > class Server:
> > def __init__(self):
> > self.queue = []
> >
> > def acceptJob(self, query):
> > self.queue.append((query,))
> >
> > def shutdownOnIdle(self):
> > self.queue.append("QUIT!")
> >
> > def jobLoop(self):
> > while True:
> > print "looping"
> > if len(self.queue) > 0:
> > nextJob = self.queue.pop(0)
> > if nextJob == "QUIT!":
> > return
> > print "I should do", nextJob
> >
> > def startServer(self):
> > Thread(target=self.jobLoop).start()
> >
> > if __name__ == '__main__':
> > server = Server()
> > server.startServer()
> > server.acceptJob("a")
> > server.acceptJob("b")
> > server.acceptJob("c")
> > server.shutdownOnIdle()
> > ############################################
> >
> >
> > Running this will show just how much work a busy-waiting thread does.
> >
> >
> > But if we change a few methods to use Queues instead of lists:
> >
> > ######
> > from threading import Thread
> > from Queue import Queue
> >
> > class Server:
> > def __init__(self):
> > self.queue = Queue()
> >
> > def acceptJob(self, query):
> > self.queue.put((query,))
> >
> > def shutdownOnIdle(self):
> > self.queue.put("QUIT!")
> >
> > def jobLoop(self):
> > while True:
> > print "looping"
> > nextJob = self.queue.get()
> > if nextJob == "QUIT!":
> > return
> > print "I should do", nextJob
> >
> > def startServer(self):
> > Thread(target=self.jobLoop).start()
> >
> > if __name__ == '__main__':
> > server = Server()
> > server.startServer()
> > server.acceptJob("a")
> > server.acceptJob("b")
> > server.acceptJob("c")
> > server.shutdownOnIdle()
> > ######
> >
> > and compare the output of this to the original implementation, it should
> > be clearer why Queues are cool. *grin*
> >
> >
> > Does this make sense? I hope this helps!
> >
> >
>
_______________________________________________
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
