I have a multi-threaded python app (CherryPy WebApp to be exact) that launches
a child process that it then needs to communicate with bi-driectionally. To
implement this, I have used a pair of Queues: a child_queue which I use for
master->child communication, and a master_queue which is used for child->master
communication.
The way I have the system set up, the child queue runs a loop in a tread that
waits for messages on child_queue, and when received responds appropriately
depending on the message received, which sometimes involves posting a message
to master_queue.
On the master side, when it needs to communicate with the child process, it
posts a message to child_queue, and if the request requires a response it will
then immediately start waiting for a message on master_queue, typically with a
timeout.
While this process works well in testing, I do have one concern (maybe
unfounded) and a real-world issue
Concern: Since the master process is multi-threaded, it seems likely enough
that multiple threads on the master side would make requests at the same time.
I understand that the Queue class has locks that make this fine (one thread
will complete posting the message before the next is allowed to start), and
since the child process only has a single thread processing messages from the
queue, it should process them in order and post the responses (if any) to the
master_queue in order. But now I have multiple master processes all trying to
read master_queue at the same time. Again, the locks will take care of this and
prevent any overlapping reads, but am I guaranteed that the threads will obtain
the lock and therefore read the responses in the right order? Or is there a
possibility that, say, thread three will get the response that should have been
for thread one? Is this something I need to take into consideration, and if so,
how?
Real-world problem: While as I said this system worked well in testing, Now
that I have gotten it out into production I've occasionally run into a problem
where the master thread waiting for a response on master_queue times out while
waiting. This causes a (potentially) two-fold problem, in that first off the
master process doesn't get the information it had requested, and secondly that
I *could* end up with an "orphaned" message on the queue that could cause
problems the next time I try to read something from it.
I currently have the timeout set to 3 seconds. I can, of course, increase that,
but that could lead to a bad user experience - and might not even help the
situation if something else is going on. The actual exchange is quite simple:
On the master side, I have this code:
config.socket_queue.put('GET_PORT')
try:
port = config.master_queue.get(timeout=3) #wait up to three seconds for a
response
except Empty:
port = 5000 # default. Can't hurt to try.
Which, as you might have been able to guess, tries to ask the child process (an
instance of a tornado server, btw) what port it is listening on. The child
process then, on getting this message from the queue, runs the following code:
elif item == 'GET_PORT':
port = utils.config.getint('global', 'tornado.port')
master_queue.put(port)
So nothing that should take any significant time. Of course, since this is a
single thread handling any number of requests, it is possible that the thread
is tied up responding to a different request (or that the GIL is preventing the
thread from running at all, since another thread might be commandeering the
processor), but I find it hard to believe that it could be tied up for more
than three seconds.
So is there a better way to do sub-process bi-directional communication that
would avoid these issues? Or do I just need to increase the timeout (or remove
it altogether, at the risk of potentially causing the thread to hang if no
message is posted)? And is my concern justified, or just paranoid? Thanks for
any information that can be provided!
-----------------------------------------------
Israel Brewster
Systems Analyst II
Ravn Alaska
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7293
-----------------------------------------------
--
https://mail.python.org/mailman/listinfo/python-list
