I would try something like this inside _process_outgoing:
while not self.stop_requested.isSet():
data_ok = False
while not self.stop_requested.isSet():
if not self.outgoing.empty():
try:
data = self.outgoing.get(True,0.1)
data_ok = True
break
except Queue.Empty:
pass
else:
time.sleep(0.1) # maybe, if the thread usess too much
CPU
I tried your modifications. The result is 9 messages/sec. I guess it is
because time.sleep(0.1) - if that is called after each message, at most
10 message/sec is possible.
The hypotesis I made for this suggestion are:
- if the queue is found empty, the queue.get could keep the global
interpreter lock until a message arrive
(blocking also the thread that put the message in the queue)
I hope it is not that way. Consider two threads running intensive
calculations. Both of them are running python code -> both of them must
hold the GIL. Python runs the process on one processor, and GIL must be
released on any thread that is not active on the CPU, regardless of what
it is doing.
- if the queue is found empty, the handling of the exception can slows
down the execution
Because of the timeout specified in my outgoing queue get() call, the
exception can only happen once per second. It won't slow down anything.
Not sure they are good guesses, because at the rate of your messages
the queue should be almost always full,
and most probably the implementation of Queue.get is smarted than
myself :-). ANyway, it is worth a try ...
The problem is with sending one message, and getting one answer for it.
Most of the time, the queue is empty or has only one message inside it.
Also, is you are using fixed-length queues (the ones that make the
sender wait if the queue is full), try to increase
the queue size, or to use an infinite-size queue.
Queue size was 1000 in this example, but it never contains more than two
messages.
--
http://mail.python.org/mailman/listinfo/python-list
