FWIW, I've been working on a way to simplify the use of queues with daemon
consumer threads
Sometimes, I launch one or more consumer threads that wait for a task to enter
a
queue and then work on the task. A recurring problem is that I sometimes need
to
know if all of the tasks have been completed so I can exit or do something with
the result.
If each thread only does a single task, I can use t.join() to wait until the
task is done. However, if the thread stays alive and waits for more Queue
entries, then there doesn't seem to be a good way to tell when all the
processing is done.
So, the idea is to create a subclass of Queue that increments a counter when
objects are enqueued, that provides a method for worker threads to decrement
the
counter when the work is done, and offers a blocking join() method that waits
until the counter is zero
# main thread
q = TaskQueue()
for t in worker_threads():
t.start()
for task in tasklist:
q.put(task) # increments the counter and enqueues a task
q.join() # all of the tasks are done (counter is
zero)
do_work_on_results()
# worker thread
while 1:
task = q.get() # task is popped but the counter is
unchanged
do_work(task)
q.decrement() # now the counter gets reduced
The idea is still in its infancy (no implementation and it hasn't been tried in
real-world code) but I would like to get some feedback. If it works out, I'll
post a recipe to ASPN and see how it goes.
Raymond
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
