[issue9634] Add timeout parameter to Queue.join()

[Raymond Hettinger]

Raymond Hettinger  added the comment:

It's been 4 1/2 years since last activity.  Marking as closed again due to lack 
of interest and due to the rarity of need.  Also we know that a person can 
already customize join() via a subclass.

--
resolution:  -> out of date
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Swen Wenzel]

Swen Wenzel added the comment:

I have another use case.
The Docs use the producer-consumer pattern as a usage example.
I'm also using this pattern but apparently the consumers are not that stable 
during development phase.

So if one of your consumers dies during execution of its task, before it can 
say 'task done', then you will have a deadlock.
You could of course avoid this by adding a finally block which will claim the 
task is done even if there was some error or exception but then you will lie to 
the producer!
And suppose you only have one consumer and there are still tasks waiting (which 
is the case for my application), then you'll still have a deadlock since nobody 
is there to execute the remaining tasks.
This could be solved by putting the exception handling within the consumer's 
mainloop like this:

Consumer(Thread):
def __init__(self, queue):
self.queue = queue

def run():
while True:
task = self.queue.get()
try:
# execute task
except:
# handle exception (hopefully) without reraising one
finally:
self.queue.task_done()

This way, however, the producer won't notice any issues unless the consumer's 
exception handler sets a flag or puts the exception into a collection that can 
be checked by the producer.

But even that is no solution if the consumer executes a task with an endless 
loop or runs into a deadlock itself.

I would propose to throw an exception if queue.Queue.join() returns because of 
a timeout since then you can investigate the cause within the exception handler 
and do not have to check for the consumer's status after each join(). But this 
is microoptimization so I would also be satisfied with the same solution as for 
threading.Thread.join().

--
nosy: +Swen.Wenzel

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Yuri Bochkarev]

Changes by Yuri Bochkarev baltazar...@gmail.com:


--
nosy: +Yuri.Bochkarev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Martin Morrison]

Changes by Martin Morrison m...@ensoft.co.uk:


--
nosy: +isoschiz

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

The thread pool impl where I'm using this: 
http://git.fedorahosted.org/git/?p=pulpdist.git;a=blob;f=src/pulpdist/cli/thread_pool.py

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Nick, why don't you submit your workaround as a patch to Queue.join()?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

I probably will eventually - I'll be playing catchup a bit on other tasks this 
month after doing almost no coding for CPython since some time in Feb.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Matt Joiner]

Changes by Matt Joiner anacro...@gmail.com:


--
nosy: +anacrolix

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

I just created #14487 to request a documented API (Queue.pending()) that 
provides a formal mechanism for asking whether or not any jobs are still 
pending (analagous to the existing empty() and full() query methods).

Specifically, what I have is a client process that is executed periodically, 
gathers up a set of tasks and uses a thread pool to submit them in parallel to 
a synchronous API.

If all tasks complete with no new tasks being scheduled, then the client should 
terminate. However, if a new task arrives while any existing task is still in 
progress, then the client should submit it immediately (where, due to the 
time scales involved, immediately actually means within the next few 
minutes so I have plenty of scope to let the client block for a while instead 
of implementing a busy loop).

So, a timeout on join() would actually fit my use case better than the 
pending() API I proposed in the other issue. The processing loop would then 
look something like:

while have_tasks_to_process():
   submit_tasks_to_queue()
   try:
   task_queue.join(timeout)
   except Pending:
   pass

The advantage of having the timeout is that it would avoid the clumsy 
workarounds needed to avoid the busy loop created by the use of a query based 
approach. (Of course, I'm going to have to use the workaround anyway, since my 
client runs on Python 2.6, but still, I believe it meets the concrete use 
case criterion).

From a design aesthetic point of view, a pending() query API and a timeout on 
join() that throws a Pending exception would match the existing 
empty()/get()/Empty and full()/put()/Full API triples.

--
nosy: +ncoghlan
resolution: rejected - 
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

I killed off my new issue in favour of this one, since they're covering the 
same ground.

--
versions: +Python 3.3 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

The function below is the workaround I plan to use based on the 
undocumented-but-public Thread attributes that relate to the task queue (it's 
essentially just a combination of Queue.join() with the existing structure of 
the Queue.get() implementation):

  def join_queue(q, timeout=None):
  q.all_tasks_done.acquire()
  try:
  if timeout is None:
  while q.unfinished_tasks:
  self.all_tasks_done.wait()
  elif timeout  0:
  raise ValueError('timeout' must be a positive number)
  else:
  endtime = _time() + timeout
  while q.unfinished_tasks:
  remaining = endtime - _time()
  if remaining = 0.0:
  raise Pending
  self.all_tasks_done.wait(remaining)
  finally:
  q.all_tasks_done.release()

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

Kelly, thanks for posting the idea.
I'm going to close this one for now
but it can be reopened if compelling
use cases arise.

--
resolution:  - rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Kelly Lucas]

New submission from Kelly Lucas kdlu...@gmail.com:

I've seen quite a few people requesting to add a timeout value to the 
Queue.join() method, as it seems like a nice feature to have when waiting for 
queue's to finish.

Please add a feature so that Queue.join() will issue a 
self.all_tasks_done.release() when the timeout value is reached.

--
components: Library (Lib)
messages: 114257
nosy: kdlucas
priority: normal
severity: normal
status: open
title: Add timeout parameter to Queue.join()
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Kelly Lucas]

Changes by Kelly Lucas kdlu...@gmail.com:


--
versions: +Python 3.1 -Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Brian Curtin]

Changes by Brian Curtin cur...@acm.org:


--
stage:  - needs patch
type:  - feature request
versions: +Python 3.2 -Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

Can you provide links to the other requests?

This seems to be at odds with the whole premise
of what Queue.join() is trying to do (wait until
all worker threads have marked their tasks as done).

--
assignee:  - rhettinger
nosy: +rhettinger

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

FWIW, if you want to do other tasks in a main thread and need to check on the 
status task completion in worker threads, you can already loop on: 

 while q.unfinished_tasks:
  do_other_fun_stuff()
 q.join()  # this shouldn't block anymore

--
priority: normal - low

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Kelly Lucas]

Kelly Lucas kdlu...@gmail.com added the comment:

Here are a few that I saw:

http://stackoverflow.com/questions/1564501/add-timeout-argument-to-pythons-queue-join

http://www.eggheadcafe.com/software/aspnet/36145181/max-time-threads.aspx

http://efreedom.com/Question/1-1564501/Add-timeout-argument-to-python-s-Queue-join

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

We already have an optional timeout on threading.Thread.join(), which I suppose 
means that people need/want that feature.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

One of the provided links is duplicate (repost from StackOverflow).  None of 
the links offer use cases which can be used to inform the design (i.e. is this 
the correct approach, should a timeout raise a RuntimeError like 
Condition.wait() does, should a timeout set a variable to indicate the timeout 
thread.join() does, should it just return with no indication of the reason for 
returning, should we just document the unfinished_tasks variable as read-only 
so that users can check it periodically, what are the common calling patterns, 
how does this fit in with the original design intent of letting worker threads 
finish all pending tasks before moving on to code that assumes that the work is 
done).

Without motivating use cases, it is hard to create a correct design or to know 
whether this should be done at all.  Unlike, thread.join() which has a long 
history in many contexts, the Queue.join/task_done API is newer; accordingly, 
the API should be grown somewhat cautiously.  I don't think we've established 
using queue.join() with a time-out is a best practice or preferred pattern to 
solving a given problem.  Instead, all we have is it seems like a nice 
feature which falls short of our usual standards.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Kelly Lucas]

Kelly Lucas kdlu...@gmail.com added the comment:

This is easy enough to implement by subclassing the Queue class and overriding 
join(), so it's not a big deal. Just seems like it would be a nice thing to 
have.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9634] Add timeout parameter to Queue.join()

[Raymond Hettinger]

Raymond Hettinger rhettin...@users.sourceforge.net added the comment:

I will leave this open for a while.
It may be that it is a good idea
and that some good use cases will
emerge to inform a correct design.

In the meantime, it would be great
if you could post your subclass to
the ASPN Cookbook to collect feedback.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9634
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com