[issue4660] multiprocessing.JoinableQueue task_done() issue

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

Fix checked into python trunk with r74326, 26 maint w/ r74327

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

I used the protected JoinableQueue put method suggested by Brian.

--
resolution:  - fixed
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Brian]

Brian br...@merrells.org added the comment:

Cool., let me know if there is anything I can do to help.

On Mon, Jun 29, 2009 at 7:46 AM, Jesse Noller rep...@bugs.python.orgwrote:


 Jesse Noller jnol...@gmail.com added the comment:

 I'm leaning towards the properly protecting JoinableQueue.put() fix, I'm
 not a terribly big fan of removing error checking. I'm trying to carve off
 time this week to beat on my bug queue, so I'm hoping to be able to commit
 something (once I have docs+tests) this week.

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue4660
 ___


--
Added file: http://bugs.python.org/file14473/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___Cool., let me know if there is anything I can do to help.brbrbrdiv 
class=gmail_quoteOn Mon, Jun 29, 2009 at 7:46 AM, Jesse Noller span 
dir=ltrlt;a 
href=mailto:rep...@bugs.python.org;rep...@bugs.python.org/agt;/span 
wrote:br

blockquote class=gmail_quote style=border-left: 1px solid rgb(204, 204, 
204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;br
Jesse Noller lt;a href=mailto:jnol...@gmail.com;jnol...@gmail.com/agt; 
added the comment:br
br
I#39;m leaning towards the properly protecting JoinableQueue.put() fix, 
I#39;mbr
not a terribly big fan of removing error checking. I#39;m trying to carve 
offbr
time this week to beat on my bug queue, so I#39;m hoping to be able to 
commitbr
something (once I have docs+tests) this week.br
br
--br
divdiv/divdiv class=h5br
___br
Python tracker lt;a 
href=mailto:rep...@bugs.python.org;rep...@bugs.python.org/agt;br
lt;a href=http://bugs.python.org/issue4660; 
target=_blankhttp://bugs.python.org/issue4660/agt;br
___br
/div/div/blockquote/divbr
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Jesse Noller]

Jesse Noller jnol...@gmail.com added the comment:

I'm leaning towards the properly protecting JoinableQueue.put() fix, I'm 
not a terribly big fan of removing error checking. I'm trying to carve off 
time this week to beat on my bug queue, so I'm hoping to be able to commit 
something (once I have docs+tests) this week.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Brian]

Brian br...@merrells.org added the comment:

Filipe,

Thanks for the confirmation.  While I think the second option (ie 
properly protecting JoinableQueue.put()) is best, the first option 
(simply removing the 'task_done() called too many times' check) should 
be safe (presuming your get() and put() calls actually match).  

Jesse, any luck sorting out the best fix for this?  I really think that 
JoinableQueue (in my opinion the most useful form of multiprocessing 
queues) can't be guaranteed to work on any system right now.

-brian

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Filipe Fernandes]

Filipe Fernandes fernandes...@gmail.com added the comment:

I ran into the same problem and am greatful to Brian for reporting this
as I thought I was loosing my mind.

Brian noted that he was running windows and I can confirm that Brian's
test case is reproducable on my laptop running:

Ubuntu 9.04
python 2.6.2

Although I'm reluctant to try Brian's suggestions without additional
comments even if they do work.  I'll be using this in production.

--
nosy: +ffernand

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Alvaro]

Changes by Alvaro al...@ifca.unican.es:


--
nosy: +aloga

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Brian]

Brian br...@merrells.org added the comment:

Jesse,

I am afraid my last post may have confused the issue.  As I mentioned in
my first post, the problem arises when JoinableQueue.put is preempted
between its two lines.  Perhaps the easiest way to illustrate this is to
exacerbate it by modifying JoinableQueue.put to force a preemption at
this inopportune time.

import time
def put(self, item, block=True, timeout=None):
Queue.put(self, item, block, timeout)
time.sleep(1)
self._unfinished_tasks.release()

Almost any example will now fail.

from multiprocessing import JoinableQueue, Process

def printer(in_queue):
while True:
print in_queue.get()
in_queue.task_done()

if __name__ == '__main__':
jqueue = JoinableQueue()
a = Process(target = printer, args=(jqueue,)).start()
jqueue.put(blah)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Brian]

Brian br...@merrells.org added the comment:

Hey Jesse,
It was good meeting you at Pycon.  I don't have anything handy at the moment
although, if memory serves, the most trivial of example seemed to illustrate
the problem.  Basically any situation where a joinable queue would keep
bumping up against being empty (ie retiring items faster than they are being
fed), and does enough work between get() and task_done() to be preempted
would eventually break.  FWIW I was running on a Windows box.

I am afraid I am away from my computer until late tonight but I can try to
cook something up then (I presume you are sprinting today?).  Also I think
the issue becomes clear when you think about what happens if
joinablequeue.task_done() gets preempted between its few lines.

-brian

On Mon, Mar 30, 2009 at 2:55 PM, Jesse Noller rep...@bugs.python.orgwrote:


 Jesse Noller jnol...@gmail.com added the comment:

 Hi Brian - do you have a chunk of code that exacerbates this? I'm having
 problems reproducing this, and need a test so I can prove out the fix.

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue4660
 ___


--
Added file: http://bugs.python.org/file13485/unnamed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___Hey Jesse,divbr/divdivIt was good meeting you at Pycon.  I don#39;t 
have anything handy at the moment although, if memory serves, the most trivial 
of example seemed to illustrate the problem.  Basically any situation where a 
joinable queue would keep bumping up against being empty (ie retiring items 
faster than they are being fed), and does enough work between get() and 
task_done() to be preempted would eventually break.  FWIW I was running on a 
Windows box./div

divbr/divdivI am afraid I am away from my computer until late tonight 
but I can try to cook something up then (I presume you are sprinting today?). 
 Also I think the issue becomes clear when you think about what happens if 
joinablequeue.task_done() gets preempted between its few lines./div

divbr/divdiv-brian/divdivbr/divdivdiv class=gmail_quoteOn 
Mon, Mar 30, 2009 at 2:55 PM, Jesse Noller span dir=ltrlt;a 
href=mailto:rep...@bugs.python.org;rep...@bugs.python.org/agt;/span 
wrote:br

blockquote class=gmail_quote style=margin:0 0 0 .8ex;border-left:1px #ccc 
solid;padding-left:1ex;br
Jesse Noller lt;a href=mailto:jnol...@gmail.com;jnol...@gmail.com/agt; 
added the comment:br
br
Hi Brian - do you have a chunk of code that exacerbates this? I#39;m havingbr
problems reproducing this, and need a test so I can prove out the fix.br
br
--br
divdiv/divdiv class=h5br
___br
Python tracker lt;a 
href=mailto:rep...@bugs.python.org;rep...@bugs.python.org/agt;br
lt;a href=http://bugs.python.org/issue4660; 
target=_blankhttp://bugs.python.org/issue4660/agt;br
___br
/div/div/blockquote/divbr/div
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Jesse Noller]

Changes by Jesse Noller jnol...@gmail.com:


--
priority:  - high

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Brian]

Brian br...@merrells.org added the comment:

Here are a few stabs at how this might be addressed.

1)  As originally suggested.  Allow task_done() to block waiting to 
acquire _unfinished_tasks.  This will allow the put() process to resume, 
release() _unfinished_tasks at which point task_done() will unblock.  No 
harm, no foul but you do lose some error checking (and maybe some 
performance?)

2)  One can't protect JoinableQueue.put() by simply acquiring _cond 
before calling Queue.put().  Fixed size queues will block if the queue 
is full, causing deadlock when task_done() can't acquire _cond.  The 
most obvious solution would seem to be reimplementing 
JoinableQueue.put() (not simply calling Queue.put()) and then inserting  
self._unfinished_tasks.release() into a protected portion.  Perhaps:

def put(self, obj, block=True, timeout=None):
assert not self._closed
if not self._sem.acquire(block, timeout):
raise Full

self._notempty.acquire()
self._cond.acquire()
try:
if self._thread is None:
self._start_thread()
self._buffer.append(obj)
self._unfinished_tasks.release()
self._notempty.notify()
finally:
self._cond.release()
self._notempty.release()

We may be able to get away with not acquiring _cond as _notempty would 
provide some protection.   However its relationship to get() isn't 
entirely clear to me so I am not sure if this would be sufficient.

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Brian]

New submission from Brian br...@merrells.org:

Despite carefully matching my get() and task_done() statements I would 
often trigger raise ValueError('task_done() called too many times') in 
my multiprocessing.JoinableQueue (multiprocessing/queues.py)

Looking over the code (and a lot of debug logging), it appears that the 
issue arises from JoinableQueue.put() not being protected with a locking 
mechanism.  A preemption after the first line allows other processes to 
resume without releasing the _unfinished_tasks semaphore.

The simplest solution seems to be allowing task_done() to block while 
waiting to acquire the _unfinished_tasks semaphore.

Replacing:
if not self._unfinished_tasks.acquire(False):
  raise ValueError('task_done() called too many times')

With simply:
self._unfinished_tasks.acquire()

This would however remove the error checking provided (given the many 
far more subtler error that can be made, I might argue it is of limited 
value).  Alternately the JoinableQueue.put() method could be better 
protected.

--
components: Library (Lib)
messages: 77806
nosy: merrellb
severity: normal
status: open
title: multiprocessing.JoinableQueue task_done() issue
versions: Python 2.6, Python 2.7, Python 3.0, Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Brian]

Changes by Brian br...@merrells.org:


--
type:  - behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4660] multiprocessing.JoinableQueue task_done() issue

[Benjamin Peterson]

Changes by Benjamin Peterson musiccomposit...@gmail.com:


--
assignee:  - jnoller
nosy: +jnoller

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue4660
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com