Re: Multiprocessing Queue strange behavior
Hi, thanks for the answer.
I thought about that, but the problem is that I found the problem in code
that *was* using the Queue between processes. This code for example fails
around 60% of the time in one of our linux machines (raising an Empty
exception):
from processing import Queue, Process
import time
def Consume(queue):
print queue.get_nowait()
if __name__ == '__main__':
queue = Queue()
queue.put('x')
p = Process(target=Consume, args=(queue,))
p.start()
p.join()
Again, putting a sleep call before starting the consumer process makes the
code work all time. That is the main reason I believe there is a problem in
the Queue code somewhere.
Cheers,
On Wed, Sep 15, 2010 at 5:47 PM, MRAB pyt...@mrabarnett.plus.com wrote:
On 15/09/2010 21:10, Bruno Oliveira wrote:
Hi list,
I recently found a bug in my company's code because of a strange
behavior using multiprocessing.Queue. The following code snippet:
from multiprocessing import Queue
queue = Queue()
queue.put('x')
print queue.get_nowait()
Fails with:
...
File
E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py,
line 153, in getNoWait
return self.get(False)
File
E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py,
line 129, in get
raise Empty
Queue.Empty
Strangely, changing this to:
queue = Queue()
queue.put('x')
time.sleep(0.1) #
print queue.get_nowait()
Works as expected. Using the original snippet changing the import to
threading's Queue also works.
It seems like there's a bug in multiprocessing's Queue implementation.
Opinions?
I don't think it's a bug as such.
The purpose of the multiprocessing queue is to transfer data between
different processes, which don't have a shared address space (unlike
threads, which do).
The transfer involves passing the data between the processes via a
pipe. This is done in a background thread and takes some time to
complete, so the data won't appear immediately. It looks like it
doesn't matter that the putter and the getter happen to be in the same
process, possibly because no-one expected that someone would use a
multiprocessing queue within the same process like that, so it doesn't
check for a shortcut.
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Multiprocessing Queue strange behavior
Hi list,
I recently found a bug in my company's code because of a strange behavior
using multiprocessing.Queue. The following code snippet:
from multiprocessing import Queue
queue = Queue()
queue.put('x')
print queue.get_nowait()
Fails with:
...
File
E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py,
line 153, in getNoWait
return self.get(False)
File
E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py,
line 129, in get
raise Empty
Queue.Empty
Strangely, changing this to:
queue = Queue()
queue.put('x')
time.sleep(0.1) #
print queue.get_nowait()
Works as expected. Using the original snippet changing the import to
threading's Queue also works.
It seems like there's a bug in multiprocessing's Queue implementation.
Opinions?
Thanks.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing Queue strange behavior
On 15/09/2010 21:10, Bruno Oliveira wrote:
Hi list,
I recently found a bug in my company's code because of a strange
behavior using multiprocessing.Queue. The following code snippet:
from multiprocessing import Queue
queue = Queue()
queue.put('x')
print queue.get_nowait()
Fails with:
...
File
E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py,
line 153, in getNoWait
return self.get(False)
File
E:\Shared\dist-0902\i686.win32\processing-0.52\lib\site-packages\processing\queue.py,
line 129, in get
raise Empty
Queue.Empty
Strangely, changing this to:
queue = Queue()
queue.put('x')
time.sleep(0.1) #
print queue.get_nowait()
Works as expected. Using the original snippet changing the import to
threading's Queue also works.
It seems like there's a bug in multiprocessing's Queue implementation.
Opinions?
I don't think it's a bug as such.
The purpose of the multiprocessing queue is to transfer data between
different processes, which don't have a shared address space (unlike
threads, which do).
The transfer involves passing the data between the processes via a
pipe. This is done in a background thread and takes some time to
complete, so the data won't appear immediately. It looks like it
doesn't matter that the putter and the getter happen to be in the same
process, possibly because no-one expected that someone would use a
multiprocessing queue within the same process like that, so it doesn't
check for a shortcut.
--
http://mail.python.org/mailman/listinfo/python-list
