[issue32879] Race condition in multiprocessing Queue

[Eric Meyer]

Eric Meyer  added the comment:

The problem I was seeing ended up being improper GIL management in a c++ 
extension. It seems putting items that were created without the GIL on the 
queue causes a seg fault.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32879] Race condition in multiprocessing Queue

[Eric Meyer]

Eric Meyer  added the comment:

multiprocessing.SimpleQueue is another workaround to this issue.

I agree the docs should be clearer about this. Additionally, it would be 
helpful if there was a way to optionally put an item on a multiprocessing.Queue 
and block until the item has been written to the pipe (or block until it is 
safe to mutate,garbage collect,etc the item in the calling thread).

--
nosy: +highvelcty

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32879] Race condition in multiprocessing Queue

[Tim Peters]

Tim Peters  added the comment:

The docs could be clearer about this:  the argument to .put() is _not_ pickled 
at the time .put() is called.  The object is remembered (by reference, not by 
value), and a feeder thread pickles the value and puts the pickle on the queue 
when the feeder thread gets around to that.  So if the object is mutated in any 
way in between, it's not defined whether the pre- or post-mutation state is put 
on the queue (or, in some cases, even some partially mutated value).

To make it wholly deterministic you could, e.g., change the .put() to this:

queue.put(data[:])

Or you could use a Manager().Queue() instead, which doesn't use a feeder thread.

--
nosy: +tim.peters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32879] Race condition in multiprocessing Queue

[Simon Bouchard]

New submission from Simon Bouchard :

The clear list function call in made after the put(data) on the queue. But the 
data is sometime clear in the queue (randomly). Since both function are call 
within the same process, a race condition is not expected.

--
files: code.py
messages: 312391
nosy: TwistedSim
priority: normal
severity: normal
status: open
title: Race condition in multiprocessing Queue
type: behavior
versions: Python 3.6
Added file: https://bugs.python.org/file47453/code.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com