Charles-François Natali neolo...@free.fr added the comment:
Modifying an object which is already on a traditional queue can also
change what is received by the other thread (depending on timing).
So Queue.Queue's put() is not atomic either. Therefore I do not
believe this behaviour is a
sbt shibt...@gmail.com added the comment:
Modifying an object which is already on a traditional queue can also change
what is received by the other thread (depending on timing). So Queue.Queue's
put() is not atomic either. Therefore I do not believe this behaviour is a
bug.
However the
sbt shibt...@gmail.com added the comment:
I meant Issue 6721 (Locks in python standard library should be sanitized on
fork) not 6271.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
Ask Solem a...@opera.com added the comment:
Pickling on put makes sense to me. I can't think of cases where this could
break existing code either. I think this may also resolve issue 8323
--
stage: - unit test needed
___
Python tracker
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
Please find attached an updated patch (including unit test) for the
release27-maint branch. I've run the test_multiprocessing suite against svn
revision 86129 of said branch.
--
Added file:
Ask Solem a...@opera.com added the comment:
AFAICS the object will be pickled twice with this patch.
See Modules/_multiprocessing/connection.h: connection_send_obj.
--
nosy: +asksol
___
Python tracker rep...@bugs.python.org
Ask Solem a...@opera.com added the comment:
aha, no. I see now you use connection.send_bytes instead.
Then I can't think of any issues with this patch, but I don't know why
it was done this way in the first place.
--
___
Python tracker
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
The idea is to pickle the object immediately (i.e. when added) instead of when
requested (dequeued). This means that any operations (even deletion) performed
on the original object do not make changes to the item in the queue,
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
--
versions: +Python 3.1
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
___
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
Removed file: http://bugs.python.org/file16473/queues.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
Updated patch to include new test (now against release26-maint branch).
Verified test_multiprocessing suite passes before (apart from new test) after
change. (Tested under Ubuntu 9.10 64-bit)
--
Added file:
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
Removed file: http://bugs.python.org/file16415/pickle_suggestion_v2.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
Updated patch to include new test (now against release26-maint branch).
Verified test_multiprocessing suite passes before (apart from new test) after
change. (Tested under Ubuntu 9.10 64-bit)
--
Added file:
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
Removed file: http://bugs.python.org/file16472/queues.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
Added file: http://bugs.python.org/file16475/queues.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
I couldn't see the wood for the trees:
If put() is thread-blocking, multiprocessing.Queue is reduced to the same
functionality as multiprocessing.queues.SimpleQueue, if I'm not mistaken. So
maybe there should be a warning in the
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
Removed file: http://bugs.python.org/file16412/lock_suggestion.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
Removed file: http://bugs.python.org/file16409/queue_example.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
Added file: http://bugs.python.org/file16413/queue_example.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
Alternative suggestion (since SimpleQueue doesn't provide buffering):
Allow option to force immediate pickling of the object to be en-queued, i.e.
pickling when adding to internal buffer instead of as part of Connection.send()
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
Performance comparison before/after suggested change (adding/removing 1000-item
list 1 times).
Inline
1.926 class 'multiprocessing.queues.Queue'
1.919 class 'queues_pickled.Queue'
1.907 class 'queues_pickled.Queue'
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
Removed file: http://bugs.python.org/file16414/pickle_suggestion.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
Changes by Vilnis Termanis vilnis.terma...@googlemail.com:
Added file: http://bugs.python.org/file16416/queue_perf.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue8037
___
New submission from Vilnis Termanis vilnis.terma...@googlemail.com:
If an object, which as been put() in the multiprocessing.Queue is changed
immediately after the put() call then changed version may be added to the queue
which I assume is not the expected behaviour:
from multiprocessing
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
More notes:
- Reason for non-atmoic behaviour is due to queue _feeder thread sending the
object after the put() method has already exited. Hence changing the object
straight after put() can result in the modified object being
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
Quick though (last one I promise):
Maybe put() could have an argument to indicate whether to be thread-blocking
(i.e. guaranteeing the object to be sent in its current state) or not. The
(current) non-thread-blocking mode will
Vilnis Termanis vilnis.terma...@googlemail.com added the comment:
Hi Meador,
I apologise: I really shouldn't have called it a patch. It's just to show one
way that I've found fixes the problem (but presumably reduces performance in
other cases, hence the suggestion wait for feedback on the
27 matches
Mail list logo
