[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Davin Potts added the comment: Apologies, one of the quotes in my previous response should have been attributed to @mental. I think @pierreglaser phrased it very nicely: > shared_memory is a low level python module. Precautions should be made when > handling concurrently the shared_memory ob

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Davin Potts added the comment: Short responses to questions/comments from @bjs, followed by hopefully helpful further comments: > Are you supposed to ever use a raw SharedMemory buffer directly? Yes. > What atomicity guarantees are there for ShareableList operations and > read/write to th

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Ben added the comment: It would be nice to get davin to clarify the API for this module. What are the use cases for SharedMemory and ShareableList? Are you supposed to ever use a raw SharedMemory buffer directly? What atomicity guarantees are there for ShareableList operations and read/write

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Pierre Glaser added the comment: Sure, although I won't be able to merge it. Make sure you ping a core-dev such as pitrou or davin :-) -- ___ Python tracker ___ _

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

mental added the comment: > The root of the error is that struct.pack_into starts by memsetting the > underlying memory area with NULL bytes before filling the data with memcpy. I've had a fix spinning for about a day now, it introduced a `multiprocessing.Lock` and it was simply wrapped arou

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Pierre Glaser added the comment: The root of the error is that struct.pack_into starts by memsetting the underlying memory area with NULL bytes before filling the data with memcpy. If ShareableList._get_packing_format is called between the two operations (through a concurrent __getitem__ cal

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

mental added the comment: Interestingly enough the race only seems to occur when the mutator resets the index with an identical value (try replacing the setitem value with: `not X[0]` or populate from `[b'hi']` and with the mutator code use `X[0][::1]`. I can't help but think to implement an

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

mental added the comment: I have been able to reproduce the issue. I'm currently investigating to see what exactly inside the source of ShareableList is causing what appears to be miss synchronization. Currently I think the issue is the documentation failing to mention that ShareableList do

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Change by Antoine Pitrou : -- nosy: +pierreglaser ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

Change by Karthikeyan Singaravelan : -- nosy: +davin, pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37652] Multiprocessing shared_memory ValueError on race with ShareableList

New submission from Ben : When running the attached on 3.8 and 3.9 (master) I get the following: Process Process-3: Traceback (most recent call last): File "/home/bjs/.pyen