[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access

Change by Vinay Sharma : -- pull_requests: +22087 pull_request: https://github.com/python/cpython/pull/23174 ___ Python tracker <https://bugs.python.org/issue38

[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

Change by Vinay Sharma : -- keywords: +patch pull_requests: +21002 stage: -> patch review pull_request: https://github.com/python/cpython/pull/21877 ___ Python tracker <https://bugs.python.org/issu

[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

Vinay Sharma added the comment: I have 8GB of ram and 128 GB of hard disk. Now, creating a shared memory segment of size 10^12 (1 terabyte) somehow succeeds. Creating a shared memory segment of 10^15 (1 petabyte), mmap (not ftruncate) throws an error stating cannot allocate memory

[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access

Vinay Sharma added the comment: As suggested by Guido I have floated this solution to python-dev mailing list. Link to archive: https://mail.python.org/archives/list/python-...@python.org/thread/O67CR7QWEOJ7WDAJEBKSY74NQ2C4W3AI/ -- ___ Python

[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access

Vinay Sharma added the comment: Well, the chances of resource tracker dying abruptly are very less because it's thoroughly tested, and there are mechanisms to re-spawn resource_tracker process if you see the code. There is a function called `def ensure_running`. Resource tracker is still

[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access

Vinay Sharma added the comment: That's a valid point Guido. But, I guess this can be easily handled by resource tracker. At this current moment resource tracker unlinks shared memory if the process which created it dies without unliking it. Therefore, resource tracker handles cleaning up

[issue41344] SharedMemory crash when size is 0

Vinay Sharma added the comment: Also, Linux created the shared memory with the passed name if size = 0, but threw error when it was being mapped to the process's virtual memory. Therefore, this scenario should be prevented before actually creating the shared memory segment in cases of Linux

[issue41344] SharedMemory crash when size is 0

Vinay Sharma added the comment: Hi, The patch aims to raise a value error, because before the patch also, ValueError was being raised in case size was negative. That's why I thought it would be correct behaviour if I raise ValueError in case size is 0. Do you mean to say, that OSError

[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

Change by Vinay Sharma : -- nosy: +pitrou ___ Python tracker <https://bugs.python.org/issue39584> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue38018] Increase Code Coverage for multiprocessing.shared_memory

Vinay Sharma added the comment: Closing this, as all the necessary PRs have been merged. -- stage: patch review -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/i

[issue41344] SharedMemory crash when size is 0

Change by Vinay Sharma : -- keywords: +patch pull_requests: +20702 stage: -> patch review pull_request: https://github.com/python/cpython/pull/21556 ___ Python tracker <https://bugs.python.org/issu

[issue41344] SharedMemory crash when size is 0

New submission from Vinay Sharma : On running this: shm = SharedMemory(create=True, size=0) I get the following error: Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.8/multiprocessing/shared_memory.py", line 111, in __init__ se

[issue38169] Increase Code Coverage for SharedMemory and ShareableListe

Vinay Sharma added the comment: Remove Check to prevent creating shared_memory with size 0. Will create a different issue for the same. -- ___ Python tracker <https://bugs.python.org/issue38

[issue40828] shared memory problems with multiprocessing.Pool

Vinay Sharma added the comment: Hi, Can you please confirm the python version in which you replicated this issue. I tried replicating (segmentation fault) this issue in python 3.8.0b3 but didn't succeed. And shared_memory isn't present in 3.7. Also, shared_memory is known to throw some

[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

Vinay Sharma added the comment: Hi, I tried replicating this by truncating normal files but that doesn't crash. The above mentioned call of ftruncate only crashes for when the file descriptor passed points to a shared memory segment. And only, multiprocessing.shared_memory is currently

[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access

Change by Vinay Sharma : -- pull_requests: +20652 pull_request: https://github.com/python/cpython/pull/21516 ___ Python tracker <https://bugs.python.org/issue38

[issue39959] Bug on multiprocessing.shared_memory

Vinay Sharma added the comment: @rauanargyn , persist flag won't be good idea because it cannot be supported for windows easily, since windows uses a reference counting mechanism to keep track of shared memory and frees it as soon as all the processes using it are done

[issue39959] Bug on multiprocessing.shared_memory

Vinay Sharma added the comment: Hi, shared_memory has lot of issues which are mainly being caused due to resource tracking. Initially resource tracking was implemented to keep track of semaphores only, but for some reason resource tracker also started to keep track of shared_memory

[issue38035] shared_semaphores cannot be shared across unrelated processes

Vinay Sharma added the comment: Hi @taleinat, I had sent a mail to the python mailing list as suggested. And till now it has received rather positive comments, in the sense that people who responded see this is a useful feature to be integrated into python

[issue38035] shared_semaphores cannot be shared across unrelated processes

Vinay Sharma added the comment: As suggested I have sent a mail to Python Ideas regarding this issue. Link to Python Ideas Archive: https://mail.python.org/archives/list/python-id...@python.org/thread/X4AKFFMYEKW6GFOUMXMOJ2OBINNY2Q6L

[issue38035] shared_semaphores cannot be shared across unrelated processes

Vinay Sharma added the comment: Hi, I think a use case for this is https://docs.python.org/3/library/multiprocessing.shared_memory.html If not, can you please suggest a way to synchronise the above across unrelated processes. -- ___ Python

[issue39584] MacOS crashes by running attached Python code

New submission from Vinay Sharma : Consider the following python Code. ``` from multiprocessing.shared_memory import SharedMemory shm = SharedMemory(name='test-crash', create=True, size=100) ``` This causes macOS Catalina, Mojave to freeze and then crash. Although, this works

[issue39584] MacOS crashes by running attached Python code

Change by Vinay Sharma : -- type: -> crash ___ Python tracker <https://bugs.python.org/issue39584> ___ ___ Python-bugs-list mailing list Unsubscrib

[issue38169] Increase Code Coverage for SharedMemory and ShareableListe

Change by Vinay Sharma : -- keywords: +patch pull_requests: +15749 stage: -> patch review pull_request: https://github.com/python/cpython/pull/16139 ___ Python tracker <https://bugs.python.org/issu

[issue38169] Increase Code Coverage for SharedMemory and ShareableListe

New submission from Vinay Sharma : Add Tests for SharedMemory and ShareableList. I have also added a check to prevent users from creating shared memory of size 0, because after creating mmap will throw error while memory mapping she of size 0. -- components: Tests messages: 352420

[issue38035] shared_semaphores cannot be shared across unrelated processes

Vinay Sharma added the comment: A common use for the same can be shared memory. Currently shared memory can be used by unrelated processes, but there is no mechanism as such to synchronise them at the moment. -- ___ Python tracker <ht

[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access

Vinay Sharma added the comment: Hi Davin, This PR would fix the issues mentioned by you, by not prematurely unlinking the shared memory segment. And, therefore it would make shared memory useful in a lot of use cases. But, this would still not make Unix's implementation consistent

[issue38018] Increase Code Coverage for multiprocessing.shared_memory

Vinay Sharma added the comment: Also, I haven't made a NEWS entry since it's just a short bug fix -- ___ Python tracker <https://bugs.python.org/issue38

[issue38018] Increase Code Coverage for multiprocessing.shared_memory

Vinay Sharma added the comment: Hi I have opened another PR: https://github.com/python/cpython/pull/15821 This should fix test failures in FreeBSD. FreeBSD requires a leading slash in shared memory names. That's why it was throwing the below error

[issue38018] Increase Code Coverage for multiprocessing.shared_memory

Change by Vinay Sharma : -- pull_requests: +15469 pull_request: https://github.com/python/cpython/pull/15821 ___ Python tracker <https://bugs.python.org/issue38

[issue37754] Consistency of Unix's shared_memory implementation with windows

Vinay Sharma added the comment: Hi @davin, I researched on lots of approaches to solve this problem, and I have listed down some of the best ones. 1. As Eryk Sun suggested initially to use advisory locking to implement a reference count type of mechanism. I implemented this in the current

[issue38035] shared_semaphores cannot be shared across unrelated processes

Vinay Sharma added the comment: Let's say I have two processes which are accessing a shared resource (let's say shared memory). Now, these two processes will need some kind of synchronisation mechanism, to be able to access the resource properly, and avoid race condition. Currently, shared

[issue38035] shared_semaphores cannot be shared across unrelated processes

New submission from Vinay Sharma : Currently, shared semaphores can only be created, and existing semaphores can't be opened. Shared semaphores are opened using the following command. ``` sem_open(name, O_CREAT | O_EXCL, 0600, val) ``` This will raise error if a semaphore which already exists

[issue38035] shared_semaphores cannot be shared across unrelated processes

Change by Vinay Sharma : -- type: -> enhancement ___ Python tracker <https://bugs.python.org/issue38035> ___ ___ Python-bugs-list mailing list Unsubscrib

[issue38018] Increase Code Coverage for multiprocessing.shared_memory

Vinay Sharma added the comment: Can anyone please review my pull request. No Reviewer has been assigned yet to it. And I don't have permissions to request a review. -- ___ Python tracker <https://bugs.python.org/issue38

[issue38018] Increase Code Coverage for multiprocessing.shared_memory

Change by Vinay Sharma : -- keywords: +patch pull_requests: +15327 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15662 ___ Python tracker <https://bugs.python.org/issu

[issue38018] Increase Code Coverage for multiprocessing.shared_memory

New submission from Vinay Sharma : Increase Code coverage for multiprocessing.shared_memory.SharedMemory Class -- components: Tests messages: 351085 nosy: vinay0410 priority: normal severity: normal status: open title: Increase Code Coverage for multiprocessing.shared_memory versions

[issue37964] F_GETPATH is not available in fcntl.fcntl

Vinay Sharma added the comment: I have opened a PR, but no reviewers, have been assigned. Could you please look into that ? -- ___ Python tracker <https://bugs.python.org/issue37

[issue37964] F_GETPATH is not available in fcntl.fcntl

Change by Vinay Sharma : -- keywords: +patch pull_requests: +15226 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/15550 ___ Python tracker <https://bugs.python.org/issu

[issue37964] F_GETPATH is not available in fcntl.fcntl

New submission from Vinay Sharma : F_GETPATH cmd/operator is not present in fcntl module. This is specific to macos and is only available in macos. https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fcntl.2.html This can be also be verified

[issue37754] Consistency of Unix's shared_memory implementation with windows

Vinay Sharma added the comment: Since, advisory locking doesn't work on integer file descriptors which are returned by shm_open on macos, I was thinking of an alternative way of fixing this. I was thinking of using a shared semaphore, which will store the reference count of the processes

[issue37754] Consistency of Unix's shared_memory implementation with windows

Vinay Sharma added the comment: Also, shm_open returns an integer file descriptor. And when this file descriptor is passed too fcntl.flock (in macOS) it throws the following error: OSError: [Errno 45] Operation not supported Whereas, the same code works fine on linux. Therefore, I have

[issue37754] Consistency of Unix's shared_memory implementation with windows

Vinay Sharma added the comment: Hi, I just opened a PR implementing a fix very similar to your suggestions. I am using advisory locking using fcntl.flock. And I am locking on file descriptors. If you see my PR, in resource tracker I am opening a file "/dev/shm/", and trying

[issue37754] Persistence of Shared Memory Segment after process exits

Change by Vinay Sharma : -- keywords: +patch pull_requests: +15154 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15460 ___ Python tracker <https://bugs.python.org/issu

[issue37754] Persistence of Shared Memory Segment after process exits

Change by Vinay Sharma : -- versions: +Python 3.9 ___ Python tracker <https://bugs.python.org/issue37754> ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37754] Persistence of Shared Memory Segment after process exits

Vinay Sharma added the comment: > In terms of providing "consistent behavior across platforms that can be > reasonably supported", the behavior suggested above could not > reasonably be supported in Windows. I understand that persistence of a shared memory segment after

[issue37754] Persistence of Shared Memory Segment after process exits

Change by Vinay Sharma : -- type: enhancement -> behavior ___ Python tracker <https://bugs.python.org/issue37754> ___ ___ Python-bugs-list mailing list Un

[issue37754] Persistence of Shared Memory Segment after process exits

Vinay Sharma added the comment: Hi Davin, Thanks for replying! As you said I went through the issue, and now understand why segments should not be automatically created if they don't exist. But, after reading that thread I got to know that shared memory is supposed to exist even

[issue37754] Persistence of Shared Memory Segment after process exits

Change by Vinay Sharma : -- title: alter size of segment using multiprocessing.shared_memory -> Persistence of Shared Memory Segment after process exits ___ Python tracker <https://bugs.python.org/issu

[issue37703] Inconsistent gather with child exception

Vinay Sharma added the comment: Hi Andrew, Thanks for replying! I understand that the behavior change is not backward compatible. But the current behavior is a bit ambiguous ad mentioned by Dimitri, therefore I have updated the documentation and opened a Pull request. I would be very glad

[issue37703] Inconsistent gather with child exception

Change by Vinay Sharma : -- pull_requests: +15031 stage: -> patch review pull_request: https://github.com/python/cpython/pull/15312 ___ Python tracker <https://bugs.python.org/issu

[issue37703] Inconsistent gather with child exception

Change by Vinay Sharma : Added file: https://bugs.python.org/file48548/gather_cancel_code.patch ___ Python tracker <https://bugs.python.org/issue37703> ___ ___ Python-bug

[issue37703] Inconsistent gather with child exception

Vinay Sharma added the comment: Hi Dimitri, You are right, gather.cancel() doesn't work once it has propagated an exception. This happens because after propagating the exception to the caller, gather is marked as done. And cancel doesn't work after a Future object has been marked done. You

[issue37754] alter size of segment using multiprocessing.shared_memory

New submission from Vinay Sharma : Hi, I am opening this to discuss about some possible enhancements in the multiprocessing.shared_memory module. I have recently started using multiprocessing.shared_memory and realised that currently the module provides no functionality to alter the size

[issue37185] use os.memfd_create in multiprocessing.shared_memory?

Vinay Sharma added the comment: Hi @pierreglaser, I recently started using the shared_memory module in multiprocessing , and as you said using memfd_create wouldn't require resource tracking is true. But, I was wondering if these memory segments can't be related/mapped using a unique name