[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-11-17 Thread Ronald Oussoren

Ronald Oussoren  added the comment:

I’ve filed an issue with Apple about this: FB8903019

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-11-17 Thread Ronald Oussoren

Ronald Oussoren  added the comment:

Having thought about this a little more: I agree, we shouldn’t hard code a 
limit.  I’m not against working around misfeatures, but in this case that’s 
hard to do: Even if we’d limit the size of a single shared memory segment the 
user can create a number of them to once again use enough memory to crash the 
machine.

IMHO We should document that this can happen on macOS (preferable with some 
indication of how much shared memory can safely be used). I’ve therefore added 
the “Documentation” component.

--
assignee:  -> docs@python
components: +Documentation
nosy: +docs@python

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-11-17 Thread Christian Heimes


Christian Heimes  added the comment:

I'm strong -1 on any kind of arbitrary memory limit. Python is used on all 
sorts of hardware from RPi to super computers. 1 TB sounds like a lot, but it 
really is not. These days you can buy workstation computers with more than 1 TB 
RAM.

I would be ok with a limit of 2**48 on X86_64 platforms. Current X86_64 CPUs 
have a virtual address size limit of 48 bits. Other hardware platforms may have 
a similar limit. I don't have access to AArch64 or PPC64 to verify the address 
size.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-08-14 Thread Vinay Sharma


Change by Vinay Sharma :


--
keywords: +patch
pull_requests: +21002
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/21877

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-08-12 Thread Ronald Oussoren


Ronald Oussoren  added the comment:

A workaround in the implementation of multiprocessing.SharedMemory is IMHO 
acceptable, tweaking os.ftruncate less so.

Note that a Python script can already cause problems on systems by using APIs 
as intended (such as using shutil.rmtree on the user's home directory).

There's balance between compensating for platform deviancies and having clean 
and performant implementation. 

BTW. We should file an issue with Apple about this. I'll do so when I've had 
time to crash a test VM using the C code you provided.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-08-12 Thread Vinay Sharma


Vinay Sharma  added the comment:

I have 8GB of ram and 128 GB of hard disk.

Now, creating a shared memory segment of size 10^12 (1 terabyte) somehow 
succeeds.

Creating a shared memory segment of 10^15 (1 petabyte), mmap (not ftruncate) 
throws an error stating cannot allocate memory.

Creating a shared memory segment of 10^18 (1 exabyte), causes the system to 
crash.

Now, I understand that this should be documented for a genuine user.

But, if documented this can be used by malicious softwares using python to 
crash systems abruptly.

Also, I understand that this is an issue with macos, but shouldn't python 
handle this so that atleast python's APIs are safe.

Creating shared memory segments of size 1 exabyte are not reasonable, but if 
some makes a mistake then, we must throw an error instead of a crash.

Also, can we set a max limit on creating shared memory segments to 1TB ?
because no one would genuinily need to create a segment of that size on a 
single machine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-08-12 Thread Ronald Oussoren


Ronald Oussoren  added the comment:

I expect that this is more a problem with how memory management works on macOS, 
the freeze and crash is likely an indication of eager allocation of memory by 
the system.

It is far from sure if implementing a workaround is feasible, the upper limit 
for safe calls to ftruncate likely depends on the amount of RAM in the system.

BTW. The sample code tries to allocate well over a petabyte of memory, how 
realistic is that code?  It might be good enough to document that using very 
large amounts of memory with multiprocessing.shared_memory causes problems on 
macOS.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-08-12 Thread Ned Deily


Change by Ned Deily :


--
components: +macOS
nosy: +ned.deily, ronaldoussoren

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-07-20 Thread Vinay Sharma


Change by Vinay Sharma :


--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-07-17 Thread Vinay Sharma


Vinay Sharma  added the comment:

Hi, I tried replicating this by truncating normal files but that doesn't crash. 
The above mentioned call of ftruncate only crashes for when the file descriptor 
passed points to a shared memory segment.
And only, multiprocessing.shared_memory is currently creating shared_memory 
using _posixshmem.shm_open.

So, it can be fixed in ftruncate implementation or this can also be handled by 
multiprocessing.shared_memory.

Please let me know your thoughts about the same.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-02-12 Thread Davin Potts


Davin Potts  added the comment:

My sense is that it would be nice if we can catch this before ftruncate does 
something nasty.

Where else is ftruncate used in CPython that this could similarly trigger a 
problem?  How is it handled there (or not)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue39584] multiprocessing.shared_memory: MacOS crashes by running attached Python code

2020-02-10 Thread STINNER Victor


Change by STINNER Victor :


--
title: MacOS crashes by running attached Python code -> 
multiprocessing.shared_memory: MacOS crashes by running attached Python code

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com