Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[eryk sun]

On Thu, Oct 19, 2017 at 9:05 AM, Stephan Houben  wrote:
>
> I (quickly) tried to get something to work using the win32 package,
> in particular the win32job functions.
> However, it seems setting
> "ProcessMemoryLimit" using win32job.SetInformationJobObject
> had no effect
> (i.e.  a subsequent win32job.QueryInformationJobObject
> still showed the limit as 0)?

Probably you didn't set the JOB_OBJECT_LIMIT_PROCESS_MEMORY flag.
Here's an example that tests the process memory limit using ctypes to
call VirtualAlloc, before and after assigning the current process to
the Job.

Note that the py.exe launcher runs python.exe in an anonymous Job
that's configured to kill on close (i.e. python.exe is killed when
py.exe exits) and for silent breakaway of child processes. In this
case, prior to Windows 8 (the first version to support nested Job
objects), assigning the current process to a new Job will fail, so
you'll have to run python.exe directly, or use a child process via
subprocess. I prefer the former, since a child process won't be
tethered to the launcher, which could get ugly for console
applications.

import ctypes
import winerror, win32api, win32job

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

MEM_COMMIT = 0x1000
MEM_RELEASE = 0x8000
PAGE_READWRITE = 4

kernel32.VirtualAlloc.restype = ctypes.c_void_p
kernel32.VirtualAlloc.argtypes = (ctypes.c_void_p, ctypes.c_size_t,
ctypes.c_ulong, ctypes.c_ulong)
kernel32.VirtualFree.argtypes = (ctypes.c_void_p, ctypes.c_size_t,
ctypes.c_ulong)

hjob = win32job.CreateJobObject(None, "")

limits = win32job.QueryInformationJobObject(hjob,
win32job.JobObjectExtendedLimitInformation)
limits['BasicLimitInformation']['LimitFlags'] |= (
win32job.JOB_OBJECT_LIMIT_PROCESS_MEMORY)
limits['ProcessMemoryLimit'] = 2**31
win32job.SetInformationJobObject(hjob,
win32job.JobObjectExtendedLimitInformation, limits)

addr0 = kernel32.VirtualAlloc(None, 2**31, MEM_COMMIT,
PAGE_READWRITE)
if addr0:
mem0_released = kernel32.VirtualFree(addr0, 0, MEM_RELEASE)

win32job.AssignProcessToJobObject(hjob,
win32api.GetCurrentProcess())

addr1 = kernel32.VirtualAlloc(None, 2**31, MEM_COMMIT,
PAGE_READWRITE)

Result:

>>> addr0
2508252315648
>>> mem0_released
1
>>> addr1 is None
True
>>> ctypes.get_last_error() == winerror.ERROR_COMMITMENT_LIMIT
True
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Giampaolo Rodola']

On Thu, Oct 19, 2017 at 10:05 AM, Stephan Houben 
wrote:

> Hi Steve,
>
> 2017-10-19 1:59 GMT+02:00 Steven D'Aprano :
>
>> On Wed, Oct 18, 2017 at 02:51:37PM +0200, Stefan Krah wrote:
>>
>> > $ softlimit -m 10 python3
>> [...]
>> > MemoryError
>> >
>> >
>> > People who are worried could make a python3 alias or use Ctrl-\.
>>
>> I just tried that on two different Linux computers I have, and neither
>> have softlimit.
>>
>>
> Yeah, not sure what "softlimit" is either.
> I'd suggest sticking to POSIX-standard ulimit or just stick
> something like this in the .pythonrc.py:
>
> import resource
> resource.setrlimit(resource.RLIMIT_DATA, (2 * 1024**3, 2 * 1024**3))
>
> Nor (presumably) would this help Windows users.
>>
>
> I (quickly) tried to get something to work using the win32 package,
> in particular the win32job functions.
> However, it seems setting
> "ProcessMemoryLimit" using win32job.SetInformationJobObject
> had no effect
> (i.e.  a subsequent win32job.QueryInformationJobObject
> still showed the limit as 0)?
>
> People with stronger Windows-fu may be aware what is going on here...
>
> Stephan
>

I wasn't aware Windows was capable of setting such limits in a per-process
fashion. You gave me a good idea for psutil:
https://github.com/giampaolo/psutil/issues/1149
According to this cmdline tool:
https://cr.yp.to/daemontools/softlimit.html
the limit should kick in only when the system memory is full, whatever
that means:
<<-r n: Limit the resident set size to n bytes. This limit is not enforced
unless physical memory is full.>>
...so that would explain why it had no effect.

-- 
Giampaolo - http://grodola.blogspot.com
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Stefan Krah]

On Thu, Oct 19, 2017 at 10:05:58AM +0200, Stephan Houben wrote:
> 
> 2017-10-19 1:59 GMT+02:00 Steven D'Aprano :
> 
> > On Wed, Oct 18, 2017 at 02:51:37PM +0200, Stefan Krah wrote:
> >
> > > $ softlimit -m 10 python3
> > [...]
> > > MemoryError
> > >
> > >
> > > People who are worried could make a python3 alias or use Ctrl-\.
> >
> > I just tried that on two different Linux computers I have, and neither
> > have softlimit.
> >
> >
> Yeah, not sure what "softlimit" is either.

Part of daemontools: https://cr.yp.to/daemontools/softlimit.html


Stefan Krah



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Stephan Houben]

Hi Steve,

2017-10-19 1:59 GMT+02:00 Steven D'Aprano :

> On Wed, Oct 18, 2017 at 02:51:37PM +0200, Stefan Krah wrote:
>
> > $ softlimit -m 10 python3
> [...]
> > MemoryError
> >
> >
> > People who are worried could make a python3 alias or use Ctrl-\.
>
> I just tried that on two different Linux computers I have, and neither
> have softlimit.
>
>
Yeah, not sure what "softlimit" is either.
I'd suggest sticking to POSIX-standard ulimit or just stick
something like this in the .pythonrc.py:

import resource
resource.setrlimit(resource.RLIMIT_DATA, (2 * 1024**3, 2 * 1024**3))

Nor (presumably) would this help Windows users.
>

I (quickly) tried to get something to work using the win32 package,
in particular the win32job functions.
However, it seems setting
"ProcessMemoryLimit" using win32job.SetInformationJobObject
had no effect
(i.e.  a subsequent win32job.QueryInformationJobObject
still showed the limit as 0)?

People with stronger Windows-fu may be aware what is going on here...

Stephan


>
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Steven D'Aprano]

On Wed, Oct 18, 2017 at 02:51:37PM +0200, Stefan Krah wrote:

> $ softlimit -m 10 python3
[...]
> MemoryError
> 
> 
> People who are worried could make a python3 alias or use Ctrl-\.

I just tried that on two different Linux computers I have, and neither 
have softlimit. 

Nor (presumably) would this help Windows users.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Nick Coghlan]

On 18 October 2017 at 22:51, Stefan Krah  wrote:

> On Wed, Oct 18, 2017 at 10:43:57PM +1000, Nick Coghlan wrote:
> > Per-process memory quotas *can* help avoid this, but enforcing them
> > requires that every process run in a resource controlled sandbox. Hence,
> > it's not a coincidence that mobile operating systems and container-based
> > server environments already work that way, and the improved ability to
> cope
> > with misbehaving applications is part of why desktop operating systems
> > would like to follow the lead of their mobile and server counterparts :)
>
> Does this also fall under the sandbox definition?
>
> $ softlimit -m 10 python3
>

Yeah, Linux offers good opt-in tools for this kind of thing, and the
combination of Android and containerised server environments means they're
only getting better. But we're still some time away from it being routine
for your desktop to be well protected from memory management misbehaviour
in arbitrary console or GUI applications.

The resource module (which Steven mentioned in passing) already provides
opt-in access to some of those features from within the program itself:
https://docs.python.org/3/library/resource.html

For example:

>>> import sys, resource
>>> data = bytes(2**32)
>>> resource.setrlimit(resource.RLIMIT_DATA, (2**31, sys.maxsize))
>>> data = bytes(2**32)
Traceback (most recent call last):
  File "", line 1, in 
MemoryError
>>> resource.setrlimit(resource.RLIMIT_DATA, (sys.maxsize, sys.maxsize))
>>> data = bytes(2**32)

(Bulk memory allocations start failing on my machine somewhere between
2**33 and 2**34, which is about what I'd expect, since it has 8 GiB of
physical RAM installed)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Koos Zevenhoven]

On Wed, Oct 18, 2017 at 2:38 PM, Steven D'Aprano 
wrote:

> On Wed, Oct 18, 2017 at 01:39:28PM +0300, Koos Zevenhoven wrote:
>
> > I'm writing from my phone now, cause I was dumb enough to try
> list(count())
>
> You have my sympathies -- I once, due to typo, accidentally ran
> something like range(10**100) in Python 2.
>
>
​Oh, I think I've done something like that too, and there are definitely
still opportunities in Python 3 to ask for the impossible. But what I did
now, I did "on purpose".​ For a split second, I really wanted to know how
bad it would be. But a few minutes later I had little interest left in that
;). Rebooting a computer definitely takes longer than restarting a Python
process.


>
> > But should it be fixed in list or in count?
>
> Neither. There are too many other places this can break for it to be
> effective to try to fix each one in place.
>
>
​To clarify, I was talking about allowing Ctrl-C to break it, which
somebody had suggested. That would also help if the C-implemented iterable
just takes a lot of time to generate the items.

​And for the record, I just tried

>>> sum(itertools.count())

And as we could expect, it does not respect Ctrl-C either.


––Koos



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Stefan Krah]

On Wed, Oct 18, 2017 at 10:43:57PM +1000, Nick Coghlan wrote:
> Per-process memory quotas *can* help avoid this, but enforcing them
> requires that every process run in a resource controlled sandbox. Hence,
> it's not a coincidence that mobile operating systems and container-based
> server environments already work that way, and the improved ability to cope
> with misbehaving applications is part of why desktop operating systems
> would like to follow the lead of their mobile and server counterparts :)

Does this also fall under the sandbox definition?

$ softlimit -m 10 python3
Python 3.7.0a1+ (heads/master:bdaeb7d237, Oct 16 2017, 18:54:55) 
[GCC 4.8.5] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> [0] * 10
Traceback (most recent call last):
  File "", line 1, in 
MemoryError


People who are worried could make a python3 alias or use Ctrl-\.



Stefan Krah



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Nick Coghlan]

On 18 October 2017 at 21:38, Steven D'Aprano  wrote:

> > But should it be fixed in list or in count?
>
> Neither. There are too many other places this can break for it to be
> effective to try to fix each one in place.
>

> e.g. set(xrange(2**64)), or tuple(itertools.repeat([1]))
>

A great many of these call operator.length_hint() these days in order to
make a better guess as to how much memory to pre-allocate, so while that
still wouldn't intercept everything, it would catch a lot of them.


> Rather, I think we should set a memory limit that applies to the whole
> process. Once you try to allocate more memory, you get an MemoryError
> exception rather than have the OS thrash forever trying to allocate a
> terabyte on a 4GB machine.
>
> (I don't actually understand why the OS can't fix this.)
>

Trying to allocate enormous amounts of memory all at once isn't the
problem, as that just fails outright with "Not enough memory":

>>> data = bytes(2**62)
Traceback (most recent call last):
  File "", line 1, in 
MemoryError

The machine-killing case is repeated allocation requests that the operating
system *can* satisfy, but require paging almost everything else out of RAM.
And that's exactly what "list(infinite_iterator)" entails, since the
interpreter will make an initial guess as to the correct size, and then
keep resizing the allocation to 125% of its previous size each time it
fills up (or so - I didn't check the current overallocation factor) .

Per-process memory quotas *can* help avoid this, but enforcing them
requires that every process run in a resource controlled sandbox. Hence,
it's not a coincidence that mobile operating systems and container-based
server environments already work that way, and the improved ability to cope
with misbehaving applications is part of why desktop operating systems
would like to follow the lead of their mobile and server counterparts :)

So here is my suggestion:
>
> 1. Let's add a function in sys to set the "maximum memory" available,
> for some definition of memory that makes the most sense on your
> platform. Ordinary Python programmers shouldn't have to try to decipher
> the ulimit interface.
>

Historically, one key reason we didn't do that was because the `PyMem_*`
APIs bypassed CPython's memory allocator, so such a limit wouldn't have
been particularly effective.

As of 3.6 though, even bulk memory allocations pass through pymalloc,
making a Python level memory allocation limit potentially more viable
(since it would pick up almost all of the interpeter's own allocations,
even if it missed those in extension modules):
https://docs.python.org/dev/whatsnew/3.6.html#optimizations

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Steven D'Aprano]

On Wed, Oct 18, 2017 at 01:39:28PM +0300, Koos Zevenhoven wrote:

> I'm writing from my phone now, cause I was dumb enough to try list(count())

You have my sympathies -- I once, due to typo, accidentally ran 
something like range(10**100) in Python 2.

 
> But should it be fixed in list or in count?

Neither. There are too many other places this can break for it to be 
effective to try to fix each one in place.

e.g. set(xrange(2**64)), or tuple(itertools.repeat([1]))

Rather, I think we should set a memory limit that applies to the whole 
process. Once you try to allocate more memory, you get an MemoryError 
exception rather than have the OS thrash forever trying to allocate a 
terabyte on a 4GB machine.

(I don't actually understand why the OS can't fix this.)

Being able to limit memory use is a fairly common request, e.g. on 
Stackoverflow:

https://stackoverflow.com/questions/30269238/limit-memory-usage
https://stackoverflow.com/questions/2308091/how-to-limit-the-heap-size
https://community.webfaction.com/questions/15367/setting-max-memory-for-python-script

And Java apparently has a commandline switch to manage memory:

https://stackoverflow.com/questions/22887400/is-there-an-equivalent-to-java-xmx-for-python

The problems with the resources module are that its effectively an 
interface to ulimit, which makes it confusing and platform-specific; it 
is no help to Windows users; it isn't used by default; and not many 
people know about it. (I know I didn't until tonight.)
 
So here is my suggestion:

1. Let's add a function in sys to set the "maximum memory" available, 
for some definition of memory that makes the most sense on your 
platform. Ordinary Python programmers shouldn't have to try to decipher 
the ulimit interface.

2. Have a command line switch to set that value, e.g.:

python3 -L 1073741824  # 1 GiB
python3 -L 0  # machine-dependent limit
python3 -L -1  # unlimited

where the machine-dependent limit is set by the interpreter, depending 
on the amount of memory it thinks it has available.

3. For the moment, stick to defaulting to -L -1 "unlimited", but with 
the intention to change to -L 0 "let the interpreter decide" in some 
future release, after an appropriate transition period.

On Linux, we can always run

   ulimit  python3 

but honestly I never know which option to give (maximum stack size? 
maximum virtual memory? why is there no setting for maximum real 
memory?) and that doesn't help Windows users.


Thoughts?




-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/