STINNER Victor added the comment:
It is possible to install a custom hook on memory allocators using the PEP 445:
see for example attached fatalmalloc.c.
I don't see much traction to get this feature since 2013, I reject the feature
request.
--
resolution: -> rejected
stage: patch
STINNER Victor added the comment:
This issue is maybe a new usecase of the PEP 445.
Try attached Python module fatalmalloc.c, use attached setup.py script to build
it.
$ python3.4 -c 'import fatalmalloc; x=x*(50*1024*1024); print(len(x))'
Traceback (most recent call last):
File string, line
Changes by STINNER Victor victor.stin...@gmail.com:
Added file: http://bugs.python.org/file32042/setup.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
Changes by STINNER Victor victor.stin...@gmail.com:
--
nosy: +haypo
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list
Antoine Pitrou added the comment:
If we want to get more sophisticated, I would suggest a two-pronged approach:
- first call a user-defined oom function
- if calling the user-defined oom function raises MemoryError, dump a fatal
error
--
nosy: +pitrou
STINNER Victor added the comment:
See also issue #1195571: simple callback system for Py_FatalError. It is
somehow related.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
Lukas Lueg added the comment:
Another proposal: Add a new BaseClass that, if inherited from, causes an
exception to be uncatchable (e.g. class HardMemoryError(MemoryError,
UncatchableException)).
--
___
Python tracker rep...@bugs.python.org
Christian Theune added the comment:
I feel unsure how to help this move along.
I agree that making it possible for applications to carefully work with
MemoryErrors is a good idea. I don't think heuristics to determine which
situation we are in will solve this but make it more spooky. (This
Changes by Christian Theune c...@gocept.com:
--
nosy: +barry
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing
Changes by Jesús Cea Avión j...@jcea.es:
--
nosy: +jcea
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Changes by Hynek Schlawack h...@ox.cx:
--
nosy: +hynek
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
___
Python-bugs-list mailing list
Christian Theune added the comment:
I pondered PyErr_NoMemory as well. However, I noticed not all locations in
Python use PyErr_NoMemory to raise a MemoryError, and I'm also afraid that
external libraries will have the same problem.
Can you explain why you consider PyErr_NoMemory to be the
Benjamin Peterson added the comment:
Where it is now means raising a MemoryError in Python code will cause the fatal
error. I don't know if that's a good idea.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
Guido van Rossum added the comment:
There used to be some places in the C code that raise MemoryError to
indicate that some parameter indicating a desired result size is out
of range, i.e. before even trying to allocate anything. I can't
confirm any, but these should probably be replaced with
Lukas Lueg added the comment:
I have to say this feels like spooky action at a distance.
Wouldnt it be less intrusive - while achieving the same result - to make
MemoryError uncatchable if the flag is set?
--
nosy: +ebfe
___
Python tracker
Christian Heimes added the comment:
MemoryError can be raised under two different circumstances that should be
handled differently. Either the program tries to allocate a rather large chunk
of memory for e.g. a string with a couple of hundred KB and more. Or Python
can't malloc() even small
Guido van Rossum added the comment:
Well, you can fix that, right? Just add a new function with a better
signature and use that for one of the two scenarios.
(I think third scenario might be when realloc fails -- IIRC it
doesn't guarantee that the original pointer is still valid either?)
Lukas Lueg added the comment:
In any strategy only a heuristic could be used in order to decide wether or not
it's safe to raise MemoryError. How exactly is memory pressure expected for
x=[2]*200 but not for x=2*200 ?
I don't think a new function could ultimatly achieve it's goal. If
Christian Heimes added the comment:
Of course it can be fixed. But it's going to be a long and tedious piece of
work. PyErr_NoMemory() is called 314 times. MemoryError is raised about 40
times in C code.
I'm not sure what your question about realloc() really is. Are you unsure how a
a
Christian Heimes added the comment:
We can create some kind of heuristic when the memory error handler knows the
size of the new block and the size difference on realloc(). It could be an
application specific threshold, too.
--
___
Python tracker
Lukas Lueg added the comment:
The heuristic basically has to decide if memory pressure is so high that it's
not save to return to the interpreter. Even if there is a chosen value (e.g.
failed allocation attempts below 1mb are considered fatal), there can always be
another OS-thread in the
New submission from Christian Theune:
I run long-running server processes (web apps, etc) a lot and I keep
encountering the situation that many applications will not properly deal with
MemoryError exceptions but end up in an unusable state.
From an operational perspective I wish the process
Changes by Christian Theune c...@gocept.com:
--
keywords: +patch
Added file: http://bugs.python.org/file27821/9430a5c65114.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
Christian Heimes added the comment:
Your proposal sounds like a very good idea. IMHO it should be discussed on the
python-ideas or python-dev mailing list before it gets integrated into 3.4.
Embrace yourself for some serious bike shedding! :)
By the way your patch contains several changes
Christian Theune added the comment:
Grr. Sorry. The automatic patch extraction went wrong and I didn't notice.
Here's a manual try.
--
Added file: http://bugs.python.org/file27822/issue16381.diff
___
Python tracker rep...@bugs.python.org
Changes by Christian Theune c...@gocept.com:
Removed file: http://bugs.python.org/file27821/9430a5c65114.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
Christian Heimes added the comment:
Thanks!
Py_FatalError() might be too drastic for the task. It calls abort() which kills
the process with SIGABRT. The function closes and flushes all stream but no
additional cleanup code is executed. This might be bad for resources like
shared memories,
Benjamin Peterson added the comment:
I think the fatal erroring should be done in PyErr_NoMemory.
--
nosy: +benjamin.peterson
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16381
___
28 matches
Mail list logo
