Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

2016-02-08 15:18 GMT+01:00 Victor Stinner :
>> Perhaps if you add some guards somewhere :-)
>
> We have runtime checks but only implemented in debug mode for efficiency.
>
> By the way, I proposed once to add an environment variable to allow to
> enable these checks without having to recompile Python.  Since the PEP
> 445, it became easy to implement this. What do you think?
> https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable

Ok, I wrote a patch to implement a new PYTHONMALLOC environment variable:

   http://bugs.python.org/issue26516

PYTHONMALLOC=debug installs debug hooks to:

* detect API violations, ex: PyObject_Free() called on a buffer
allocated by PyMem_Malloc()
* detect write before the start of the buffer (buffer underflow)
* detect write after the end of the buffer (buffer overflow)

https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks

The main advantage of this variable is that you don't have to
recompile Python in debug mode to benefit of these checks.

Recompiling Python in debug mode requires to recompile *all*
extensions modules since the debug ABI is incompatible. When I played
with tracemalloc on Python 2 ( http://pytracemalloc.readthedocs.org/
), I had such issues, it was very annoying with non-trivial extension
modules like PyQt or PyGTK. With PYTHONMALLOC, you don't have to
recompile extension modules anymore!


With tracemalloc and PYTHONMALLOC=debug, we will have a complete tool
suite to "debug memory"!

My motivation for PYTHONMALLOC=debug is to detect API violations to
prepare my change on PyMem_Malloc() allocator (
http://bugs.python.org/issue26249 ), but also to help users to detect
bugs.

It's common that users report a bug: "Python crashed", but have no
idea of the responsible of the crash. I hope that detection of buffer
underflow & overflow will help them to detect bugs in their own
extension modules.


Moreover, I added PYTHONMALLOC=malloc to ease the use of external
memory debugger on Python. By default, Python uses pymalloc allocator
for PyObject_Malloc() which raises a lot of false positive in
Valgrind. We even have a configuration (--with-valgrind) and a
Valgrind suppressino file to be able to skip these false alarms in
Valgrind. IMHO PYTHONMALLOC=malloc is a simpler option to use Valgrind
(or other tools).

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New OpenSSL - has anyone ever looked at (in)compatibility with LibreSSL

[Michael Felt]

Can look at it. There has been a lot of discussion, iirc, between 
OpenSSL and LibreSSL re: version identification.

Thx for the reference.

On 08-Mar-16 14:55, Hasan Diwan wrote:


On 8 March 2016 at 00:49, Michael Felt > wrote:


As a relative newcomer I may have missed a long previous
discussion re: linking with OpenSSL and/or LibreSSL.
In an ideal world this would be rtl linking, i.e., underlying
complexities of *SSL libraries are hidden from applications.

In short, when I saw this http://bugs.python.org/issue26465 Title:
Upgrade OpenSSL shipped with python installers, it reminded me I
need to start looking at LibreSSL again - and that, if not already
done - might be something "secure" for python as well.


According to the libressl website, one of the projects primary goals 
is to remain "backwards-compatible with OpenSSL", which is to say, to 
either have code work without changes or to fail gracefully when it 
uses the deprecated bits. It does seem it ships with OpenBSD. There is 
an issue open on bugs to address whatever incompatibilities remain 
between LibreSSL and OpenSSL[1]. Perhaps you might want to take a look 
at that? -- H

1. https://bugs.python.org/issue23177


Michael
___
Python-Dev mailing list
Python-Dev@python.org 
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/hasan.diwan%40gmail.com




--
OpenPGP: http://hasan.d8u.us/gpg.asc
Sent from my mobile device
Envoyé de mon portable


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Brett Cannon]

On Wed, 9 Mar 2016 at 06:57 Victor Stinner  wrote:

> 2016-02-08 15:18 GMT+01:00 Victor Stinner :
> >> Perhaps if you add some guards somewhere :-)
> >
> > We have runtime checks but only implemented in debug mode for efficiency.
> >
> > By the way, I proposed once to add an environment variable to allow to
> > enable these checks without having to recompile Python.  Since the PEP
> > 445, it became easy to implement this. What do you think?
> >
> https://www.python.org/dev/peps/pep-0445/#add-a-new-pydebugmalloc-environment-variable
>
> Ok, I wrote a patch to implement a new PYTHONMALLOC environment variable:
>
>http://bugs.python.org/issue26516
>
> PYTHONMALLOC=debug installs debug hooks to:
>
> * detect API violations, ex: PyObject_Free() called on a buffer
> allocated by PyMem_Malloc()
> * detect write before the start of the buffer (buffer underflow)
> * detect write after the end of the buffer (buffer overflow)
>
> https://docs.python.org/dev/c-api/memory.html#c.PyMem_SetupDebugHooks
>
> The main advantage of this variable is that you don't have to
> recompile Python in debug mode to benefit of these checks.
>

I just wanted to say this all sounds awesome! Thanks for all the hard work
on making our memory management story easier to work with, Victor.

-Brett


>
> Recompiling Python in debug mode requires to recompile *all*
> extensions modules since the debug ABI is incompatible. When I played
> with tracemalloc on Python 2 ( http://pytracemalloc.readthedocs.org/
> ), I had such issues, it was very annoying with non-trivial extension
> modules like PyQt or PyGTK. With PYTHONMALLOC, you don't have to
> recompile extension modules anymore!
>
>
> With tracemalloc and PYTHONMALLOC=debug, we will have a complete tool
> suite to "debug memory"!
>
> My motivation for PYTHONMALLOC=debug is to detect API violations to
> prepare my change on PyMem_Malloc() allocator (
> http://bugs.python.org/issue26249 ), but also to help users to detect
> bugs.
>
> It's common that users report a bug: "Python crashed", but have no
> idea of the responsible of the crash. I hope that detection of buffer
> underflow & overflow will help them to detect bugs in their own
> extension modules.
>
>
> Moreover, I added PYTHONMALLOC=malloc to ease the use of external
> memory debugger on Python. By default, Python uses pymalloc allocator
> for PyObject_Malloc() which raises a lot of false positive in
> Valgrind. We even have a configuration (--with-valgrind) and a
> Valgrind suppressino file to be able to skip these false alarms in
> Valgrind. IMHO PYTHONMALLOC=malloc is a simpler option to use Valgrind
> (or other tools).
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com