[issue25300] Enable Intel MPX (Memory protection Extensions) feature

STINNER Victor added the comment: > I'm going to formally reject this, since GCC removed -fmpx support. "The MPX extensions to the C and C++ languages have been deprecated and will be removed in a future release." https://gcc.gnu.org/gcc-8/changes.html Oh. It seems like the reason is to

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Benjamin Peterson added the comment: I'm going to formally reject this, since GCC removed -fmpx support. -- nosy: +benjamin.peterson resolution: -> rejected stage: -> resolved status: open -> closed ___ Python tracker

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi Christian, There is a Skylake buildbot that has MPX capabilities. Please find it here: http://buildbot.python.org/all/builders/x86-64%20Ubuntu%2015.10%20Skylake%20CPU%203.6 Regards, Florin -- ___ Python tracker

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Christian Heimes added the comment: Let's have another look at this enhancement for 3.7. Hopefully we have some machines to develop with and test MPX, too. I don't have any machine at home that supports hardware MPX. Does any of our buildbots have a Skylake with MPX? -- nosy:

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Changes by Christian Heimes : -- versions: +Python 3.7 ___ Python tracker ___ ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Antoine Pitrou added the comment: For the record, the llvm-dev is currently having a discussion thread about MPX. Some of the feedback is a bit critical: http://lists.llvm.org/pipermail/llvm-dev/2016-February/094828.html Also this quite comprehensive report:

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Changes by Joseph Hackman : -- nosy: +Joseph.Hackman ___ Python tracker ___ ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi all, Our latest effort on enabling MPX in CPython has concentrated around eliminating all INIT_BOUNDS and BND_LEGACY attributes that are used to bypass bounds checking for certain pointers. In order to avoid using these attributes, we needed to find and fix

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi Antoine, The Py_INIT_BOUNDS calls were used because MPX generated a very large number of error messages about pointer bounds violations at compile or run time, that made Python unusable. The approach was to analyze the errors and ignore checking if no

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi Stefan, > Yes, I dislike that, too. The question is why gcc has supported > the "struct hack" for more than 20 years but needs an annotation > just for MPX. The "struct hack" represents a problem for MPX because it intentionally exceeds the bounds of the

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Stefan Krah added the comment: > Py_ssize_t ob_array[1] Py_VARIABLE_SIZE; /* Looks weird and confusing */ Yes, I dislike that, too. The question is why gcc has supported the "struct hack" for more than 20 years but needs an annotation just for MPX. Is the annotation also needed for the C99

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Raymond Hettinger added the comment: I'm wary of making such extensive changes throughout the codebase for an optional processor-specific feature of limited benefit and unproven worth. With each new macro and trick (Py_VARIABLE_SIZE and Py_INIT_BOUNDS), we make it harder for people to

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Antoine Pitrou added the comment: Hi, I've reviewed the latest patch. I have a more general question: the patch sprinkles some Py_INIT_BOUNDS() calls in various places. What the macro does is set the pointer's bounds to "INIT", which basically means disable any checks (the pointer is allowed

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi all, We have set up a Skylake buildbot, which could be used for MPX builds. The errors that appear when running regrtest are proxy related and we are looking into a solution for that. There is no problem for tests that do not access network resources. The

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Please see the updated patch after Zachary Ware's review. Thank you for the feedback. -- Added file: http://bugs.python.org/file40931/mpx_enable_3_6_v6.patch ___ Python tracker

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Raymond Hettinger added the comment: [ Stefan Krah] > I'd suggest to wait until the mpx support is stable in gcc. I concur. -- ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Stefan Krah added the comment: Florin, thanks for the explanation. I'd suggest to wait until the mpx support is stable in gcc. Some security features in gcc never take off (-ftrapv is broken, libmudflap is deprecated, ...) so it would be nice to have some reassurance. --

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi Stefan, The GCC MPX feature proved stable when using O0, having tested multiple times with regrtest on Skylake, Broadwell and Haswell. Still, this is a new feature and GCC might improve it with time. -- ___ Python

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Stefan Krah added the comment: > The Python MPX binary will be built with –O0 because other optimization > levels will change instruction order and cause crashes, making it useful for > debugging purposes. I've trouble understanding this: Is the gcc mpx feature not stable? Looking at the

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi all, I updated the patch after fixing crashes that occurred at compile and run time on Ubuntu 15.10, with kernel version 4.2.0-16-generic, LD version 2.25.1 and GCC version 5.2.1. The Python MPX binary will be built with –O0 because other optimization levels

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: I have modified the small object allocator to set proper bounds for the pointers it returns. Please see the updated patches. -- Added file: http://bugs.python.org/file40728/mpx_enable_3_6_v4.patch ___ Python tracker

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Changes by Florin Papa : Added file: http://bugs.python.org/file40729/mpx_enable_2_7_v4.patch ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Antoine Pitrou added the comment: Thanks. For the record, the whole test suite still passes? This is good to know :) -- ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: I ran the full Grand Unified Python Benchmarks suite on Python 3.6 and 2.7 before submitting the patches and everything went well. Sorry I did not mention it :) -- ___ Python tracker

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Antoine Pitrou added the comment: I meant the test suite ("python -m test.regrtest"). I presume you did, just checking :-) -- ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: I ran regrtest now and there are a few tests that fail. I will look into the cause of this and provide a solution. -- ___ Python tracker

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Antoine Pitrou added the comment: > This means that when you perform a malloc, the pointer bounds will be set > inside the malloc call, and then passed on to your variable. For this to be useful in Python, you would have to annotate Python's small object allocator to designate it as a

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

STINNER Victor added the comment: FYI In the discussion of the PEP 445 "Add new APIs to customize Python memory allocators" it was proposed to add runtime option to choose the memory allocator. At least for debug purpose, it would help to be able to use malloc() for *all* Python memory

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: No modifications need to be made to the small object allocator. The malloc situation was just an example, it does not mean that setting pointer bounds occurs only in a malloc call. It also occurs when you declare a static array or when you initialize a new

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Antoine Pitrou added the comment: The small object allocator uses large mmap-allocated arenas of 256 KB (IIRC) and carves small objects out of it. So unless the pointer returned by PyObject_Malloc() has its bounds set manually using one of the intrinsic functions (*), MPX would believe the

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: > Is there a runtime cost or does the hardware run the bounds checks in > parallel with memory accesses? Are the bounds set when malloc() is called > or elsewhere? I read the provided links but can't say I fully understand how > it works exactly (which memory

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi Antoine, I understand the problem with the small object allocator now. I will have a closer look at it and come back with a solution. Thank you for pointing this out. -- ___ Python tracker

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Raymond Hettinger added the comment: Is there a runtime cost or does the hardware run the bounds checks in parallel with memory accesses? Are the bounds set when malloc() is called or elsewhere? I read the provided links but can't say I fully understand how it works exactly (which memory

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi all, I added LDFLAGS_NODIST in order to avoid distutils problems. Please see the updated patches. Thank you, Florin -- type: enhancement -> security Added file: http://bugs.python.org/file40697/mpx_enable_3_6_v3.patch

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Changes by Florin Papa : Added file: http://bugs.python.org/file40698/mpx_enable_2_7_v3.patch ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi Stefan, When activating the "--with-mpx" flag in ./configure, a check is also performed to see if the compiler is GCC version 5.1 or higher in order to support MPX (and an appropriate error is shown, indicating that the compiler version is wrong). Setting

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi Antoine, Third party modules might need patching to work with MPX, as did the math module, included in this patch. Everything else worked fine in Python 2.7.10+ and 3.6 when running the Grand Unified Python Benchmarks suite and Openstack Swift ssbench

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi all, Thank you for your feedback on the code I submitted. I added the MPX flags to CFLAGS, not CFLAGS_NODIST, because they also need to be included in LDFLAGS. Please see the new set of patches. Thank you, Florin -- Added file:

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Changes by Florin Papa : Added file: http://bugs.python.org/file40682/mpx_enable_2_7_v2.patch ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Antoine Pitrou added the comment: This looks ok to me as long as it's disabled by default. -- type: security -> enhancement ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Stefan Krah added the comment: > Third party modules might need patching to work with MPX... If the flags go into CFLAGS, these modules won't compile with distutils. Perhaps we should also add LDFLAGS_NODIST, if that would solve your problem. --

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Stefan Krah added the comment: Wouldn't setting CFLAGS be sufficient? ./configure CFLAGS="-fcheck-pointer-bounds -mmpx" Otherwise we could also add --with-stack-protector and --with-fortify-source and others. I'm not sure if we should add a special --with* option for every gcc flag.

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Stefan Krah added the comment: > Whether the hardware is MPX enabled is irrelevant for the build process. The build process is only one part of the equation. Compilers do have bugs (Python has been affected by gcc, Visual Studio and suncc bugs), and we should test the actual generated MPX

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

R. David Murray added the comment: No go on the ICC mac buildbot then, it's an i7-4578U, so not even gen 5. -- ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Antoine Pitrou added the comment: This presents the risk of breaking third-party extensions, right? -- nosy: +pitrou ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Changes by Florin Papa : Added file: http://bugs.python.org/file40658/mpx_enable_2_7.patch ___ Python tracker ___

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

New submission from Florin Papa: Hi all, My name is Florin Papa and I work in the Server Languages Optimizations Team at Intel Corporation. I would like to submit a patch that enables the use of the Intel MPX (Memory Protection Extensions) feature on Python default and 2.7. Invalid memory

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Changes by Stefan Krah : -- nosy: +skrah ___ Python tracker ___ ___ Python-bugs-list

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: > Why not enablind the option by default if GCC 5.1 or newer is detected? Is > there a risk of breaking third-party extensions? > Should we pass the MPX compiler options to third-party extensions by way? On a processor that does not support MPX technology, MPX

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Stefan Krah added the comment: In principle adding the option is fine, but do we have a buildbot with a Skylake processor? We have lots of options that are untested and that break periodically. -- nosy: +zach.ware ___ Python tracker

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

R. David Murray added the comment: I think Intel will make sure we do have one. Florin: how does one determine if a processor supports mxp? I don't know if the mac mini running the ICC buildbot is new enough to have a Skylake processor, but I suppose there's at least some chance it does.

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

STINNER Victor added the comment: Why not enablind the option by default if GCC 5.1 or newer is detected? Is there a risk of breaking third-party extensions? Should we pass the MPX compiler options to third-party extensions by way? I'm not sure that it's ok to add such change to Python 2.7.

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

R. David Murray added the comment: Well, since it is arguably security related, I think that pushes it toward the "yes" side. And yes, I think we could end up "breaking" third party code, as well as python (thus the need for a buildbot), but that may not be a bad thing in this case (that

[issue25300] Enable Intel MPX (Memory protection Extensions) feature

Florin Papa added the comment: Hi David, If you are not sure about MPX hardware support, you can find out the processor model with the following commands: Linux - cat /proc/cpuinfo | grep model Mac - sysctl -n machdep.cpu.brand_string If the processor code is one of the following -