[issue28821] generate_opcode_h.py crash when run with python2
Florin Papa added the comment: I tested and the script is no longer called if you do a "make touch" beforehand. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28821> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28821] generate_opcode_h.py crash when run with python2
Florin Papa added the comment: Whenever I did a fresh clone, the Tools/scripts/generate_opcode_h.py would be called. Thank you for fixing this. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28821> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28821] generate_opcode_h.py crash when run with python2
New submission from Florin Papa: Hello, This is Florin Papa from the Dynamic Scripting Languages Optimizations team in Intel Corporation. In changeset 105360:46e2755b022c [1] the generate_opcode_h.py script was modified to use tokenize.open() in order to avoid a Resource Warning. The tokenize module does not have an 'open' attribute in python2, therefore the build will crash if python3 is not present on the system. The patch attached checks if the tokenize module has the 'open' attribute. If it does, the current implementation will be used. Otherwise, it will fall back to the old implementation. Thank you, Florin [1] https://hg.python.org/cpython/rev/46e2755b022c -- components: Build files: generate_opcode_h.patch keywords: patch messages: 281882 nosy: florin.papa priority: normal severity: normal status: open title: generate_opcode_h.py crash when run with python2 type: crash versions: Python 3.7 Added file: http://bugs.python.org/file45674/generate_opcode_h.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28821> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi Christian, There is a Skylake buildbot that has MPX capabilities. Please find it here: http://buildbot.python.org/all/builders/x86-64%20Ubuntu%2015.10%20Skylake%20CPU%203.6 Regards, Florin -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26383] number of decimal places in csv output
New submission from Florin Papa: Hi, This is Florin Papa from the Dynamic Scripting Languages Optimizations Team at Intel Corporation. This patch checks whether the benchmark measurement that will be included in the csv output has more than 3 of the first decimal places equal to 0 and outputs more decimal places. The problem is that each measurement in the csv output has only 6 decimal places (default for "%f" format), while some benchmarks, like unpack_sequence, will output values of the magnitude 0.xx (first four decimal places 0). This truncation of the original value can lead to incorrect results. -- components: Benchmarks files: decimal_csv_output.csv messages: 260460 nosy: brett.cannon, florin.papa, pitrou priority: normal severity: normal status: open title: number of decimal places in csv output type: behavior versions: Python 2.7, Python 3.6 Added file: http://bugs.python.org/file41955/decimal_csv_output.csv ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26383> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26275] perf.py: calibrate benchmarks using time, not using a fixed number of iterations
Florin Papa added the comment: I ran perf to use calibration and there is no difference in stability compared to the unpatched version. With patch: python perf.py -b json_dump_v2 -v --csv=out1.csv --affinity=2 ../cpython/python ../cpython/python INFO:root:Automatically selected timer: perf_counter [1/1] json_dump_v2... Calibrating INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 1 -l 1 --timer perf_counter` INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 1 -l 2 --timer perf_counter` INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 1 -l 4 --timer perf_counter` INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 1 -l 8 --timer perf_counter` INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 1 -l 16 --timer perf_counter` INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 1 -l 32 --timer perf_counter` Calibrating => num_runs=10, num_loops=32 (0.50 sec < 0.87 sec) INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 10 -l 32 --timer perf_counter` INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 10 -l 32 --timer perf_counter` Report on Linux centos 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 Total CPU cores: 18 ### json_dump_v2 ### Min: 0.877497 -> 0.886482: 1.01x slower <-- Avg: 0.878150 -> 0.888351: 1.01x slower Not significant Stddev: 0.00054 -> 0.00106: 1.9481x larger Without patch: python perf.py -b json_dump_v2 -v --csv=out1.csv --affinity=2 ../cpython/python ../cpython/python INFO:root:Automatically selected timer: perf_counter [1/1] json_dump_v2... INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 50 --timer perf_counter` INFO:root:Running `taskset --cpu-list 2 ../cpython/python performance/bm_json_v2.py -n 50 --timer perf_counter` Report on Linux centos 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 Total CPU cores: 18 ### json_dump_v2 ### Min: 2.755514 -> 2.764131: 1.00x slower <-- (almost) same as above Avg: 2.766546 -> 2.775587: 1.00x slower Not significant Stddev: 0.00538 -> 0.00382: 1.4069x smaller -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26275> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26275] perf.py: calibrate benchmarks using time, not using a fixed number of iterations
Florin Papa added the comment: I was also talking about the variance/deviation of the mean value displayed by perf.py, sorry if I was unclear. The perf.py output in my previous message showed little difference between the patched and non-patched version. I will also try increasing the number of runs to see if there is any change. The CPU isolation feature is a great finding, thank you. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26275> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26275] perf.py: calibrate benchmarks using time, not using a fixed number of iterations
Florin Papa added the comment: I have tested the patch and it does not seem to solve the stability problem. With patch: python performance/bm_json_v2.py -n 10 --timer perf_counter 0.02829067991115153 0.027778947027400136 0.027756798081099987 0.027362094027921557 0.027256828034296632 0.02766443998552859 0.027390124974772334 0.027398303849622607 0.02731629996560514 0.02743363007903099 Without patch: python performance/bm_json_v2.py -n 10 --timer perf_counter 2.732215315802023 2.7394272631499916 2.731212269049138 2.7511520159896463 2.739834441104904 2.754247901029885 2.734024364966899 2.7447318979538977 2.744731454877183 2.7513630259782076 Instead, I notice that, especially for the first runs, the measured values tend to be higher. This is expected, since the cache has not been populated yet, but this can also happen after a context switch. Therefore, we could eliminate these "odd" values by calculating the median value of all the runs and only keeping measurements within 1% of that median value. What do you think? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26275> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26274] Add CPU affinity to perf.py
New submission from Florin Papa: Hi all, This is Florin Papa from the Dynamic Scripting Languages Optimizations Team from Intel Corporation. The patch submitted adds an affinity feature to the Grand Unified Python Benchmarks suite to allow running benchmarks on a given CPU/set of CPUs. It is implemented for Linux and uses the taskset command to bond a command to the CPUs specified. This minimizes run to run variation, as we can get considerable differences in measured performance from running a benchmark on different cores. The taskset command receives a CPU mask that specifies which cores in the system will be used for the command. Example: python perf.py –-affinity=0x1 will use processor #0 python perf.py –-affinity=0x3 will use processors #0 and #1 Thank you, Florin -- components: Benchmarks files: affinity.patch keywords: patch messages: 259464 nosy: brett.cannon, florin.papa, pitrou priority: normal severity: normal status: open title: Add CPU affinity to perf.py versions: Python 2.7, Python 3.6 Added file: http://bugs.python.org/file41788/affinity.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26274] Add CPU affinity to perf.py
Florin Papa added the comment: Thank you for the feedback. Please see the updated patch. -- Added file: http://bugs.python.org/file41791/affinity_v2.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26274] Add CPU affinity to perf.py
Florin Papa added the comment: I was thinking that a warning might go unnoticed, that is why I chose to end the program if taskset is not found. It would force the user to install missing packages. Otherwise the patch looks ok. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26274] Add CPU affinity to perf.py
Florin Papa added the comment: I have updated the patch to throw an exception if the taskset command is not available. Not checking that taskset works will generate an ambiguous error (if there is no taskset on the system) that does not explicitly point to the problem. I believe that users working on performance have knowledge on the platform and OS setup they are using and they will draw benefit from this feature because it gives more accurate results. -- Added file: http://bugs.python.org/file41792/affinity_v3.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26274] Add CPU affinity to perf.py
Florin Papa added the comment: No problem :) -- resolution: fixed -> status: closed -> open ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26274> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26110] Speedup method calls 1.2x
Changes by Florin Papa <florin.p...@intel.com>: -- nosy: +florin.papa ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26110> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi all, Our latest effort on enabling MPX in CPython has concentrated around eliminating all INIT_BOUNDS and BND_LEGACY attributes that are used to bypass bounds checking for certain pointers. In order to avoid using these attributes, we needed to find and fix the root cause of the problems. The main issue was represented by the small object allocator (Objects/obmalloc.c), which was performing some operations that MPX considered unsafe (pointer jumping). A similar problem was found in the allocator used by the garbage collection module (Modules/gcmodule.c). These issues, as well as other minor operations considered unsafe by MPX (Objects/listobject.c, Objects/dictobject.c) have been addressed so far. As a result, we were able to eliminate all INIT_BOUNDS and BND_LEGACY attributes from the code. Also, we identified the optimization flag that caused crashes when compiling with –O3, which is –fipa-icf. Compiling with “-O3 –fno-ipa-icf” now works fine. The entire regrtest suite passes, except test_capi and test_faulthandler. Test_capi fails because of a “\n” outputted by the MPX runtime at stdout instead of stderr (fixed in the GCC 6 trunk). Test_faulthandler fails because we have disabled the faulthandler module when MPX is active, as it produced crashes since both the faulthandler and the MPX runtime overwrite the default SIGSEGV handler and the new handlers would interfere with each other. The current patch works on GCC 5.3.0, which solves a linking problem with libmpx, present in GCC 5.2.1. We still have some problems, such as bounds warnings that only appear once in 10 runs for a few of the tests, but do not cause crashes or failed tests. The biggest problem we face is the presence of pointers that do not have bounds. These could be the result of some bugs we found in MPX: 1. Calling strlen and memset (possibly others) for the first time in a program will not be subject to MPX checks 2.Copying an array of pointers to a new location will reset the first pointer’s bounds (deep copy of the pointer bounds fails) The first problem was solved by upgrading ldd to version 2.22, while the second issue will be solved by the GCC 6 release (around April 2016), which will offer more stable support for MPX. Therefore, we have decided to wait until the GCC 6 release to provide a final version of the MPX patch for CPython. Meanwhile, you can see the latest modifications we have made in the patch attached. Thank you, Florin Papa -- Added file: http://bugs.python.org/file41781/mpx_enable_3_6_v7.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25961] Disallow the null character in type name
Florin Papa added the comment: Hi all, I fixed a compile error introduced in Python 2.7 by this issue. There is a jump made to an nonexistent label "error" in type_new function in Objects/typeobject.c. Please see the attached patch. Regards, Florin Papa -- nosy: +florin.papa Added file: http://bugs.python.org/file41460/type_name_null_fix.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25961> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Florin Papa added the comment: No problem. Thank you for merging the patch! -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi Antoine, The Py_INIT_BOUNDS calls were used because MPX generated a very large number of error messages about pointer bounds violations at compile or run time, that made Python unusable. The approach was to analyze the errors and ignore checking if no obvious violation took place, in the hope to find the root cause. The problem is that the errors can come from an actual bug or from a false positive that can be propagated throughout the code. While we could not find evidence of actual bugs, we found 2 examples of the latter, in listobject.c (line 1100, in the binarysort function) and dictobject.c (line 1797 in dict_keys function and line 1892 in dict_items). The problem is caused by this coding pattern that is used in the two instances mentioned above: a pointer to an allocated memory zone is used to access a different allocated memory zone by adding the difference between their start addresses to that pointer. Although this newly formed address is valid, in the context of the pointer used for this operation it is outside its bounds so it’s signaled as a bounds violation. p ** q *---* <--> offset p and q point to valid memory zones, being separated by an offset. If we do something like new_pointer = q – offset; // which is actually equal to p value = *new_pointer; // dereferencing will generate an MPX bounds violation, because // new_pointer will keep q’s original bounds I will rewrite the code for these cases and provide the new patch as soon as I have it. Regards, Florin -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi Stefan, > Yes, I dislike that, too. The question is why gcc has supported > the "struct hack" for more than 20 years but needs an annotation > just for MPX. The "struct hack" represents a problem for MPX because it intentionally exceeds the bounds of the array declared inside in order to support a variable sized structure. Without the Py_VARIABLE_SIZE annotation, MPX will generate a bounds violation when accessing outside the declared size of the array. > Is the annotation also needed for the C99 version (ob_array[])? I tested and for the C99 version the annotation is not needed. Apparently ob_array inherits the bounds of the structure in this case. Regards, Florin -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Florin Papa added the comment: I have adapted the existent patch to the new benchmarks codebase. Please see the new patch (django_v3_2.patch) in the files section. The benchmark expects to have django in lib/Django-1.9 folder. The Django-1.9 sources are attached separately because of their size. hg clone https://hg.python.org/benchmarks cd benchmarks/ copy django_v3_2.patch to the current directory and Django-1.9.tar.gz to the lib/ directory hg import --no-commit django_v3_2.patch cd lib/ tar -xvf Django-1.9.tar.gz -- Added file: http://bugs.python.org/file41221/django_v3_2.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Changes by Florin Papa <florin.p...@intel.com>: Added file: http://bugs.python.org/file41222/Django-1.9.tar.gz ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Florin Papa added the comment: I have attached django_v3_3.patch, as the previous one did not include the added file (performance/bm_django_v3.py). -- Added file: http://bugs.python.org/file41223/django_v3_3.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25719] Deprecate spitfire benchmark
Florin Papa added the comment: No problem. Thank you! -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25719> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25705] Fix regex_compile benchmark crash
Florin Papa added the comment: No problem. Thank you for fixing this. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25705> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25721] Fix pybench always_display error
Florin Papa added the comment: Yes, thank you for fixing this. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25721> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25719] Deprecate spitfire benchmark
New submission from Florin Papa: Hi all, This is Florin Papa from the Server Languages Optimizations Team at Intel Corporation. The patch submitted here deprecates the spitfire benchmark, which uses psyco. Psyco is "unmaintained and dead", according to their website (http://psyco.sourceforge.net/) and it has not received any updates since 2012. The spitfire benchmark crashes because psyco does not have support for x64 architectures. To apply the patch please follow these steps: hg clone https://hg.python.org/benchmarks cd benchmarks/ copy deprecate_spitfire.patch to the current directory hg import --no-commit deprecate_spitfire.patch Thank you, Florin Papa -- components: Benchmarks files: deprecate_spitfire.patch keywords: patch messages: 255260 nosy: brett.cannon, florin.papa, pitrou priority: normal severity: normal status: open title: Deprecate spitfire benchmark type: crash versions: Python 2.7 Added file: http://bugs.python.org/file41147/deprecate_spitfire.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25719> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25721] Fix pybench always_display error
Changes by Florin Papa <florin.p...@intel.com>: -- type: -> crash ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25721> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25721] Fix pybench always_display error
New submission from Florin Papa: Hi all, This is Florin Papa from the Server Languages Optimizations Team at Intel Corporation. The patch submitted here solves a crash occurring on Python 2.7 after running pybench from perf.py (python perf.py -r -b pybench base_python modified_python). The error appears when trying to display the results of the run because the PyBenchBenchmarkResult class does not include the attribute "always_display". Here is the error message received: File "../benchmarks/perf.py", line 2574, in main if result.always_display: AttributeError: 'PyBenchBenchmarkResult' object has no attribute 'always_display' To apply the patch please follow these steps: hg clone https://hg.python.org/benchmarks cd benchmarks/ copy fix_pybench_results.patch to the current directory hg import --no-commit fix_pybench_results.patch Thank you, Florin Papa -- components: Benchmarks messages: 255262 nosy: brett.cannon, florin.papa, pitrou priority: normal severity: normal status: open title: Fix pybench always_display error versions: Python 2.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25721> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25721] Fix pybench always_display error
Changes by Florin Papa <florin.p...@intel.com>: -- keywords: +patch Added file: http://bugs.python.org/file41149/fix_pybench_results.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25721> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25705] Fix regex_compile benchmark crash
New submission from Florin Papa: Hi all, This is Florin Papa from the Server Languages Optimizations Team at Intel Corporation. The patch submitted here solves a crash occurring on Python 2.7 and 3.6 when running the regex_compile benchmark. The clear_cache() call was replaced with re.purge(). To apply the patch please follow these steps: hg clone https://hg.python.org/benchmarks cd benchmarks/ copy fix_regex_compile.patch to the current directory hg import --no-commit fix_regex_compile.patch Thank you, Florin Papa -- components: Benchmarks files: fix_regex_compile.patch keywords: patch messages: 255152 nosy: brett.cannon, florin.papa, pitrou priority: normal severity: normal status: open title: Fix regex_compile benchmark crash type: crash versions: Python 2.7, Python 3.6 Added file: http://bugs.python.org/file41136/fix_regex_compile.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25705> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi all, We have set up a Skylake buildbot, which could be used for MPX builds. The errors that appear when running regrtest are proxy related and we are looking into a solution for that. There is no problem for tests that do not access network resources. The buildbot is available here: http://buildbot.python.org/all/builders/x86-64%20Ubuntu%2015.10%20Skylake%20CPU%202.7 Thank you, Florin Papa -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Please see the updated patch after Zachary Ware's review. Thank you for the feedback. -- Added file: http://bugs.python.org/file40931/mpx_enable_3_6_v6.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi Stefan, The GCC MPX feature proved stable when using O0, having tested multiple times with regrtest on Skylake, Broadwell and Haswell. Still, this is a new feature and GCC might improve it with time. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi all, I updated the patch after fixing crashes that occurred at compile and run time on Ubuntu 15.10, with kernel version 4.2.0-16-generic, LD version 2.25.1 and GCC version 5.2.1. The Python MPX binary will be built with –O0 because other optimization levels will change instruction order and cause crashes, making it useful for debugging purposes. The minimum GCC version was upgraded to 5.2 for the MPX build in order to make sure there is full support for this feature. All tests from regrtest pass, except pest_pyclbr which also fails on a non-MPX build and two other tests, test_capi and test_faulthandler, which fail because the output expected after a segmentation fault is generated is modified by MPX (which reacts to a memory violation). The built MPX Python binary was evaluated (“./python -m test.regrtest”) on two older Intel generation processors, Haswell and Broadwell, and no functional issue was observed. However, we don’t know its impact on third-party extensions. Thank you, Florin -- Added file: http://bugs.python.org/file40862/mpx_enable_3_6_v5.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: I have modified the small object allocator to set proper bounds for the pointers it returns. Please see the updated patches. -- Added file: http://bugs.python.org/file40728/mpx_enable_3_6_v4.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Changes by Florin Papa <florin.p...@intel.com>: Added file: http://bugs.python.org/file40729/mpx_enable_2_7_v4.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: I ran the full Grand Unified Python Benchmarks suite on Python 3.6 and 2.7 before submitting the patches and everything went well. Sorry I did not mention it :) -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: I ran regrtest now and there are a few tests that fail. I will look into the cause of this and provide a solution. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: No modifications need to be made to the small object allocator. The malloc situation was just an example, it does not mean that setting pointer bounds occurs only in a malloc call. It also occurs when you declare a static array or when you initialize a new pointer: int *x = (int*)malloc(10 * sizeof(int)); int *y = x; // the bounds for x will be passed on to y When using "fcheck-pointer-bounds -mmpx" _all_ memory accesses will be instrumented. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: > Is there a runtime cost or does the hardware run the bounds checks in > parallel with memory accesses? Are the bounds set when malloc() is called > or elsewhere? I read the provided links but can't say I fully understand how > it works exactly (which memory blocks are protected, where the bounds get > set, when the registers are loaded, etc). There is a runtime cost associated with MPX instrumentation, which is kept to a minimum due to the use of hardware instructions and registers. When "fcheck-pointer-bounds -mmpx" compilation flags are set, instrumentation is enabled for _all_ memory acceses in the code. This means that when you perform a malloc, the pointer bounds will be set inside the malloc call, and then passed on to your variable. Alternatively, you can manually instrument only regions of interest in your code using GCC functions described here [1]. > Also, I'm curious about whether we have direct controls over the bounds. For > example, in a listobject.c or _collections.c object could the bounds be > tightened to only include the active data and excluded the unused part of the > overallocation? Please see __bnd_set_ptr_bounds here [1] for bound manipulation. In order to have a better understanding of what happens when using MPX, I suggest writing a simple C program and look at the assembly code generated (MPX instructions and registers begin with bnd). You can use the following steps: gcc -g -c test.c -O2 -fcheck-pointer-bounds -mmpx objdump -d -M intel -S test.o [1] https://gcc.gnu.org/wiki/Intel%20MPX%20support%20in%20the%20GCC%20compiler#Compiler_intrinsics_and_attributes -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi Antoine, I understand the problem with the small object allocator now. I will have a closer look at it and come back with a solution. Thank you for pointing this out. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi all, I added LDFLAGS_NODIST in order to avoid distutils problems. Please see the updated patches. Thank you, Florin -- type: enhancement -> security Added file: http://bugs.python.org/file40697/mpx_enable_3_6_v3.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Changes by Florin Papa <florin.p...@intel.com>: Added file: http://bugs.python.org/file40698/mpx_enable_2_7_v3.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi Stefan, When activating the "--with-mpx" flag in ./configure, a check is also performed to see if the compiler is GCC version 5.1 or higher in order to support MPX (and an appropriate error is shown, indicating that the compiler version is wrong). Setting the flags manually will only result in a compiler error. Also, it might not be obvious for the user from the GCC documentation that both "-fcheck-pointer-bounds" and "-mmpx" flags are needed to enable MPX instrumentation, so adding the "--with-mpx" flag to configure will make it easier to use this feature. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi Antoine, Third party modules might need patching to work with MPX, as did the math module, included in this patch. Everything else worked fine in Python 2.7.10+ and 3.6 when running the Grand Unified Python Benchmarks suite and Openstack Swift ssbench benchmark (ssbench supports only 2.7.10+). -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi all, Thank you for your feedback on the code I submitted. I added the MPX flags to CFLAGS, not CFLAGS_NODIST, because they also need to be included in LDFLAGS. Please see the new set of patches. Thank you, Florin -- Added file: http://bugs.python.org/file40681/mpx_enable_3_6_v2.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Changes by Florin Papa <florin.p...@intel.com>: Added file: http://bugs.python.org/file40682/mpx_enable_2_7_v2.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Changes by Florin Papa <florin.p...@intel.com>: Added file: http://bugs.python.org/file40658/mpx_enable_2_7.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
New submission from Florin Papa: Hi all, My name is Florin Papa and I work in the Server Languages Optimizations Team at Intel Corporation. I would like to submit a patch that enables the use of the Intel MPX (Memory Protection Extensions) feature on Python default and 2.7. Invalid memory access problem is commonly found in C/C++ programs and leads to time consuming debugging, program instability and vulnerability. Many attacks exploit software bugs related to invalid memory access caused by buffer overflow/underflow. Existing set of techniques to avoid such memory bugs represent software only solutions which result in poor protection and performance. Intel MPX introduces new registers and new instructions that operate on these registers. Some of the registers added are bounds registers which store a pointer’s lower bound and upper bound limits. Whenever the pointer is used, the requested reference is checked against the pointer’s associated bounds, thereby preventing out-of-bound memory access (such as buffer overflows and overruns). Out-of-bounds memory references initiate a #BR exception which can then be handled in an appropriate manner. MPX technology was introduced on the sixth generation of Intel Core processors, code name SkyLake, the successor to Broadwell microarchitecture (only desktop processor is available on the market, server has not been released). For older generations of Intel processors that do not support MPX, nop’s will be added to the instruction stream [1]. MPX enabled builds are only possible using gcc 5.1 and newer [2]. To apply the patch please follow these steps: hg clone https://hg.python.org/cpython cd cpython copy mpx_enable.patch to the current directory hg import --no-commit mpx_enable.patch ./configure –with-mpx make –j #cores [1] https://software.intel.com/en-us/articles/introduction-to-intel-memory-protection-extensions [2] https://gcc.gnu.org/wiki/Intel%20MPX%20support%20in%20the%20GCC%20compiler Regards, Florin -- components: Extension Modules files: mpx_enable_3_6.patch keywords: patch messages: 252110 nosy: florin.papa priority: normal severity: normal status: open title: Enable Intel MPX (Memory protection Extensions) feature type: security versions: Python 3.6 Added file: http://bugs.python.org/file40657/mpx_enable_3_6.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: > Why not enablind the option by default if GCC 5.1 or newer is detected? Is > there a risk of breaking third-party extensions? > Should we pass the MPX compiler options to third-party extensions by way? On a processor that does not support MPX technology, MPX specific instructions will be replaced by nop's and will introduce a performance loss. Also, third party extensions might need to be patched in order to work with MPX. One example is the math module, which needed the bnd_legacy attribute to disable instrumentation when calling libc functions (which are not instrumented and generated compile errors). > I'm not sure that it's ok to add such change to Python 2.7. It's border line > between new feature and supporting a new architecture. I believe that Python 2.7 should benefit from this change, since it is still used in real life applications, one example being Openstack Swift. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25266] mako benchmark not working in Python 3.6
Florin Papa added the comment: It's OK. I just saw one of Claudiu Popa's issues and figured we have similar last names. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25266> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25266] mako benchmark not working in Python 3.6
Florin Papa added the comment: No problem. The name is Florin Papa :) -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25266> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25300] Enable Intel MPX (Memory protection Extensions) feature
Florin Papa added the comment: Hi David, If you are not sure about MPX hardware support, you can find out the processor model with the following commands: Linux - cat /proc/cpuinfo | grep model Mac - sysctl -n machdep.cpu.brand_string If the processor code is one of the following - i7-6xxx, i7-6xxxT, i5-6xxx, i5-6xxxT, i3-6xxx, i3-6xxxT - it is definitely part of the 6th Generation Intel processors and it supports MPX. Otherwise, it does not. Whether the hardware is MPX enabled is irrelevant for the build process. GCC 5.1 and higher will generate the MPX specific instructions, which will be turned into nop's at runtime if the processor is not MPX enabled. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25300> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25266] mako benchmark not working in Python 3.6
New submission from Florin Papa: Hi All, My name is Florin Papa and I work in the Server Languages Optimizations Team at Intel Corporation. I would like to submit a patch that deprecates the mako benchmark for Python 3.6 and above. The mako benchmark uses inspect.getargspec(), which is deprecated and was removed in Python 3.6. Therefore, it crashes with the message "AttributeError: module 'inspect' has no attribute 'getargspec'" when using the latest version of Python on the default branch. The patch limits the version range of the mako benchmark to Python 3.5 and below. To apply the patch please follow these steps: hg clone https://hg.python.org/benchmarks cd benchmarks/ copy mako_deprecation.patch to the current directory hg import --no-commit mako_deprecation.patch Regards, Florin -- components: Benchmarks files: mako_deprecation.patch keywords: patch messages: 251844 nosy: brett.cannon, florin.papa, pitrou priority: normal severity: normal status: open title: mako benchmark not working in Python 3.6 type: crash versions: Python 3.6 Added file: http://bugs.python.org/file40618/mako_deprecation.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25266> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25266] mako benchmark not working in Python 3.6
Florin Papa added the comment: There is already a mako_v2 benchmark in the Grand Unified Python Benchmarks, which does not use inspect.getargspec() and does not crash when using Python 3.6. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue25266> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Florin Papa added the comment: Please download Django from their github website: https://github.com/django/django I checked and their latest stable release - Django 1.8.4 - from https://pypi.python.org/pypi/Django still has the getargspec issue. Apparently the github version is 1.9.0 alpha. I have modified the patch and archive accordingly. -- Added file: http://bugs.python.org/file40314/django_v3.patch ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Changes by Florin Papa <florin.p...@intel.com>: Added file: http://bugs.python.org/file40315/Django-1.9.zip ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Florin Papa added the comment: The Django 1.8 archive that I attached is not patched, it is downloaded from the django github page without modifications. I have tested and Django 1.8 works with the latest Python version pulled from hg.python.org/cpython. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue24934> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Florin Papa added the comment: I have modified the Django-1.8.zip archive to extract to Django-1.8 directory. To apply the patch please follow these steps: hg clone https://hg.python.org/benchmarks cd benchmarks/ copy django_v3.patch to the current directory and Django-1.8.zip to the lib/ directory hg import --no-commit django_v3.patch cd lib/ unzip Django-1.8.zip -- Added file: http://bugs.python.org/file40272/Django-1.8.zip ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24934 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Florin Papa added the comment: I have modified the patch to introduce django_v3 to the benchmark suite and deprecate django_v2 for Python 3.6 and up. In order for django_v3 to work, the latest django release must be present in lib/Django-1.8. Django-1.8 is attached as a zip file because the patch would be too large if it included all these files. In order for the modifications to work, extract the archive to lib/Django-1.8 . Thank you, Florin -- Added file: http://bugs.python.org/file40259/django_v3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24934 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
Changes by Florin Papa florin.p...@intel.com: Added file: http://bugs.python.org/file40260/Django-1.8.zip ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24934 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24934] django_v2 benchmark not working in Python 3.6
New submission from Florin Papa: Hi All, My name is Florin Papa and I work in the Server Languages Optimizations Team at Intel Corporation. I would like to submit a patch that solves compatibility issues of the django_v2 benchmark in the Grand Unified Python Benchmark. The django_v2 benchmark uses inspect.getargspec(), which is deprecated and was removed in Python 3.6. Therefore, it crashes with the message ImportError: cannot import name 'getargspec' when using the latest version of Python on the default branch. The patch modifies the benchmark to use inspect.signature() when Python version is 3.6 or above and keep using inspect.getargspec() otherwise. Regards, Florin -- components: Benchmarks files: django_v2_compat_3_6.patch keywords: patch messages: 249119 nosy: florin.papa priority: normal severity: normal status: open title: django_v2 benchmark not working in Python 3.6 type: crash versions: Python 3.6 Added file: http://bugs.python.org/file40253/django_v2_compat_3_6.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue24934 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
